DOA estimation method, device and system using uncertainty quantification

Abstract

DOA estimation methods, apparatuses, and systems utilizing uncertainty quantification are disclosed. The apparatus includes a memory, an interface, and one or more processors. The one or more processors include a first neural network configured to operate according to a first machine learning (ML) model. The one or more processors are configured to obtain audio data. The one or more processors are configured to process the audio data using the first neural network to provide a first model output containing a direction of arrival estimate. The one or more processors are configured to obtain camera data. The one or more processors are configured to process the camera data to provide a video output. The one or more processors are configured to determine an uncertainty parameter based on the first model output and control the processing of the camera data to provide the video output based on the uncertainty parameter.

Description

Technical Field

[0001] This disclosure relates to methods, apparatus, and systems for estimating the direction of arrival (DOA) of communication signals using uncertainty quantization. Background Technology

[0002] Video conferencing is a form of communication or conference in which one or more people connect electronically and participate using audio and / or video technologies, rather than participating in person. Video conferencing can be conducted or joined using any combination of mobile or computing devices, specially designed rooms or environments, camera components, microphones, and / or displays. Whether video conferencing serves as a useful alternative to face-to-face communication often depends on the accuracy of the visual and auditory capture of participants. The framing of participants (the capture and presentation or depiction of participants within a video conference) affects the quality and experience of the video conference. Video conferencing systems that attempt to automatically frame participants may experience inaccuracies in the input used to perform the framing operation. Summary of the Invention

[0003] Accurate participant mapping in video conferencing can benefit from precise direction of arrival (DOA) estimation, such as estimating the DOA of the audio signal, which can be used to identify, highlight, and map the current speaker in the video. Poor mapping can occur in some systems due to inaccuracies in sound source localization caused by ambient noise, reverberation, and / or the presence of multiple speakers.

[0004] Therefore, there is a need for methods, devices, and systems for video conferencing that mitigate, reduce, or resolve existing shortcomings and provide accurate direction-of-arrival estimation for signals.

[0005] A video conferencing apparatus is provided. The apparatus includes memory, an interface, and / or one or more processors. The one or more processors include a neural network, such as a first neural network configured to run according to a first machine learning (ML) model. The one or more processors are configured to acquire audio data. The one or more processors are configured to process the audio data, for example, using the first neural network, to provide, for example, a first model output including a direction-of-arrival estimate from the first ML model. The one or more processors are configured to acquire camera data. The one or more processors are configured to process the camera data to provide video output. The one or more processors are configured to determine uncertainty parameters, for example, based on the first model output. The one or more processors are optionally configured to control the processing of the camera data to provide video output based on the uncertainty parameters.

[0006] Furthermore, a video conferencing system is provided. The system includes a camera, multiple microphones, memory, an interface, and / or one or more processors. The one or more processors include a neural network, such as a first neural network configured to run according to a first machine learning (ML) model. The one or more processors are configured to acquire audio data. The one or more processors are configured to process the audio data, for example, using the first neural network, to provide a first model output including a direction-of-arrival estimate. The one or more processors are configured to acquire camera data. The one or more processors are configured to process the camera data to provide video output. The one or more processors are configured to determine uncertainty parameters, for example, based on the first model output. The one or more processors are configured to control the processing of the camera data to provide video output, for example, based on the uncertainty parameters.

[0007] A method for operating a video conferencing system or apparatus is provided. The system or apparatus includes memory, an interface, and / or one or more processors. The one or more processors include a neural network, such as a first neural network configured to run according to a first machine learning (ML) model. The method includes acquiring audio data. The method includes processing the audio data, for example, using the first neural network (such as the first ML model), to provide a first model output including a direction-of-arrival estimate. The method includes acquiring camera data. The method includes processing the camera data to provide video output. The method includes determining uncertainty parameters, for example, based on the first model output (such as the direction-of-arrival estimate). The method includes controlling the processing of the camera data to provide video output based on the uncertainty parameters.

[0008] The advantage of this disclosure is that the arrival direction estimation in the video conferencing system can be accurate enough to support the management aspects of the video conferencing system.

[0009] Furthermore, the advantage of this disclosure is that uncertainty quantification (such as uncertainty parameters) can be used for arrival direction estimation and / or for managing various aspects of video conferencing systems.

[0010] Another advantage of this disclosure is that the composition of participants in a video conference can be managed using arrival direction estimation that uses uncertainty quantification (such as uncertainty parameters).

[0011] Another advantage of this disclosure is that the device can employ a neural network, which can be trained to improve the composition and thus improve the video conferencing experience for a particular video conferencing system.

[0012] Another advantage of this disclosure is that active participants (such as the currently speaking participant) can be identified and mapped to be different from inactive participants (such as the currently non-speaking participant), and such identification and mapping can be done without supporting methods for detecting lip movements or other non-auditory processes.

[0013] Using audio signal estimation of DOA can quickly identify event participants, thereby improving participant composition in video conferencing. Attached Figure Description

[0014] The above and other features and advantages of this disclosure will become apparent to those skilled in the art from the following detailed description of exemplary embodiments thereof, with reference to the accompanying drawings, in which: Figure 1 This is a diagram illustrating an example video conferencing system including one or more example video conferencing devices according to the present disclosure; Figure 2A and Figure 2B This is a flowchart illustrating an example method performed in an example video conferencing system and / or apparatus according to the present disclosure, and Figure 3 This is a block diagram illustrating an example video conferencing apparatus according to the present disclosure. Detailed Implementation

[0015] In the following description, various exemplary embodiments and details are illustrated with reference to the accompanying drawings. It should be noted that the drawings may be drawn to scale or not, and throughout the drawings, elements with similar structures or functions are indicated by the same reference numerals. It should also be noted that the drawings are intended only to facilitate the description of embodiments. They are not intended as an exhaustive description of this disclosure or as a limitation on the scope of this disclosure. Furthermore, the illustrated embodiments do not need to have all the aspects or advantages shown. Aspects or advantages described in connection with a particular embodiment are not necessarily limited to that embodiment and may be practiced in any other embodiment, even if not so stated or explicitly described.

[0016] In one or more examples, a video conferencing system or apparatus is provided. The system or apparatus may be a video conferencing device. The system or apparatus may include memory, an interface (e.g., an interface for a camera, microphone, display, and / or network), and / or one or more processors. In one or more examples, the one or more processors include one or more neural networks, such as a first neural network configured to run according to a first machine learning (ML) model. For example, one or more processors may each include a first neural network and / or one or more processors may collectively include a first neural network. The one or more processors may additionally or alternatively include a second neural network and / or other neural networks. The first neural network may be one of a plurality of neural networks included by the one or more processors. The first ML model may be one of a plurality of models configured to run according to the one or more processors. Thus, one or more processors may include a plurality of neural networks, each configured to run according to one or more ML models. The first neural network disclosed herein can be considered as an arrival direction DoA estimator and / or DoA predictor (such as a DoA estimator / predictor with uncertainty parameter output) configured to run according to a first machine learning model. One or more processors including neural networks (such as a first neural network) can be understood as one or more processors including dedicated modules or units configured to run neural network and machine learning models as disclosed herein, such as being configured to act as a DoA estimator and / or DoA predictor. For example, neural networks (such as a first neural network) may be stored on the memory of a video conferencing device and run by a dedicated processor or a portion of one or more processors.

[0017] As used herein, one or more processors including a first neural network can refer to one or more processors that read data and / or other instructions from memory of the neural network. Thus, one or more processors can be general-purpose processors capable of executing instructions representing a neural network. Therefore, for example, when a device including one or more processors is powered off and coupled to memory (e.g., a non-volatile memory device) that itself includes (e.g., stores) data representing a neural network (or multiple neural networks), one or more processors can be understood as including a neural network.

[0018] As used in this paper, an ML model can be understood as or includes algorithms, such as computational algorithms, that map input data to a desired output by identifying statistical patterns through an iterative training process. The process of training an ML model can be called learning. An ML model can include parameters, such as numerical values, for example, numbers representing weights and biases. During training, these parameters can be tuned to minimize the difference between the model's predictions and the actual correct output. An ML model, such as the first ML model described in this paper, can have a specific configuration of interconnected computational units that define how the input data is processed and transformed. Such an ML model can therefore be understood as a neuron or operates like a neuron, and can therefore be called a neural network, such as the first neural network described in this paper.

[0019] Neural networks (such as the first neural network described in this paper) can therefore be understood as layers of interconnected nodes, where each connection has weights that are adjusted during training. A graphical representation of such a network can resemble the representation of a biological neural network. In one or more examples, the ML model learns by minimizing a loss function that quantifies the difference between predicted and actual results. This can include techniques similar to backpropagation, which propagates error information backward through the network and updates the parameters accordingly.

[0020] The application or use of neural networks and / or ML models differs from other programming or software implementations because the model may not follow predefined rules; instead, ML models can have decision-making (e.g., reasoning) logic created based on or according to patterns present in the training data.

[0021] Training an ML model (or neural network) can be understood as the process or procedure of systematically updating the parameters of the ML model. This can include mathematical and / or statistical analysis of the error between the predicted results and the actual or expected results. Example training algorithms can include gradient descent, which gradually adjusts the parameters and thereby reduces prediction error.

[0022] As described herein, one or more processors (such as one or more processors in a video conferencing system or apparatus) are configured to acquire audio data and process the audio data using a neural network (e.g., a first neural network) to provide a first model output (e.g., the output of a first ML model) that includes an estimate of the direction of arrival. In other words, the first model output may include a direction parameter indicating the estimated direction of arrival of the audio.

[0023] One or more processors may additionally acquire camera data. Camera data may be acquired, for example, based on the output of a first model. In other words, camera data may be selected, identified, and / or acquired based on the determination and / or output of a first neural network. Camera data may be acquired, for example, from memory coupled to one or more processors, and / or camera data may be acquired, for example, from a camera via an interface. In one or more examples, camera data is acquired from multiple sources, which may include memory coupled to one or more processors (or to a device comprising one or more processors), one or more cameras, and / or other devices (such as computing devices).

[0024] One or more processors can process camera data to provide video output. Processing camera data may include processing with and / or without using neural networks (such as a first neural network). For example, the processor may use a first neural network to process the camera data; for example, the camera data may be additional input to the first neural network. In one or more examples, camera processing is based on automatic photographic mapping and / or speaker tracking. The processor may employ a specific algorithm, such as part of a second neural network, which may be based on face detection or the direction of arrival of one or more audio / sound signals, or both. Alternatively, the processor may apply another algorithm employing predefined rules to process the camera data.

[0025] As used herein, composition can be understood as the arrangement and / or positioning of participants within the field of view of one or more cameras. As mentioned above, composition affects the quality of a video conference. Appropriately or accurately capturing the composition of participants in a video conference can create an engaging and visually appealing video communication experience, and thus create a quality video conference. Poor composition can produce negative or poor video, resulting in a poor video conference. Therefore, composition is an important consideration for continuous monitoring and adjustment within a video conferencing system, and is thus a crucial aspect. Composition can be modified or adjusted based on different camera settings or schemes.

[0026] This disclosure enables the identification and mapping of active participants (such as the currently speaking participant) that are distinct from passive participants (such as a participant who is not currently speaking). Other systems may rely primarily or exclusively on motion, such as monitoring lip or other facial movements, to identify active speakers. For example, as described herein, the use of audio data can achieve more accurate and / or faster mapping compared to other methods by leveraging direction-of-arrival estimation.

[0027] One or more processors can determine uncertainty parameters based on the output of a first model. Uncertainty parameters can be understood as a quantitative measure of the confidence or variability of a neural network or ML model (such as the first ML model) in its predictions. This paper describes various aspects and techniques used for uncertainty quantification. For example, using heteroscedasticity regression, the model can be applied with the learned variance or standard deviation accompanying each prediction.

[0028] Additionally or alternatively, the uncertainty parameter may be or include the prediction variance, which can directly represent the extent of the possible prediction. The prediction variance can be determined or calculated as a parameter parallel to the master prediction, and / or it can indicate the deviation of the ML model from the true value.

[0029] Additionally or alternatively, uncertainty parameters may include cognitive uncertainty, which can capture or reflect a model’s lack of training or learning about the underlying data distribution. This can be estimated using techniques similar to lossy or ensemble methods discussed in this paper.

[0030] Additionally or alternatively, the uncertainty parameter may include random uncertainty, which can represent inherent noise or randomness in the data, such as audio data. This can reflect errors that cannot be reduced by utilizing more data, such as by obtaining additional audio data, and can be used in scenarios with inherent metric variability.

[0031] Additionally or alternatively, the estimation of the uncertainty parameter or such parameter may include or may employ one or more other techniques, including Bayesian neural networks, Monte Carlo dropout, deep ensembles, quantile regression, and / or evidence-based deep learning. In one or more examples, such tools and techniques may provide different approaches for estimating uncertainty and may be adopted based on the specific needs and constraints of the application.

[0032] In one or more examples, the uncertainty parameter, or the estimation of such parameter, may include and / or can be determined using multiple techniques, such as by combining uncertainty parameters, each of which is determined using a different technique. Such an approach can produce accurate uncertainty predictions relative to, for example, those dependent on a single technique, because the advantages or benefits of various techniques can be taken advantage of; for example, a first ML model (such as a Bayesian neural network) can provide a probabilistic framework to capture model uncertainty, which can be understood as explaining cognitive uncertainty, while a second ML model (such as a deep ensemble) can effectively model data uncertainty, which can be understood as explaining random uncertainty.

[0033] Uncertainty parameters can be measures of the uncertainty of a direction-of-arrival (DOA) estimate and / or the confidence level of the DOA estimate. A DOA estimate can be understood as determining the direction of arrival of an audio or sound signal, such as an audio signal from a participant (e.g., a person) speaking or otherwise making noise during a video conference. A first model output may include DOA estimates associated with estimates of one or more participants in the video conference, such as estimates of the presence of one or more participants in the video conference. For example, processing of audio data may include providing a first model output that includes one or more DOA estimates associated with estimates of one or more participants in the video conference, such as estimates of the presence of one or more participants in the video conference. For example, the first model output may include a first DOA associated with the presence of a first participant in the video conference, a second DOA associated with the presence of a second participant in the video conference, and / or a third DOA associated with the presence of a third participant in the video conference, etc. In other words, the DOA estimate may also provide information about the number of participants in the video conference, the location of the participants, the identity of the participants, and / or the tracking of the participants. In one or more example video conferencing setups, the first model output (such as an estimate of one or more participants) can provide an estimate of whether audio emanating from a certain direction originates from a participant in the video conference or from another source (such as noise from outside the video conference, i.e., not estimated to be a participant).

[0034] In some examples, the direction of arrival estimation can be associated with multiple participants. For example, when two or more participants are approximately in the same line of sight on a video conferencing device. In one or more examples, the ML model (such as the first model) has one or more nodes configured to be deactivated, to be deactivated when processing audio data to provide the direction of arrival estimation.

[0035] The first neural network can be configured to output an uncertainty parameter based on a loss function that includes the variance of the direction of arrival (DOA) estimate. For example, such a configuration can be understood as enabling the first neural network to account for variability in observation noise and input dependencies during training and inference. For instance, in the case of a probabilistic regression task, a neural network (such as the first neural network) can be trained using a loss function that includes the variance of the DOA estimate (such as negative log-likelihood) to directly output uncertainty (e.g., an uncertainty parameter). Such a loss function can be understood as effective (e.g., more effective than other techniques) in scenarios where noise levels vary across the input, because it allows the first neural network to assign higher uncertainty to noisier observations while maintaining confidence in cleaner (e.g., relatively less noisy) observations. The loss function or another function used to determine or select the uncertainty parameter can be based on or related to heteroscedasticity regression, which may include modeling the variance of the input dependencies to capture different levels of observation noise.

[0036] In one or more examples, the first neural network is configured to perform uncertainty prediction, such as heteroscedastic regression and / or other uncertainty estimation, by outputting an estimated direction of arrival (DOA) and its associated variance parameter, and / or utilizing a loss function based on observation noise in the DOA estimation. The variance parameter can, for example, represent the predictive uncertainty of the DOA estimate given the input, which allows the first neural network to dynamically adapt to varying levels of observation noise. In one or more examples, the training process can incorporate regularization techniques (such as variance smoothing or clipping) to stabilize the learning of the uncertainty parameter, preventing excessively high and / or low estimates of uncertainty.

[0037] For example, a video conferencing system (such as a video conferencing device or a processor of such a device or a neural network of such a device) can utilize a loss function based on observation noise in the direction of arrival, including observation noise that takes into account variations in the direction of arrival.

[0038] In one or more examples, one or more processors of a video conferencing system (such as one or more processors of a device within the system) are configured to determine whether an uncertainty parameter meets a first criterion, such as a criterion associated with a confidence level of the DOA estimate. Based on the uncertainty parameter meeting the first criterion, the one or more processors can process camera data according to a first processing scheme to provide video output. The first processing scheme can be a first camera processing scheme, such as a first framing scheme. For example, camera data can be processed based on a framing (e.g., image capture settings) associated with one or more cameras. In other words, the processing scheme disclosed herein can be considered as a scheme for processing camera data in a particular manner. For example, the processing scheme can include framing settings, zoom settings, and / or windowing settings. Framing settings can include, for example, settings that determine how many frames the camera data should be divided into based on the number of participants (such as the number of participants in an event). In other words, the processing scheme can ensure that camera data is processed such that each participant has a separate frame. Thus, camera data can be processed such that each participant is divided into a separate frame in the video output. Zoom settings can include settings for determining the zoom (such as digital zoom) of the camera data. For example, if different participants are located at different distances from the video conferencing device, different zoom settings can be applied to different participants and therefore to different frames. This allows all participants to appear at essentially the same size on the video output. Window settings can include configurations for how camera data is processed into different windows (e.g., different windows for each participant or each group of participants).

[0039] One or more processors may be configured to process camera data according to a second processing scheme to provide video output, for example, when the uncertainty parameters do not meet a first criterion and / or when the uncertainty parameters meet a second criterion. The second processing scheme can be understood as different settings or configurations of camera, composition, and / or image capture settings relative to the first processing scheme.

[0040] One or more processors may be configured to process camera data according to a third processing scheme to provide video output if the uncertainty parameters do not meet the first criterion and / or the second criterion. The third processing scheme can be understood as a different setting or configuration of camera, composition, and / or image capture settings relative to the first and / or second processing schemes.

[0041] Video conferencing systems (such as devices within the system, which may be or contain processors of such devices) can therefore control the processing of camera data to provide video output based on uncertain parameters.

[0042] In one or more examples, the processing of camera data includes identifying one or more frames of one or more active participants in a video conference based on arrival direction estimation and uncertainty parameters.

[0043] In one or more examples, controlling the processing of camera data includes adjusting the camera pattern based on uncertainty parameters and direction-of-arrival estimates.

[0044] In one or more examples, the system (such as the device or one or more processors or such devices) is configured to control one or more cameras based on uncertainty parameters and direction-of-arrival estimates.

[0045] In one or more examples, a video conferencing apparatus is provided, including a memory, an interface, and / or one or more processors, the one or more processors including a first neural network configured to run according to a first machine learning (ML) model. The one or more processors are configured to: acquire audio data; process the audio data, for example, using the first neural network to provide a first model output (e.g., including a direction-of-arrival estimate); acquire camera data; process the camera data to provide video output; determine uncertainty parameters, for example, based on the first model output; and control the processing of the camera data to provide video output, for example, based on the uncertainty parameters.

[0046] The following reference is made to the accompanying drawings, which are schematic and simplified for clarity, and illustrate only the details that aid in understanding this disclosure, while other details have been omitted. Throughout the text, the same reference numerals are used for the same or corresponding parts.

[0047] Figure 1This is a diagram illustrating an example video conferencing system 1 according to the present disclosure. The video conferencing system 1 includes a camera 10, multiple microphones 12, memory 18, an interface 20, and one or more processors 16. A server 14, camera 10, and / or microphones 12 are examples of video conferencing devices. Server 14 may be a physical or virtual computer or machine. Server 14 may be a mobile computing device, such as a mobile phone or tablet. Server 14 may be a control unit or console for system 14. In some examples, server 14 is integrated with a display unit, such as a television. Additionally or alternatively, server 14 may be an element or aspect of camera 10. Camera 10 may be a standalone camera optimized for video conferencing. Alternatively, camera 10 may be an element of another device, such as a computer (e.g., a laptop computer) or a mobile device (such as a tablet or mobile phone). Camera 10 may include optics adjustable to control the composition. Camera 10 may include an image sensor, an image processor, and / or a user interface. Similarly, microphone 12 may be a standalone device optimized for video conferencing. Alternatively, microphone 12 may be a component of another device, such as a computer (e.g., a laptop computer) or a mobile device (such as a tablet or mobile phone). Although depicted as a component of server 14, in some examples, processor 16, memory 18, and / or interface 20 may be components of camera 10 and / or microphone 12.

[0048] In one or more examples, processor 16, memory 18, and interface 20 are components of server 14 or other video conferencing devices of system 1. One or more processors include neural networks, such as a first neural network 22 configured to operate according to a first ML model. One or more processors 16 are configured to, for example, acquire audio data 24 from multiple microphones 12 via interface 20. One or more processors 16 are configured to process the audio data 24 using the first neural network 22 to provide a first model output including direction-of-arrival estimates, such as direction-of-arrival estimates of audio / sound signals 26 from participants 28 of a video conference using system 1. One or more processors are configured to, for example, acquire camera data 25 from camera 10 via interface 20. One or more processors 16 are configured to process the camera data 25, for example, via the first neural network 22 to provide video output. One or more processors 16 are configured to determine uncertainty parameters based on the first model output. One or more processors 16 are configured to control the processing of camera data 25 to provide video output based on the uncertainty parameters.

[0049] In one or more examples, using the first neural network 22 includes outputting uncertainty parameters based on a loss function that incorporates the variance of the arrival direction estimate. In one or more examples, using the first neural network 22 includes performing uncertainty prediction (e.g., heteroscedastic regression) by outputting the estimated arrival direction and the associated variance parameter; and utilizing a loss function based on observation noise in the arrival direction estimate. In one or more examples, utilizing the loss function based on observation noise in the arrival direction includes taking into account changes in observation noise in the arrival direction.

[0050] In one or more examples, the first model output includes an arrival direction estimate associated with the estimates of one or more participants 28 in the video conference. In one or more examples, the first model has one or more nodes configured to be deactivated, to perform the deactivation of one or more nodes while processing audio data to provide the arrival direction estimate.

[0051] In one or more examples, processing of camera data 25 includes identifying (e.g., using processor 16) one or more frames of one or more active participants 28 in a video conference based on direction-of-arrival estimation (such as an estimation of the direction of arrival of audio / sound signal 26) and uncertainty parameters. In one or more examples, controlling the processing of camera data 25 includes adjusting phase pattern 30 based on uncertainty parameters and direction-of-arrival estimation. Phase pattern 30 may also be referred to as one or more frames. Adjusting phase pattern 30 and / or identifying one or more frames may include or be based on changes to the view or area captured by camera 10. This may include zooming, panning, tilting, focusing, or other similar electronic physical manipulations of one or more images captured by camera 10.

[0052] In one or more examples, the uncertainty parameter indicates a measure of the uncertainty of the predicted direction of arrival estimate and / or the confidence level of the direction of arrival estimate. In one or more examples, one or more processors 16 are configured to control one or more cameras 10 based on the uncertainty parameter and the direction of arrival estimate.

[0053] Processor 16 can take various actions or process data, such as camera data 25, based on different criteria met by uncertainty parameters. In one or more examples, processor 16 can determine whether the uncertainty parameters meet a first criterion, and if the uncertainty parameters meet the first criterion, process camera data 25 according to a first processing scheme to provide video output to a display or network. In one or more examples, if the uncertainty parameters do not meet the first criterion, and / or meet a second criterion, processor 16 processes camera data 25 according to a second processing scheme for purposes such as providing video output to a display or network. In one or more examples, if the uncertainty parameters do not meet the first criterion and do not meet the second criterion, processor 16 processes camera data 25 according to a third processing scheme for purposes such as providing video output to a display or network.

[0054] Figure 2A and Figure 2B This is a flowchart illustrating an example method performed in an example video conferencing system and / or apparatus according to this disclosure. The video conferencing apparatus may be a video conferencing system (such as reference ... Figure 1 The system described 1) or video conferencing device (such as reference) Figure 3 The described device 300). The video conferencing system and / or device may include a memory (such as... Figure 1 The memory 18 and / or Figure 3 The memory circuit 301 in the middle), and the interface (such as Figure 1 Interface 20 and / or Figure 3 Interface 303 in the middle) and / or one or more processors (such as Figure 1 The processor 16 and / or Figure 3 (Processor circuitry 302 in the example). In one or more examples, the processor includes one or more neural networks, such as a first neural network configured to run according to a first ML. Method 100 includes, at S102, obtaining audio data, such as reference... Figure 1 The described audio data is 24.

[0055] Method 100 includes, at S104, using a first neural network (such as...) Figure 1 The neural network 22 in the image is used to process (e.g., utilizing one or more processors) audio data to provide a first model output containing an estimate of the direction of arrival. In one or more examples, the first model output includes information about one or more participants in a video conference (such as...). Figure 1 The estimated associated directions of arrival (such as those of the participants in the study) 28) Figure 1The first neural network is used to estimate the direction of arrival (ROA) of the audio / sound signal 26. In one or more examples, using the first neural network includes: outputting an uncertainty parameter S104A based on a loss function that incorporates the variance of the ROA estimate. In one or more examples, using the first neural network includes: performing an uncertainty prediction (such as heteroscedastic regression and / or other uncertainty estimation) S104B by outputting the estimated ROA and the associated variance parameter, and utilizing a loss function S104C based on observation noise in the ROA estimate. In one or more examples, utilizing the loss function based on observation noise in the ROA includes: taking into account the variation of observation noise in the ROA direction S104C1. In one or more examples, the first model has one or more nodes configured to be deactivated to perform deactivation of one or more nodes while processing audio data to provide an ROA estimate.

[0056] Method 100 includes, at S106, obtaining camera data, such as Figure 1 Camera data 25.

[0057] Method 100 includes processing camera data at S108 to provide video output.

[0058] Method 100 includes, at S110, determining uncertainty parameters based on the first model output. In one or more examples, the uncertainty parameters indicate a measure of the uncertainty of the predicted direction of arrival estimate and / or the confidence level of the direction of arrival estimate. In one or more examples, one or more processors are configured to control one or more cameras based on the uncertainty parameters and the direction of arrival estimate.

[0059] Method 100 includes, at S112, controlling the processing of camera data based on uncertainty parameters to provide video output. In one or more examples, the processing of camera data includes identifying one or more frames of one or more active participants in a video conference based on direction-of-arrival estimation and uncertainty parameters at S112A. In one or more examples of the method, the processing of camera data includes adjusting a phase pattern at S112B based on uncertainty parameters and direction-of-arrival estimation.

[0060] In one or more examples, the method includes determining whether the uncertainty parameter S114 meets a first criterion, and processing the camera data S116 to provide video output according to a first processing scheme based on the uncertainty parameter meeting the first criterion.

[0061] In one or more examples, the method includes processing S118 camera data to provide video output according to a second processing scheme based on whether the uncertainty parameter does not meet a first criterion and / or whether the uncertainty parameter meets a second criterion.

[0062] In one or more examples, the method includes processing S120 camera data to provide video output according to a third processing scheme based on the uncertainty parameter not meeting a first criterion and the uncertainty parameter not meeting a second criterion.

[0063] Figure 3 This is a block diagram illustrating an example video conferencing apparatus 300 according to the present disclosure. The video conferencing apparatus 300 includes memory circuitry 301, processor circuitry 302, and interface 303. In one or more examples, the video conferencing apparatus 300 may be a camera (such as...) Figure 1 Camera 10), microphone (such as...) Figure 1 Microphone 12) or server (such as Figure 1 (Server 14 in the middle). In other words, the video conferencing device 300 can be a standalone electronic device, or it can be, for example, a server 14 in the middle. Figure 1 It is a component of System 1 and can perform various other functions, such as those associated with a camera, microphone, or other video conferencing device.

[0064] Memory circuit 301 can be Figure 1 An example of memory 18 in the memory. Processor circuit 302 may be... Figure 1 Example of processor 16 in the example. Interface 303 can be... Figure 1 An example of interface 20. In one or more examples, interface 303 is configured to receive signals from one or more cameras (such as...). Figure 1 (camera 10) and / or one or more microphones (such as Figure 1 Microphone 12) receives signals and / or sends signals to one or more cameras (such as...) Figure 1 (camera 10) and / or one or more microphones (such as Figure 1 Microphone 12 in the middle).

[0065] Processor circuitry 302 and / or memory circuitry 301 include neural networks, such as a first neural network configured to run according to a first ML model. In one or more examples, processor circuitry 302 executes code or instructions read from memory circuitry 301, and therefore includes the first neural network. The first neural network may be... Figure 1 The neural network in 22.

[0066] The video conferencing device 300 can be configured to perform any of the methods disclosed herein, such as those described above. Figure 2A and Figure 2B The method described. In other words, the video conferencing device 300 can be configured to use uncertainty quantization to estimate the direction of arrival of communication signals.

[0067] The video conferencing device 300 is configured to acquire audio data (e.g., via interface 303), such as Figure 1 Audio data 24.

[0068] The video conferencing device 300 is configured to use a first neural network to process (e.g., using processor circuitry 302) audio data to provide information including direction-of-arrival estimation (such as... Figure 1 The first model output is the direction of arrival estimation of the audio / sound signal 24 in the model.

[0069] The video conferencing device 300 is configured to acquire camera data (e.g., via interface 303), such as Figure 1 Camera data 25.

[0070] The video conferencing device 300 is configured (such as by utilizing processor circuitry 302) to process camera data to provide video output.

[0071] The video conferencing device 300 is configured to determine (e.g., using processor circuitry 302) uncertainty parameters based on a first model output. In one or more examples, the first model output includes parameters related to one or more participants in the video conference (such as...). Figure 1 The estimates of the associated direction of arrival (such as those of participants 28) Figure 1 (Estimation of the direction of arrival of audio / sound signal 26 in the video / sound signal).

[0072] In one or more examples, the first neural network is configured to output uncertainty parameters based on a loss function that includes the variance of the arrival direction estimate. In one or more examples, the first neural network is configured to perform uncertainty prediction (such as heteroscedastic regression) by outputting the estimated arrival direction and the associated variance parameter, and / or utilize a loss function based on observation noise in the arrival direction estimate. In one or more examples, uncertainty prediction includes heteroscedastic regression. In one or more examples, utilizing a loss function based on observation noise in the arrival direction includes taking into account changes in observation noise in the arrival direction. In one or more examples, the first model has one or more nodes configured to be deactivated to perform deactivation of one or more nodes while processing audio data to provide an arrival direction estimate.

[0073] The video conferencing device 300 is configured to control (such as using processor circuitry 302) the processing of camera data based on uncertainty parameters to provide video output. In one or more examples, controlling the processing of camera data includes based on direction-of-arrival estimation (such as... Figure 1 The arrival direction estimation of the audio / sound signal 26 in the video conference and uncertainty parameters are used to identify one or more active participants in the video conference (such as...). Figure 1One or more frames of participant 28). In one or more examples, controlling the processing of camera data includes adjusting the phase diagram based on uncertainty parameters and direction-of-arrival estimates (e.g., based on...). Figure 1 (The composition of the position 30 of camera 10). In one or more examples, the uncertainty parameter indicates the uncertainty of the predicted direction of arrival and / or a measure of the confidence level of the direction of arrival estimate.

[0074] In one or more examples, the video conferencing device 300 is configured to control (such as using processor circuitry 302) one or more cameras based on uncertainty parameters and direction of arrival estimation.

[0075] In one or more examples, the video conferencing device 300 is configured (such as using processor circuitry 302) to determine whether an uncertainty parameter meets a first criterion, such as if the uncertainty parameter is within or outside a range, if the uncertainty parameter is less than or equal to a threshold, or if the uncertainty parameter is greater than or equal to a threshold, and to process camera data according to a first processing scheme to provide video output based on the uncertainty parameter meeting the first criterion.

[0076] In one or more examples, the video conferencing device 300 is configured (such as using processor circuitry 302) to process camera data according to a second processing scheme to provide video output based on the uncertainty parameters not meeting a first criterion and / or meeting a second criterion based on the uncertainty parameters.

[0077] In one or more examples, the video conferencing device 300 is configured (such as using processor circuitry 302) to process camera data according to a third processing scheme to provide video output based on the uncertainty parameters not meeting a first criterion and the uncertainty parameters not meeting a second criterion.

[0078] The operation and / or configuration of the video conferencing device 300 may be embodied in the form of executable logic routines (e.g., lines of code, software programs, etc.) stored on a non-transitory computer-readable medium (e.g., memory circuitry 301) and executed by processor circuitry 302.

[0079] Furthermore, the operation of device 300 can be considered as the electronic device 300 being configured to perform a method. Moreover, although the described functions and operations can be implemented in software, such functions can be additionally or alternatively performed via dedicated hardware or firmware, or a combination of hardware, firmware and / or software.

[0080] The memory circuit 301 may be one or more of a buffer, flash memory, hard disk drive, removable media, volatile memory, non-volatile memory, random access memory (RAM), or other suitable devices. In a typical arrangement, the memory circuit 301 may include non-volatile memory for long-term data storage and volatile memory used as system memory for the processor circuit 302. The memory circuit 301 may exchange data with the processor circuit 302 via a data bus. Control lines and an address bus may also exist between the memory circuit 301 and the processor circuit 302. Figure 3 (Not shown in the image). The memory circuit 301 is considered a non-transitory computer-readable medium.

[0081] The memory circuit 301 can be configured to store information (such as information indicating a second set of paging resources) in a portion of the memory.

[0082] The processor circuit 302 may be or may include one or more integrated circuits. The processor circuit 302 may be or include at least one of a CPU, GPU, GPGPU, ASIC, FPGA, and any combination thereof.

[0083] Interface 303 may be or may include hardware, software, and / or firmware for this communication. Interface 303 may be or may include an audio codec or a video codec or both.

[0084] The embodiments of the methods and products disclosed herein are described in the following: Item 1. A video conferencing apparatus, comprising a memory, an interface, and one or more processors, the one or more processors including a first neural network configured to operate according to a first machine learning (ML) model, wherein the one or more processors are configured to: acquire audio data; process the audio data using the first neural network to provide a first model output including a direction-of-arrival estimate; acquire camera data; process the camera data to provide a video output; determine uncertainty parameters based on the first model output; and / or control the processing of the camera data to provide the video output based on the uncertainty parameters.

[0085] Item 2. The video conferencing apparatus according to Item 1, wherein processing of camera data includes: identifying one or more frames of one or more active participants in the video conference based on direction of arrival estimation and uncertainty parameters.

[0086] Item 3. The video conferencing apparatus according to any one of items 1 to 2, wherein controlling the processing of camera data includes: adjusting the camera pattern based on uncertainty parameters and direction of arrival estimation.

[0087] Item 4. The video conferencing apparatus according to any one of items 1 to 3, wherein the uncertainty parameter indicates the uncertainty of the prediction of the direction of arrival estimate and / or a measure of the confidence level of the direction of arrival estimate.

[0088] Item 5. The video conferencing apparatus according to any one of items 1 to 4, wherein the first model output includes: an arrival direction estimate associated with estimates of one or more participants in the video conference.

[0089] Item 6. The video conferencing apparatus according to any one of items 1 to 5, wherein one or more processors are configured to control one or more cameras based on uncertainty parameters and direction of arrival estimation.

[0090] Item 7. The video conferencing apparatus according to any one of items 1 to 6, wherein one or more processors are configured to: determine whether an uncertainty parameter satisfies a first criterion, and, based on the uncertainty parameter satisfying the first criterion, process camera data according to a first processing scheme to provide video output.

[0091] Item 8. The video conferencing apparatus according to Item 7, wherein one or more processors are configured to process camera data to provide video output according to a second processing scheme based on whether the uncertainty parameter does not meet a first criterion and / or whether the uncertainty parameter meets a second criterion.

[0092] Item 9. The video conferencing apparatus according to Item 8, wherein one or more processors are configured to process camera data according to a third processing scheme to provide video output based on the uncertainty parameter not meeting a first criterion and the uncertainty parameter not meeting a second criterion.

[0093] Item 10. The video conferencing apparatus according to any one of items 1 to 9, wherein the first neural network is configured to output uncertainty parameters based on a loss function that includes variance of the direction of arrival estimation.

[0094] Item 11. The video conferencing apparatus according to any one of items 1 to 10, wherein the first neural network is configured to: perform uncertainty prediction (such as heteroscedastic regression) by outputting estimated directions of arrival and associated variance parameters; and / or utilize a loss function based on observation noise in the direction of arrival estimation.

[0095] Item 12. The video conferencing apparatus according to Item 11, wherein utilizing the loss function based on observation noise in the direction of arrival includes: taking into account the variation of observation noise in the direction of arrival.

[0096] Item 13. The video conferencing apparatus according to any one of items 1 to 12, wherein the first model has one or more nodes configured to be deactivated to perform the deactivation of one or more nodes while processing audio data to provide a direction of arrival estimate.

[0097] Item 14. A video conferencing system comprising a camera, a plurality of microphones, a memory, an interface, and one or more processors, the one or more processors including a first neural network configured to operate according to a first machine learning (ML) model, wherein the one or more processors are configured to: acquire audio data; process the audio data using the first neural network to provide a first model output including a direction-of-arrival estimate; acquire camera data; process the camera data to provide video output; determine uncertainty parameters based on the first model output; and / or control the processing of the camera data to provide video output based on the uncertainty parameters.

[0098] Item 15. A method of operating a video conferencing apparatus, the video conferencing apparatus including a memory, an interface, and one or more processors, the one or more processors including a first neural network configured to operate according to a first machine learning (ML) model, the method comprising: acquiring audio data; processing the audio data using the first neural network to provide a first model output including a direction-of-arrival estimate; acquiring camera data; processing the camera data to provide a video output; determining uncertainty parameters based on the first model output; and / or controlling the processing of the camera data to provide the video output based on the uncertainty parameters.

[0099] Item 16. The method according to Item 15, wherein processing of the camera data includes: identifying one or more frames of one or more active participants in a video conference based on direction-of-arrival estimation and uncertainty parameters.

[0100] Item 17. The method according to any one of items 15 to 16, wherein controlling the processing of camera data includes: adjusting the phase pattern based on uncertainty parameters and direction of arrival estimates.

[0101] Item 18. The method according to any one of items 15 to 17, wherein the uncertainty parameter indicates the uncertainty of the prediction of the direction of arrival estimate and / or a measure of the confidence level of the direction of arrival estimate.

[0102] Item 19. The method according to any one of items 15 to 18, wherein the first model output includes: an arrival direction estimate associated with estimates of one or more participants in the video conference.

[0103] Item 20. The method according to any one of items 15 to 19, wherein one or more processors are configured to control one or more cameras based on uncertainty parameters and direction-of-arrival estimates.

[0104] Item 21. The method according to any one of items 15 to 20, the method comprising: determining whether an uncertainty parameter satisfies a first criterion, and processing camera data according to a first processing scheme to provide video output based on the uncertainty parameter satisfying the first criterion.

[0105] Item 22. The method according to Item 21, wherein the camera data is processed according to the second processing scheme to provide video output based on the uncertainty parameter not meeting the first criterion and / or based on the uncertainty parameter meeting the second criterion.

[0106] Item 23. The method according to Item 22, wherein the camera data is processed according to a third processing scheme to provide video output based on the uncertainty parameter not meeting the first criterion and the uncertainty parameter not meeting the second criterion.

[0107] Item 24. The method according to any one of items 15 to 23, wherein using the first neural network includes: outputting an uncertainty parameter based on a loss function that includes the variance of the arrival direction estimate.

[0108] Item 25. The method according to any one of items 15 to 24, wherein using the first neural network includes: performing uncertainty prediction (such as heteroscedastic regression) by outputting an estimated direction of arrival and an associated variance parameter; and utilizing a loss function based on observation noise in the direction of arrival estimation.

[0109] Item 26. The method according to any one of items 15 to 25, wherein utilizing a loss function based on observation noise in the direction of arrival includes: taking into account the variation of observation noise in the direction of arrival.

[0110] Item 27. The method according to any one of items 15 to 26, wherein the first model has one or more nodes configured to be deactivated to perform deactivation of one or more nodes while processing audio data to provide a direction of arrival estimate.

[0111] The use of the terms "first," "second," "third," and "fourth," "primary," "auxiliary," and "third," etc., does not imply any particular order, but is included to identify individual elements. Furthermore, the use of the terms "first," "second," "third," and "fourth," "primary," "auxiliary," and "third," etc., does not indicate any order or importance, but is used to distinguish one element from another. Note that the terms "first," "second," "third," and "fourth," "primary," "auxiliary," and "third," etc., are used here and elsewhere for labelling purposes only and are not intended to indicate any particular spatial or temporal order. Moreover, the labeling of a first element does not imply the existence of a second element, and vice versa.

[0112] It is understood that the accompanying drawings include some circuits or operations shown using solid lines and some circuits or operations shown using dashed lines. The circuits or operations included in the solid lines are those included in the most widely applicable exemplary embodiments. The circuits or operations included in the dashed lines are exemplary embodiments that may be included in or part of the circuits or operations of the solid-lined exemplary embodiments, or are additional circuits or operations that may be taken besides those of the solid-lined exemplary embodiments. It should be understood that these operations do not need to be performed in the order presented. Furthermore, it should be understood that not all operations need to be performed. The exemplary operations can be performed in any order and in any combination.

[0113] Note that the word "includes" does not necessarily exclude the presence of other elements or steps besides those listed.

[0114] Note that the word "a" or "one" preceding the element does not preclude the existence of multiple such elements.

[0115] Note that the term "indicator" can be considered as "associated with," "related to," "describe," "represent," and / or "define." The terms "indicator," "associated with," "related to," "describe," "represent," and "define" are used interchangeably. The term "indicator" can be considered as indicating a relationship. For example, weight data indicating weights may include one or more weight parameters.

[0116] Note that the word "based on" can be considered as "according to" and / or "derived from". The terms "based on" and "according to" are used interchangeably. For example, a parameter determined "based on" a dataset can be considered as a parameter determined "according to" a dataset. In other words, the parameter can be the output of one or more functions that take the dataset as input.

[0117] Functions can represent the relationship between inputs and outputs, such as mathematical relationships, database relationships, hardware relationships, logical relationships, and / or other suitable relationships.

[0118] It should also be noted that any reference numerals do not limit the scope of the claims, the exemplary embodiments can be implemented at least in part by means of a combination of hardware and software, and multiple “means”, “units” or “apparatus” can be represented by the same hardware item.

[0119] The various example methods, apparatuses, nodes, and systems described herein are described in the general context of method steps or processes that can be implemented by a computer program product, which is contained in a computer-readable medium and includes computer-executable instructions (such as program code) executed by a computer in a networked environment. The computer-readable medium can include removable and non-removable storage devices, including but not limited to read-only memory (ROM), random access memory (RAM), optical disc (CD), digital versatile optical disc (DVD), etc. Typically, program circuitry can include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. The computer-executable instructions, associated data structures, and program circuitry represent examples of program code for performing the method steps disclosed herein. A particular sequence of such executable instructions or associated data structures represents examples of corresponding actions for implementing the functions described in such steps or processes.

[0120] Although features have been shown and described, it will be understood that these features are not intended to limit the claimed disclosure, and it will be apparent to those skilled in the art that various changes and modifications can be made without departing from the scope of the claimed disclosure. Therefore, the specification and drawings are to be regarded as illustrative rather than restrictive. The claimed disclosure is intended to cover all substitutions, modifications, and equivalents.

Claims

1. A video conferencing device, comprising a memory, an interface, and one or more processors, said one or more processors including a first neural network configured to operate according to a first machine learning model, wherein, The one or more processors are configured to: Obtain audio data; The first neural network is used to process the audio data to provide a first model output that includes a direction of arrival estimate; Obtain camera data; The camera data is processed to provide video output; The uncertainty parameters are determined based on the output of the first model. and The processing of the camera data is controlled based on the uncertainty parameters to provide the video output.

2. The video conferencing device according to claim 1, wherein, The processing of the camera data includes: identifying one or more frames of one or more active participants in a video conference based on the direction of arrival estimation and the uncertainty parameter.

3. The video conferencing apparatus according to any one of the preceding claims, wherein, Controlling the processing of the camera data includes adjusting the phase pattern based on the uncertainty parameter and the direction of arrival estimation.

4. The video conferencing apparatus according to any one of the preceding claims, wherein, The uncertainty parameter indicates the uncertainty of the predicted direction of arrival and / or is a measure of the confidence level of the direction of arrival estimate.

5. The video conferencing apparatus according to any one of the preceding claims, wherein, The first model output includes: the arrival direction estimate associated with the presence estimate of one or more participants in the video conference.

6. The video conferencing apparatus according to any one of the preceding claims, wherein, The one or more processors are configured to control one or more cameras based on the uncertainty parameters and the direction of arrival estimation.

7. The video conferencing apparatus according to any one of the preceding claims, wherein, The one or more processors are configured to: Determine whether the uncertainty parameter meets the first criterion, and If the uncertainty parameter satisfies the first criterion, the camera data is processed according to the first processing scheme to provide the video output.

8. The video conferencing device according to claim 7, wherein, The one or more processors are configured to process the camera data according to a second processing scheme to provide the video output, based on whether the uncertainty parameter does not meet the first criterion and / or whether the uncertainty parameter meets the second criterion.

9. The video conferencing device according to claim 8, wherein, The one or more processors are configured to process the camera data according to a third processing scheme to provide the video output, based on the uncertainty parameter not meeting the first criterion and the uncertainty parameter not meeting the second criterion.

10. The video conferencing apparatus according to any one of the preceding claims, wherein, The first neural network is configured to output the uncertainty parameter based on a loss function that includes the variance of the arrival direction estimate.

11. The video conferencing apparatus according to any one of the preceding claims, wherein, The first neural network is configured as follows: Uncertainty prediction is performed by outputting the estimated direction of arrival and the associated variance parameter; and The loss function is based on the observation noise in the arrival direction estimation.

12. The video conferencing apparatus according to claim 11, wherein, The loss function based on the observation noise in the direction of arrival includes taking into account the variation of the observation noise in the direction of arrival.

13. The video conferencing apparatus according to any one of the preceding claims, wherein, The first model has one or more nodes configured to be deactivated, to be deactivated when processing the audio data to provide the direction of arrival estimate.

14. A video conferencing system comprising a camera, a plurality of microphones, a memory, an interface, and one or more processors, said one or more processors including a first neural network configured to operate according to a first machine learning model, wherein, The one or more processors are configured to: Obtain audio data; The first neural network is used to process the audio data to provide a first model output that includes a direction of arrival estimate; Obtain camera data; The camera data is processed to provide video output; The uncertainty parameters are determined based on the output of the first model. and The processing of the camera data is controlled based on the uncertainty parameters to provide the video output.

15. A method of operating a video conferencing apparatus, the video conferencing apparatus including a memory, an interface, and one or more processors, the one or more processors including a first neural network configured to operate according to a first machine learning model, the method comprising: Obtain audio data; The first neural network is used to process the audio data to provide a first model output that includes a direction of arrival estimate; Obtain camera data; The camera data is processed to provide video output; The uncertainty parameters are determined based on the output of the first model. as well as The processing of the camera data is controlled based on the uncertainty parameters to provide the video output.

Images