Conference system and method for speaker tracking and camera positioning

Abstract

Conference systems and methods configured to generate speaker coordinates to guide cameras towards speaker locations in an environment, and speaker tracking using a plurality of microphones and a plurality of cameras are disclosed. A method includes determining, using a first microphone array (104) and based on audio associated with a speaker (102), a first speaker position (P1) in a first coordinate system relative to the first microphone array; determining, using a second microphone array (104) and based on the audio associated with the speaker (102), a second speaker position (P2) in a second coordinate system relative to the second microphone array; determining an estimated speaker position (P3) in a third coordinate system relative to a camera (106) based on the first speaker position and the second speaker position; and transmitting the estimated speaker position in the third coordinate system to the camera to cause the camera to point the camera to the estimated speaker position.

Description

[0001] Cross-references

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 367,438, filed on June 30, 2022, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates generally to speaker tracking and camera positioning in a conference environment, and more particularly to conferencing systems and methods for positioning a camera toward a speaker location determined using one or more microphones and / or one or more cameras. Background Art

[0004] Meeting environments (e.g., conference rooms, boardrooms, video conferencing environments, etc.) typically involve the use of microphones (including microphone arrays) to capture sound from various audio sources in the environment (also referred to as the "near end"), and the use of speakers to present audio from remote locations (also referred to as the "far end"). For example, people in a conference room may be on a conference call with people at a remote location. Typically, speech and sound from the conference room may be captured by the microphones and transmitted to the remote location, while speech and sound from the remote location may be received and played on the speakers in the conference room. Multiple microphones may be used to optimally capture speech and sound in the conference room.

[0005] Such a conference environment may also include one or more image capture devices, such as cameras, which may be used to capture and provide images and video of people and objects in the environment to be transmitted for viewing at a remote location. However, for example, if the camera is configured to show the entire room, or if the camera is fixed to a specific preconfigured portion of the room and speakers move in and out of that portion during a meeting or event, it may be difficult for a viewer at the remote location to see a particular speaker. Speakers may include, for example, people in the environment who are speaking or making other sounds.

[0006] Additionally, in environments where multiple cameras and / or multiple microphones (or microphone arrays) are desired to achieve adequate video and audio coverage, it may be difficult to accurately identify a unique speaker in the environment and / or identify which camera and / or microphone should be oriented toward the speaker. Furthermore, in some environments where there are multiple cameras and / or multiple microphones, the relative positions of the cameras and microphones may not be known or predefined. In such environments, it may be difficult to accurately associate camera angles with speaker positions. Although professional installers or integrators can manually configure camera zones or presets based on position information from microphone arrays, this is typically a time-consuming, laborious, and inflexible process. For example, if the seating arrangement in the room is changed after the initial setup of the conference system, the preconfigured camera zones may not adequately cover the participants, and such zones may be difficult to modify after they are set up, and / or may only be modified by professional installers or integrators. Summary of the invention

[0007] The disclosed techniques provide systems and methods designed to, among other things, (1) determine coordinates for positioning a camera toward a speaker based on a speaker location identified by using two or more microphones (or microphone arrays); (2) adjust a microphone lobe position or other audio pickup coverage area based on the speaker location identified by using a camera; and (3) select a camera from a plurality of cameras for steering toward a speaker based on the position of a microphone lobe or other audio beam directed toward the speaker, the lobe position being selected based on the speaker location identified by using two or more microphones.

[0008] In an embodiment, a method performed by one or more processors in communication with each of a first microphone, a second microphone, and a camera includes: using a first microphone array and determining a first speaker position in a first coordinate system relative to the first microphone array based on audio associated with the speaker; using a second microphone array and determining a second speaker position in a second coordinate system relative to the second microphone array based on the audio associated with the speaker; using at least one processor and determining an estimated speaker position in a third coordinate system relative to the camera based on the first speaker position and the second speaker position; and transmitting the estimated speaker position in the third coordinate system to the camera so that the camera points the image capture component of the camera toward the estimated speaker position.

[0009] In another embodiment, a system includes: a first microphone array, which is configured to determine a first speaker position in a first coordinate system relative to the first microphone array based on audio associated with the speaker; a second microphone array, which is configured to determine a second speaker position in a second coordinate system relative to the second microphone array based on the audio associated with the speaker; a camera, which includes an image capture component; and one or more processors, which are communicatively coupled to each of the first microphone array, the second microphone array and the camera, the one or more processors configured to determine an estimated speaker position in a third coordinate system relative to the camera based on the first speaker position and the second speaker position, and transmit the estimated speaker position in the third coordinate system to the camera, wherein the camera is configured to point the image capture component at the estimated speaker position received from the one or more processors.

[0010] In a further embodiment, a non-transitory computer-readable storage medium includes instructions that, when executed by one or more processors in communication with each of a first microphone array, a second microphone array, and a camera, cause the one or more processors to: determine a first speaker position in a first coordinate system relative to the first microphone array using the first microphone array and based on audio associated with the speaker; determine a second speaker position in a second coordinate system relative to the second microphone array using the second microphone array and based on the audio associated with the speaker; determine an estimated speaker position in a third coordinate system relative to the camera based on the first speaker position and the second speaker position; and transmit the estimated speaker position in the third coordinate system to the camera so that the camera points the camera's image capture component toward the estimated speaker position.

[0011] In another embodiment, a method performed by one or more processors in communication with each of a first microphone, a second microphone, and a camera includes: determining a first speaker position of the microphone array in a first coordinate system relative to the first microphone array based on audio associated with a speaker using a microphone array; converting the first speaker position from the first coordinate system to a second coordinate system relative to a camera using at least one processor; transmitting the first speaker position in the second coordinate system to the camera so that the camera points an image capture component of the camera at the first speaker position; receiving from the camera a second speaker position in the second coordinate system identified by the camera using a speaker detection component of the camera; and adjusting a lobe position of the microphone array based on the second speaker position received from the camera using the microphone array.

[0012] According to some aspects, adjusting the lobe position includes adjusting the distance coordinate of the lobe position in the second coordinate system based on the distance coordinate of the second talker position in the second coordinate system; and converting the adjusted lobe position from the second coordinate system to the first coordinate system.

[0013] According to some aspects, adjusting the lobe position includes converting the second speaker position from the second coordinate system to the first coordinate system; and using the at least one processor to adjust the distance coordinate of the lobe position in the first coordinate system based on the distance coordinate of the second speaker position in the first coordinate system.

[0014] In accordance with some aspects, determining the first talker location includes determining a location of a sound generated proximate to the microphone array using an audio localization algorithm executed by an audio activity locator.

[0015] In a further embodiment, a method performed by one or more processors communicating with each of a plurality of microphone arrays and a plurality of cameras includes: determining a speaker position in a first coordinate system relative to a first microphone array among the plurality of microphone arrays using the plurality of microphone arrays and based on audio associated with the speaker; selecting a lobe position of a selected one of the plurality of microphone arrays in the first coordinate system based on the speaker position in the first coordinate system; selecting a first camera among the plurality of cameras based on the lobe position; converting the lobe position to a second coordinate system relative to the first camera; and transmitting the lobe position in the second coordinate system to the first camera so that the first camera directs an image capture component of the first camera toward the lobe position.

[0016] According to some aspects, selecting the lobe position includes: determining a distance between the talker position and each of the multiple microphone arrays in the first coordinate system; and identifying the selected one of the multiple microphone arrays as being closest to the talker position in the first coordinate system.

[0017] According to some aspects, selecting the first camera includes: converting the lobe position from the first coordinate system to a common coordinate system; identifying a first area of ​​a plurality of areas in the common coordinate system as including the lobe position in the common coordinate system, each area being assigned to one or more of the plurality of cameras; and identifying the first camera as being assigned to the first area.

[0018] According to some aspects, determining the talker location includes determining a location of a sound generated proximate the plurality of microphone arrays using an audio localization algorithm executed by an audio activity locator.

[0019] These and other embodiments and various arrangements and aspects will become apparent and more fully understood from the following detailed description and accompanying drawings, which set forth illustrative embodiments that are indicative of the various ways in which the principles of the invention may be employed. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 and 2 is a schematic diagram of an exemplary environment including a conferencing system that can be used to determine the location of a speaker using speaker coordinates obtained by multiple microphones according to one or more embodiments.

[0021] Figure 3 is a block diagram of an exemplary microphone array configured to detect audio activity in an environment and to communicate with an Figure 1 and 2 conference system.

[0022] Figure 4 is a block diagram of an exemplary conferencing system with multiple microphones and / or multiple cameras, which may be used as any of the conferencing systems described herein, according to one or more embodiments.

[0023] Figure 5 is a flow diagram illustrating exemplary operations for determining an estimated speaker location in an environment using speaker coordinates obtained by multiple microphones in accordance with one or more embodiments.

[0024] Figure 6 is a schematic diagram of an exemplary environment including a conferencing system that can be used to refine a microphone's lobe position based on a talker's position determined by a camera, according to one or more embodiments.

[0025] Figure 7 is a flow diagram illustrating exemplary operations for refining a microphone's lobe position based on a talker's position determined by a camera in accordance with one or more embodiments.

[0026] Figures 8 to 10 is a schematic diagram of an exemplary environment including a conferencing system that can be used to determine camera positioning of one or more speakers based on lobe positions selected based on speaker coordinates obtained from one or more microphones in accordance with one or more embodiments.

[0027] Fig.11 is a flow diagram illustrating operations for determining camera positioning in an environment based on lobe positions selected based on talker coordinates obtained by one or more microphones in accordance with one or more embodiments.

[0028] Fig.12is a schematic diagram of an exemplary environment including a conferencing system that may be used to determine camera positioning of a talker in one area of ​​the environment using lobe positions selected based on talker coordinates obtained from microphones in the same area and another adjacent area, according to one or more embodiments. DETAILED DESCRIPTION

[0029] The systems and methods described herein can improve the configuration and use of conference systems by using audio localization information collected by multiple microphones (or microphone arrays) to position the camera toward an active speaker or other audio source in the environment. For example, each microphone can use an audio localization algorithm to detect the position of a speaker in the environment and provide the detected speaker position or corresponding audio localization coordinates to a common aggregator. Typically, the audio localization information obtained by the microphone is relatively accurate with respect to the azimuth and elevation coordinates, but is less accurate for the radius coordinate or the distance between the audio source and the microphone array. In an embodiment, the aggregator can improve the accuracy of the radius or distance information by aggregating or combining the time-synchronized audio localization coordinates obtained by multiple microphones for the same audio source (or audio event) to determine the estimated speaker position. The estimated speaker position can be provided to the camera for positioning the camera's image capture component toward the speaker. Prior to the transmission, the aggregator can convert the coordinates of the estimated speaker position to a coordinate system relative to the camera, or to a previously determined common coordinate system (e.g., a coordinate system relative to the room), so that the camera receives the estimated speaker position in a format that the camera can understand and use. The camera can utilize the received speaker position for moving, zooming, panning, framing, or otherwise adjusting the images and videos captured by the camera. In this way, for example, the systems and methods described herein can be used by a conferencing system to enable a camera to more accurately capture images and / or videos of active speakers.

[0030] The systems and methods described herein may also improve the configuration and use of conference systems by using a speaker detection component of a camera to obtain a speaker location, which can be used to direct an audio beam or lobe of a microphone (or microphone array) toward an active speaker in the environment. For example, a microphone array may detect a first location of an active speaker in the environment using an audio localization algorithm and point the lobe of the microphone array in the direction of the perceived audio source or first speaker location. As mentioned above, the radius or distance coordinate in audio localization may not be as accurate as other coordinates (e.g., azimuth and elevation), so that the speaker may be located anywhere along a straight line formed from the microphone array toward the perceived audio source based on the azimuth and elevation coordinates. The systems and methods described herein may be used by a conference system to improve the distance coordinate of audio localization by pointing the image capture component of a camera at the first speaker location and utilizing a speaker detection component or other suitable image processing algorithm to identify a human face (e.g., the speaker's face) along a line where the speaker may be located. In this way, the camera may determine that the speaker is actually located at a second location near or approximately near the first speaker location. A second, more precise location can be provided to the microphone array, and the coordinates of that location can be converted to the coordinate system of the microphone array or other coordinate system recognized by the microphone array. Based on the second location, the microphone array can adjust the position of the lobe directed toward the first talker location, or otherwise direct the lobe toward the second location. Thus, for example, the systems and methods described herein can be used by a conferencing system to enable a microphone array to improve the beamforming accuracy of the array for capturing active talkers.

[0031] In addition, the systems and methods described herein can improve the configuration and use of conference systems by determining which camera and which microphone are best suited to capture the video and audio of a given speaker, respectively, in an environment with multiple cameras and multiple microphones. Typically, when there are multiple microphones (each with multiple audio beams or lobe positions) and multiple cameras, it may be difficult to identify the only speaker in the environment and which camera is pointed at the speaker. The systems and methods described herein can be used by conference systems to determine or identify the position of an active speaker based on audio source localization information obtained by two or more of the microphones. Based on the speaker position, the conference system can select the lobe and corresponding microphone that are best suited to capture the audio produced at the identified speaker position. Then, the conference system can select a camera that can optimally capture the video of the speaker (specifically the speaker's face) at the selected lobe position. In addition, the speaker position can be converted into the coordinate system of the camera or other common coordinate system and transmitted to the camera for positioning the image capture component of the camera toward the speaker position. In this way, the systems and methods described herein may be used by a conferencing system to automatically identify which microphone and / or lobe and camera should be focused on each unique speaker in an environment, i.e., without the need for manual installation or setup by one or more users.

[0032] As used herein, the terms "lobe" and "microphone lobe" refer to an audio beam generated by a given microphone array (or array microphone) to pick up audio signals at selected locations (e.g., locations toward which the lobe is directed). Although the techniques disclosed herein are described with reference to microphone lobes generated by array microphones, the same or similar techniques may be used for other forms or types of microphone coverage (e.g., cardioid patterns, etc.) and / or microphones that are not array microphones (e.g., handheld microphones, boundary microphones, lavalier microphones, etc.). Thus, the term "lobe" is intended to encompass any type of audio beam or coverage.

[0033] Figure 1 and 2 An exemplary environment 10 is depicted in which one or more of the systems and methods disclosed herein may be used. As shown, according to an embodiment, the environment 10 includes a conference system 100 that may be used to determine the location of a speaker 102 in the environment 10 for beamforming and / or camera positioning purposes. It should be understood that although Figure 1 and 2 One potential environment is described, but the systems and methods disclosed herein may be used in any applicable environment, including but not limited to conference rooms, offices, huddle rooms, theaters, arenas, music venues, etc.

[0034] As shown, the conferencing system 100 includes a plurality of microphones 104, at least one camera 106, and an aggregator 108. The system 100 may also include Figure 1 and 2 102, such as one or more speakers, desktop microphones, display screens, and / or computing devices. In embodiments, one or more of the components in system 100 may include one or more digital signal processors or other processing components, controllers, wireless receivers, wireless transceivers, etc. In addition, environment 10 may also include one or more other persons other than speaker 102 and / or other objects not shown (e.g., musical instruments, phones, tablet computers, computers, HVAC equipment, etc.). It should be understood that Figure 1 and 2 The components shown in are merely exemplary, and any number, type, and placement of various components in environment 10 are contemplated and possible.

[0035] Microphone 104 may be a microphone array (also referred to as an "array microphone") or any other type of microphone, including non-array microphones, such as directional microphones (e.g., lavalier, boundary, handheld, etc.), etc. The type of transducer (e.g., microphone and / or speaker) and its placement in a particular environment may depend on the location of the audio source, the audience, physical space requirements, aesthetics, room layout, stage layout, and / or other considerations. For example, one or more microphones may be placed on a table, podium, or other surface near the audio source, or attached to the audio source (e.g., a performer). The microphones may also be mounted overhead or on a wall to capture sound from a larger area (e.g., an entire room). Figure 1 and 2 The microphone 104 shown in the can be placed in any suitable location, including a wall, ceiling, table, and / or any other surface in the environment 10. Similarly, the speaker can be placed on a wall, ceiling, or table surface to emit sound to an audience in the environment 10, such as sound from the far end of a meeting, pre-recorded audio, streaming audio, etc. The microphone and speaker can conform to a variety of sizes, form factors, mounting options, and wiring options to suit the needs of a particular environment. In the illustrated embodiment, the microphone 104 can be placed at different locations in the environment 10 so as to adequately capture the sound in the entire environment 10.

[0036] In the case where environment 10 is a conference room, environment 10 may be used for meetings, conference calls, or other events in which local participants in the room communicate with each other and / or with remote participants. In this case, microphone 104 may detect and capture sound from audio sources within environment 10. The audio source may be a local participant, such as Figure 1 , and the sound may be a voice spoken by a local participant, or music or other sounds generated by it. In common cases, the local participant may sit on a chair at a table, but other configurations and locations of the audio source are also contemplated and possible.

[0037] The camera 106 can capture still images and / or videos of the environment 10 in which the conference system 100 is located. In some embodiments, the camera 106 can be a stand-alone camera, while in other embodiments, the camera 106 can be a component of an electronic device (e.g., a smartphone, a tablet computer, etc.). In some cases, the camera 106 can be included in the same electronic device as one or more of the aggregator 108 and the microphone 104. The camera 106 can be a pan-tilt-zoom (PTZ) camera that can be physically moved and zoomed to capture the desired images and videos, or can be a virtual PTZ camera that can digitally crop and zoom images and videos to one or more desired portions. The system 100 can also include, for example, a display such as a television or a computer monitor for showing other images and / or videos (e.g., remote participants of a meeting or other images or video content). In some embodiments, for example, the display can include one or more microphones, cameras, and / or speakers in addition to or including the microphone 104 and / or camera 106.

[0038] Additional references Figure 3 , showing an exemplary microphone array 200, which may be any of the microphones 104, and according to an embodiment may be used with Figure 1 and 2 . The microphone array 200 includes a plurality of microphone elements 202a, b, ..., zz (or microphone transducers), and may form one or more pickup patterns with lobes so that sounds from an audio source, such as sounds produced by a human speaker 102 or other objects or speakers in the environment 10, may be detected and captured. In some embodiments, each of the microphone elements 202a, b, ..., zz may be a MEMS (micro-electromechanical system) microphone with an omnidirectional pickup pattern. In other embodiments, the microphone elements 202a, b, ..., zz may have other pickup patterns and / or may be electret condenser microphones, dynamic microphones, ribbon microphones, piezoelectric microphones, and / or other types of microphones. In various embodiments, the microphone elements 202a, b, ..., zz may be arranged in one dimension or in multiple dimensions.

[0039] In some embodiments, each microphone element 202a, b, ..., zz can detect sound and convert the detected sound into an analog audio signal. In this case, other components in the microphone array 200 (such as analog / digital converters, processors and / or other components (not shown)) can process the analog audio signal and ultimately generate one or more digital audio output signals. The digital audio output signal may comply with a suitable standard and / or transmission protocol for transmitting audio. In other embodiments, each of the microphone elements 202a, b, ..., zz in the microphone array 200 can detect sound and convert the detected sound into a digital audio signal.

[0040] like Figure 3 As shown in , one or more digital audio output signals 204a, b, ..., z may be generated to correspond to each of one or more pickup patterns. The pickup pattern may consist of or include one or more lobes (e.g., main lobe, side lobe, and back lobe) and / or one or more nulls. The pickup pattern that may be formed by the microphone array 200 may depend on the type of beamformer (e.g., beamformer 206) used with the microphone elements. For example, a delay and sum beamformer may form a frequency-dependent pickup pattern based on its filter structure and the layout geometry of the microphone elements. As another example, a differential beamformer may form a cardioid, subcardioid, supercardioid, hypercardioid, or bidirectional pickup pattern. Other suitable types of beamformers may include a minimum variance distortionless response ("MVDR") beamformer, etc. In an embodiment, the beamformer 206 may communicate with the microphone elements 202a, b, ..., zz in wired or wireless communication. In some embodiments, the beamformer 206 may be a separate device in communication with the microphone array 200 .

[0041] The microphone array 200 may also determine the position of a speaker or other object in the environment 10 relative to the array 200, or more specifically, the coordinate system of the array 200, based on the sound or audio activity detected by the microphone elements 202a, b, ..., zz. For example, the microphone array 200 may include an audio activity locator 208 that communicates with the microphone elements 202a, b, ..., zz by wire or wirelessly. The audio activity locator 208 may determine or identify a speaker or other object in the environment (e.g., Figure 1The audio activity locator 208 may determine the position (position / location) of the audio activity detected in the environment 10 of the microphone array 200, or otherwise locate the audio detected by the microphone array 200. In an embodiment, the audio activity locator 208 may utilize a controlled response power phase transform (SRP-PHAT) algorithm, a generalized cross-correlation phase transform (GCC-PHAT) algorithm, an arrival time (TOA)-based algorithm, an arrival time difference (TDOA)-based algorithm, a multiple signal classification (MUSIC) algorithm, an artificial intelligence-based algorithm, a machine learning-based algorithm, or other suitable audio or sound source localization algorithm to determine the direction of arrival of the detected audio activity and generate an audio localization or other data representing the position (location / position) of the detected sound relative to the microphone array 200. The detected audio activity may include an audio source, such as a human speaker (e.g., Figure 1 ), or from a camera (e.g., Figure 1 As will be appreciated, the location obtained by the sound source localization algorithm may represent a perceived location of the audio activity or other estimate based on the audio signals received from the microphone elements 202a, b, ..., zz, which may or may not coincide with the actual or true location of the audio activity.

[0042] The audio activity locator 208 may be configured to indicate the location of the detected audio activity as a set of three-dimensional coordinates relative to the location of the microphone array 200, or as a set of three-dimensional coordinates in a coordinate system in which the microphone array 200 is the origin of the coordinate system. The coordinates may be Cartesian coordinates (i.e., x, y, z) or spherical coordinates (i.e., azimuth). (phi or "az"), elevation angle θ (theta or "elev"), radial distance / magnitude (R)). It should be noted that Cartesian coordinates can be easily converted to spherical coordinates and vice versa as needed. Spherical coordinates can be used in various embodiments to determine the Figure 1 and 2 The conference system 100 may also include additional information about the conference system 100, such as the distance between the audio source 102 and a given microphone 104, the distance between two microphones 104, the distance between a given microphone 104 and the camera 106, and / or the relative positions of the camera 106 and / or the microphone 104 within the environment 10.

[0043] In the illustrated embodiment, the audio activity locator 208 is included in the microphone array 200. In other embodiments, the audio activity locator 208 may be included in another component of the conferencing system 100, or may be a separate component. In various embodiments, the detected talker locations, or more specifically, location coordinates representing each location, may be transmitted to one or more other components of the conferencing system, such as Figure 1 aggregator 108 and / or camera 106.

[0044] In various embodiments, the location data generated by the audio activity locator 208 also includes a timestamp or other timing information to indicate the time when the coordinates were generated, the order in which the coordinates were generated, and / or any other information to help identify coordinates that were generated simultaneously or nearly simultaneously for the same audio source. In some embodiments, the microphones 104 in the conference system 100 have synchronized clocks (e.g., using the Network Time Protocol, etc.). In other embodiments, other techniques may be used to determine the timing or simultaneous output of coordinates, such as setting up a time-synchronized data channel to transmit the location coordinates from the microphones 104 to the aggregator 108, etc.

[0045] like Figure 3 , the microphone array 200 may also include a lobe selector 210 that communicates wired or wirelessly with the audio activity locator 208. The microphone array 200 may be capable of forming one or more pickup patterns in which lobes may be directed to sense audio in a particular location within an environment. The audio activity locator 208 may send location data including a speaker location or audio localization coordinates obtained by the locator 208 for a detected speaker to the lobe selector 210 for selecting a lobe that can optimally capture the speaker location. For example, the lobe selector 210 may determine which of the multiple lobes of the microphone array 200 is best suited for picking up sound or capturing audio at the coordinates determined by the audio activity locator 208 based on distance (e.g., the distance between the microphone array 200 and the speaker location), availability of lobes, coverage areas assigned to each lobe, and / or any other suitable factors. The location of the selected lobe may be provided to the beamformer 206 for deploying the audio pickup beam toward the indicated lobe location. The lobe position may be a set of coordinates in a coordinate system relative to the microphone array 200, or in any other suitable format. In various embodiments, the position data including the selected lobe position may also be transmitted to one or more other components of the conferencing system, such as Figure 1 and 2 aggregator 108 and / or camera 106 in.

[0046] Return to reference Figure 1 and 2, the aggregator 108 may be configured to receive time-synchronized talker positions and / or lobe positions from one or more of the microphones 104, and based thereon provide coordinates or other positional information to the camera 106 to position the camera 106 toward the talker 102. In some embodiments, the positional data (or positioning coordinates) received at the aggregator 108 may be relative to the coordinate system of the microphone 104 that generated the data. In this case, if the relative positions and orientations of the microphones 104 and / or the camera 106 in the environment 10 are known, a coordinate system transformation may be derived based thereon to convert the positional data generated by the microphone 104 into the coordinate system of the receiving component (e.g., the camera 106) or other common coordinate system of the environment 10.

[0047] In various embodiments, the aggregator 108 may include a conversion unit (not shown) configured to convert the position data from its original coordinate system (e.g., relative to the microphones 104) to another coordinate system readily used by the camera 106 before transmitting the position data to the camera 106. For example, the conversion unit may be configured to convert positioning coordinates in a first coordinate system relative to a first microphone array of the plurality of microphones 104 (e.g., where the first microphone array 104 is the origin of the first coordinate system) to positioning coordinates in a second coordinate system relative to the camera 106 (e.g., where the camera 106 is the origin of the second coordinate system). In other embodiments, the conversion unit may be included in each of the microphones 104, so that the positioning coordinates generated by each microphone 104 can be converted to the coordinate system of the intended recipient (e.g., the camera 106) before being transmitted to the aggregator 108.

[0048] In some cases, the conversion unit may be configured to convert the location data received at the aggregator 108 into a common coordinate system associated with the environment 10, e.g., a coordinate system relative to the room in which the conferencing system 100 is located, so that the location data can be readily used by any component of the conferencing system 100. In such an embodiment, each component of the conferencing system 100 (e.g., microphone 104 and camera 106) may also include a conversion unit for converting any received location data (or coordinates) into the coordinate system of that component for ease of processing and use.

[0049] In some cases, a conversion unit included in aggregator 108 may convert the position data received from camera 106 into another coordinate system of environment 10 before transmitting the position data to another component of system 100. For example, a speaker's position in a coordinate system relative to camera 106 may be converted to a speaker's position in a coordinate system relative to one of microphones 104 and / or a common coordinate system of environment 10.

[0050] In various cases, the aggregator 108 may combine the time-synchronized positioning from two different microphones 104 to obtain a more accurate estimate of the speaker's position, as described herein. In this case, where the relative positions and orientations of the microphones 104 in the environment 10 are known, a coordinate system transformation may be derived based thereon to convert the position data generated by the first microphone array in the plurality of microphones 104 to the coordinate system of the second microphone array in the plurality of microphones 104, or other common coordinate system of the environment 10. For example, the aggregator 108 may convert the position data in the coordinate system (e.g., x, y, z) of the first microphone 104 to the position data in the coordinate system (e.g., x', y', z') of the second microphone 104 before combining the position coordinates.

[0051] In some embodiments, the conversion unit (whether located in the aggregator 108 or in another component of the system 100) may also be configured to convert the lobe positions received from a given microphone 104 to another coordinate system of the environment 10 before transmitting the lobe positions to the camera 106 or other components of the system 100. For example, the lobe positions in the coordinate system relative to the first microphone 104 may be converted to lobe positions in the coordinate system relative to the camera 106.

[0052] Reference now Figure 1 and 2 During operation, a first one of the microphones 104 may send a first speaker position (e.g., x1, y1, z1) representing the localization of a sound produced by a given audio source (e.g., speaker 102) at a first point in time as perceived or detected by the first microphone 104 to the aggregator 108. Similarly, a second one of the microphones 104 may send a second speaker position (e.g., x2, y2, z2) corresponding to the same audio event as perceived by the second microphone 104 (i.e., representing the localization of the same sound at approximately the same point in time) to the aggregator 108. As will be appreciated, in some cases, the two microphones 104 may localize the speaker 102 to the same location (location / position) in the environment 10, while in other cases, the microphones 104 may localize the speaker 102 to different locations in the environment 10, such as Figure 1 and 2. In the latter case, aggregator 108 may combine the time-synchronized position fixes from multiple microphones 104 to obtain (or triangulate) an estimated speaker position with higher accuracy than the individual speaker positions. Aggregator 108 may use one or more different techniques to calculate the estimated speaker position depending on how close the two position fixes are to each other, the degree of overlap between them, and / or other relevant considerations. Thus, aggregator 108 can provide a more accurate position of speaker 102 to camera 106 and minimize or avoid erroneous camera positions.

[0053] More specifically, in Figure 1 , the localization of the sound produced by the speaker 102 and detected by the first microphone 104 is determined by the distance from the first microphone 104 to the first speaker position (e.g., Figure 1 Similarly, the localization of the same sound as detected by the second microphone 104 is represented by the line L1 drawn from the second microphone 104 to the second talker position (e.g., Figure 1 A second line L2 drawn from point P2 in the environment 10 indicates that the second speaker location is different from the first speaker location. For example, the first speaker location (az1,elev1,R1) and the second speaker location (az2,elev2,R2) may have different radius or distance coordinates ("R"), but have the same or similar azimuth ("az") and elevation ("elev") coordinates. Therefore, the two lines L1 and L2 may end at different locations in the environment 10, and both locations may not be the true or actual location of the speaker 102. In contrast, the true location of the speaker 102 (e.g., Figure 1 Point P3) in may be located at different radial coordinates along each of lines L1 and L2.

[0054] It should be understood that Figure 1 and 2 The straight lines shown in are intended to represent detected talker locations in three-dimensional space and may be constructed based on coordinates received at the aggregator 108. For example, each audio localization n may be represented using a straight line starting from the microphone 104 that produced the localization (i.e., coordinates (az,elev,R)) and extending into three-dimensional space at an angle specified by the azimuth and elevation coordinates of the localization, the line ending at a point indicated by the radius coordinate of the same localization. The aggregator 108 may calculate each line L(n) using the following equation: L(n) = P(n) + t(n) * V(n), where P(n) is a point on the line L(n) and V(n) is a direction vector of the line L(n). Prior to such calculations, the aggregator 108 may first convert the received coordinates into a common coordinate system (e.g., a coordinate system of one of the microphones 104 or the environment 10, etc.) and / or spherical coordinates, as desired.

[0055] While talker locations with imprecise radial coordinates can still be used relatively successfully to direct the microphone lobe toward the approximate location of the active audio source, camera positioning requires greater accuracy. Figure 1 , directing camera 106 toward point P1 or point P2 will not properly capture the face of speaker 102, which may be located at point P3. As another example, in some cases, camera 106 may be configured to follow the active speaker as the active speaker moves around the room (including as the active speaker moves from a sitting or standing position or vice versa) so that the active speaker is always within the frame or field of view of camera 106.

[0056] In an embodiment, the aggregator 108 is configured to improve the audio localization accuracy of the conference system 100 by determining an estimated position of a talker 102 in the environment 10 based on two or more time-synchronized localizations (or position coordinates) generated by two or more different microphones 104 for the same audio activity or event. Various linear algebra techniques may be used to calculate or determine the estimated talker position, such as Figure 1 and 2 , and described herein. In addition to or in lieu of aggregating the audio localization obtained by microphone 104, other techniques for estimating the true speaker position may be used. For example, in some cases, aggregator 108 may be configured to improve the quality of localization data used for camera localization by using voice activity detection techniques to prevent localization of noise in environment 10 and / or transmitting only those coordinates that are in the predetermined vicinity of the current lobe position and / or within a predefined audio coverage area of ​​environment 10. As another example, aggregator 108 may be configured to improve localization quality by filtering out (or not transmitting) continuous coordinates of nearby positions or other minor changes due to autofocus activity of a particular lobe to minimize jitter of camera 106 when an active speaker moves his head or mouth. In this case, a significant movement or significant change in the detected speaker position may be required before camera 106 moves to a new position, and before that occurs, a constant value may be provided to camera 106 to prevent or minimize jitter. As another example, the aggregator 108 may be configured to improve positioning quality by transmitting the coordinates of multiple positions at a predefined rate at one time, and organize them according to lobe proximity, coverage area affiliation, priority (e.g., meeting leader, speaker, company CEO, etc.), and / or other relevant factors.

[0057] First reference Figure 1In some embodiments, the aggregator 108 may determine the estimated speaker location by identifying a common point (e.g., the intersection point of the first line L1 and the second line L2) based on the first and second speaker locations. For example, the aggregator 108 may determine the intersection point P3 of the lines L1 and L2 by equating the line equations (e.g., P1+t1*V1=P2+t2*V2) and solving for t1, t2 (where t1=t2). Other techniques may also be used to combine or aggregate the first speaker location and the second speaker location to identify a common point, as will be appreciated.

[0058] In other embodiments, for example, Figure 2 , aggregator 108 may determine an estimated speaker location by identifying a point based on the nearest or optimal point of the detected speaker location, or the minimum distance between the areas defined by the azimuth and elevation coordinates of the speaker location detected by microphone 104. This technique is particularly useful in situations where the lines representing the detected speaker locations are skewed and therefore do not intersect due to azimuth and / or elevation inaccuracies and / or other mismatches detected by microphone 104. For example, in Figure 2 , a third line L3 constructed based on the third speaker position (az3, elev3, R3) detected by the first microphone 104 does not intersect a fourth line L4 constructed based on the fourth speaker position (az4, elev4, R4) detected by the second microphone 104. In addition, the third line L3 ends at point P4, and the fourth line L4 ends at point P5, neither of which corresponds to the actual speaker position (e.g., point P3).

[0059] In this case, aggregator 108 may determine an estimated speaker location based on the detected speaker location by locating the point closest to both lines L3 and L4, or otherwise finding the point of minimum distance (or error) taking into account the uncertainty in the radius coordinate R. For example, aggregator 108 may be configured to determine an estimated speaker location based on the detected speaker location by finding the point on line L3 that is closest to line L4 (e.g., Figure 2 4) and the point on line L4 closest to line L3 (e.g., point P5), and draw a new line segment L5 connecting the two closest points between lines L3 and L4 to resolve the inaccurate radius coordinate. Line segment L5 may be perpendicular to both lines L3 and L4. Aggregator 108 may calculate line segment L5 using the equation L5=P4+t4*V4+t6*V6, where V6=V2*V1 and is a vector perpendicular to both P4 and P5. Given that line segment L5 should contain a point such that P4+t4*V4+t6*V6=P5+t5*V5, aggregator 108 may solve for t4, t5, t6, where t4=t5=t6, and determine the unknown value. Once line segment L5 is drawn, aggregator 108 places the midpoint of line segment L5 (e.g., Figure 2The point Pm in the figure is identified as the estimated speaker position, or the point on the line segment L5 that is most likely the location of the audio source is identified as the estimated speaker position. In various embodiments, before calculating the estimated speaker position, the audio localization received at the aggregator 108 can be converted to a common coordinate system (e.g., the coordinate system of the first microphone 104 or the environment 10, etc.) and / or converted from Cartesian coordinates (x, y, z) to spherical coordinates (az, elev, R) as needed.

[0060] In some cases, if the coordinates of the audio localization, including the azimuth and / or elevation, are less precise or contain slight inaccuracies, the audio localization obtained by a given microphone 104 (or microphone array) may represent a three-dimensional area or region in the environment 10, rather than a straight line. For example, the audio localization may be directed to a wider area (area / region), such as a rectangular, cylindrical, or conical area that includes (or is centered on) the corresponding straight line L(n). In various embodiments, the techniques described herein may also be used to calculate an estimated speaker position based on such audio localization (or "local area"). For example, the aggregator 108 may be configured to identify intersection points of local areas, which are three-dimensional areas defined by the coordinates of corresponding time-synchronized audio localizations and their deviations. In some embodiments, the aggregator 108 may provide intersection points or overlapping areas as estimated speaker positions. In other embodiments, the aggregator 108 may further use one or more of the techniques described herein to identify points within overlapping areas and provide the identified points as estimated speaker positions. For example, the estimated speaker position may be the center point of the overlapping region, or the point in the overlapping region that is closest to the two local regions (eg, the closest point).

[0061] Figure 4 Depicts a system 300 (eg, a conferencing system) according to an embodiment that can be used as Figure 1 and 2 The conference system 100 shown in the environment 10 of , or any of the other conference systems described herein (e.g., Figures 8 to 10 The system 300 may include a plurality of microphones 302a, ..., z (e.g., Figure 1 A microphone 104 can detect and capture audio sources from within the environment (e.g., Figure 1 The microphones 302a, ...z can also detect or determine the environment (e.g., Figure 1 The system 300 may also include an aggregator unit 304 (e.g., Figure 1The system 300 may further include one or more cameras 306a, ..., z (e.g., Figure 1 , which can capture the environment 10 (including active speakers in the environment (e.g., Figure 1 The aggregator unit 304 may provide an image and / or video of a speaker 102) of the active speaker, and may be controlled by a camera controller 308 of the system 300. The aggregator unit 304 may provide the estimated speaker position to the camera controller 308 for positioning at least one camera 306a, ..., z toward the estimated speaker position. For example, the camera controller 308 may provide appropriate signals to at least one camera 306a, ..., z to move and / or zoom the camera 306a, ..., z toward the active speaker. In some embodiments, the camera controller 308 and / or the cameras 306a, ..., z may be configured to move with the active speaker as the active speaker moves around in the environment (or room).

[0062] In some embodiments, one of the microphones 302a, ..., z may function as the aggregator unit 304. In other embodiments, the aggregator unit 304 and the camera controller 308 may be included in the same device (e.g., a computing device). In still other embodiments, the camera controller 308 and one of the cameras 306a, ..., z may be integrated together. The components of the system 300 may communicate wired and / or wirelessly with each other and / or with other components of the system 100.

[0063] For example, each of microphones 302a, ..., z can detect sounds in environment 10 and determine the location of the sounds in a coordinate system relative to microphone 302a, ..., z itself or taking a given microphone 302a, ..., z as its origin. Figure 1 The microphones 302a, ..., z may transmit the position of the detected sound in its respective coordinate system (or "detected speaker position") to the aggregator unit 304. In some cases, one or more of the microphones 302a, ..., z may transmit the position of the lobe deployed by the one or more microphones 302a, ..., z in its respective coordinate system to the aggregator unit 304. For example, the lobe may be deployed based on the detected speaker position.

[0064] Thus, the aggregator unit 304 may receive detected speaker locations and / or lobe locations of the microphones 302a, ..., z from each microphone 302a, ..., z. Each location received by the aggregator unit 304 may be in a respective coordinate system of the microphone 302a, ..., z that provided the location. Thus, the aggregator unit 304 may convert the received locations into a common coordinate system that may be readily used by the aggregator unit 304 to perform one or more calculations and / or may be readily used by one or more of the cameras 306a, ..., z for camera positioning. The common coordinate system may be a coordinate system relative to a given camera 306a, ..., z, a coordinate system of a room or environment in which the camera 306a, ..., z is located, or a coordinate system of another component of the system 300 (e.g., aggregator unit 304, one of the microphones 302a, ..., z, etc.). In other embodiments, converting the positions to a common coordinate system may be performed by another component included in or in communication with system 300 (e.g., a computing device (not shown), a remote computing device (e.g., a cloud-based device), and / or any other suitable device).

[0065] For example, as described herein, the aggregator unit 304 may calculate or determine an estimated speaker position based on time-synchronized detected speaker positions received from two or more microphones 302a, ..., z. In this case, the aggregator unit 304 may convert the detected speaker positions received from the individual microphones 302a, ..., z to a first common coordinate system associated with one of the microphones 302a, ..., z (e.g., the coordinate system of the first microphone 302a), and may calculate the estimated speaker position in the first common coordinate system. Before transmitting the estimated speaker position to the camera controller 308, the aggregator unit 304 may convert the estimated speaker position from the first common coordinate system to a second common coordinate system associated with one of the cameras 306a, ..., z (e.g., the coordinate system of the camera 306a), so that the estimated speaker position can be readily used by the cameras 306a, ..., z. In some cases, the aggregator unit 304 may also transmit the positions of the microphones 302a, ..., z that detected the sound or otherwise generated the detected sound location to the camera controller 308. In this case, the aggregator unit 304 may also convert the position of each microphone 302a, ..., z to a common coordinate system that can be readily used by the cameras 306a, ..., z.

[0066] In embodiments, the aggregator unit 304 and the camera controller 308 may communicate via a suitable application programming interface (API), which may enable the camera controller 308 to query the aggregator unit 304 for the location of a particular microphone 302a, ..., z, enable the aggregator unit 304 to transmit a signal to the camera controller 308, and / or enable the camera controller 308 to transmit a signal to the aggregator unit 304. For example, in some cases, the aggregator unit 304 may transmit the converted location to the camera controller 308 in response to a query from the camera controller 308 via the API. Similarly, each microphone 302a, ..., z may be configured to communicate with the aggregator unit 304 using a suitable API, which may enable the microphone 302a, ..., z to transmit the location coordinates to the aggregator unit 304 upon a query received from the aggregator unit 304.

[0067] The camera controller 308 may receive the positions of the microphones 302a, ..., z, the lobe positions, and / or the estimated speaker positions from the aggregator unit 304. Based on the received positions, the camera controller 308 may select which of the cameras 306a, ..., z is used to capture images and / or video of a particular location (e.g., where an active speaker is located). The camera controller 308 may provide appropriate signals to the selected camera 306a, ..., z to cause the camera 306a, ..., z to move and / or zoom. For example, the camera controller 308 may utilize the positions of the microphones 302a, ..., z, the received lobe positions, and / or the estimated speaker positions in the coordinate system of the cameras 306a, ..., z in order to generate optimized camera parameters that allow for more accurate zooming, panning, and / or framing of the speaker 102.

[0068] It should be understood that Figures 1 to 4 The components shown in are merely exemplary, and any number, type, and placement of the various components of system 100, array 200, and / or system 300 are contemplated and possible. Figure 1 In the embodiment, there may be multiple cameras 106 and / or camera controllers coupled between the cameras 106 and the aggregator 108, such as Figure 3 As shown in .

[0069] Figure 5 An exemplary method or process 400 is described for providing an estimated speaker position to a camera based on speaker coordinates obtained by multiple microphones according to an embodiment. A camera (e.g., Figure 1 and 2 The camera 106 and / or Figure 3 , z) and microphones (e.g., Figure 1 and 2 The microphone 104 and / or Figure 3The microphones 302a, ..., z) can form a plurality of microphones located in the environment (eg, Figure 1 and 2 10) in a conference system (e.g. Figure 1 and 2 System 100 and / or Figure 3 The environment can be a conference room, an event space, or a room containing one or more speakers (e.g., Figure 1 and 2 The camera may be configured to detect and capture images and / or video of the speaker. The microphone may be configured to detect and capture sounds produced by the speaker and determine the location of the detected sounds. The method 400 may be performed by one or more processors of the conferencing system, such as a processor of a computing device included in the system and communicatively coupled to the camera and microphone. In some cases, the method 400 may be performed by an aggregator (e.g., Figure 1 Aggregator 108 or Figure 4 aggregator unit 304) and / or a camera controller (e.g., Figure 4 Executed by camera controller 308).

[0070] like Figure 5 , method 400 may include, at step 402, using a first microphone (or microphone array) and determining a first speaker position in a first coordinate system relative to the first microphone array based on audio associated with the speaker. For example, the first coordinate system may be a coordinate system in which the first microphone array is located at an origin, or any other suitable coordinate system. In an embodiment, determining the first speaker position includes detecting sounds generated near the first microphone array (e.g., audio associated with the speaker) using the first microphone array, and using an audio activity locator (e.g., Figure 2 An audio localization algorithm executed by the audio activity locator 208) determines the location of the detected sound.

[0071] Step 404 includes using a second microphone (or microphone array) and based on the same speaker (e.g., Figure 1 The second speaker position is determined in a second coordinate system relative to the second microphone array using the same audio associated with the speaker 102 (e.g., the speaker 102). For example, the second coordinate system can be a coordinate system in which the second microphone array is located at an origin, or any other suitable coordinate system. In an embodiment, determining the second speaker position includes using the second microphone array to detect sounds generated near the second microphone array (e.g., audio associated with the speaker), and using an audio activity locator (e.g., Figure 2 An audio localization algorithm executed by the audio activity locator 208) determines the location of the detected sound.

[0072] Step 406 includes using at least one processor (e.g., Figure 1 and determining a position relative to the camera (eg, Figure 1 In some embodiments, determining the estimated speaker position includes identifying a common point based on the first speaker position and the second speaker position using at least one processor, for example, as described herein with respect to Figure 1 In other embodiments, determining the estimated speaker location includes identifying, using at least one processor, a closest point based on the first speaker location and the second speaker location, for example, as described herein with respect to Figure 2 described.

[0073] In an embodiment, method 400 further includes converting the detected speaker position to a common coordinate system before calculating the estimated speaker position. The common coordinate system may be relative to a camera, one of the microphone arrays, a room or environment in which the conferencing system is located, or another component of the system. For example, in some embodiments, method 400 further includes converting the first speaker position from the first coordinate system to the second coordinate system using at least one processor; determining the estimated speaker position in the second coordinate system based on the first and second speaker positions in the second coordinate system using at least one processor; and converting the estimated speaker position from the second coordinate system to a third coordinate system using at least one processor. In other embodiments, method 400 further includes converting the first speaker position from the first coordinate system to the third coordinate system using at least one processor; converting the second speaker position from the second coordinate system to the third coordinate system using at least one processor; and determining the estimated speaker position in the third coordinate system based on the first and second speaker positions in the third coordinate system using at least one processor.

[0074] Step 408 includes transmitting the estimated speaker position in the third coordinate system from the at least one processor to the camera so that the camera points the image capture component of the camera to the estimated speaker position in the third coordinate system. The camera may point the image capture component to the estimated speaker position in the third coordinate system by adjusting one or more of the camera's angle, tilt, zoom, and framing, or any other relevant parameters of the camera.

[0075] Figure 6An exemplary environment 50 is depicted in which one or more of the systems and methods disclosed herein may be used. As shown, according to an embodiment, the environment 50 includes a conferencing system 500 that can be used to detect the location of a speaker 502 in the environment 50, direct a lobe of a microphone 504 toward the detected speaker location, and refine the lobe location based on a second speaker location determined by a camera 506. It should be understood that although Figure 6 One potential environment is described, but the systems and methods disclosed herein may be used in any applicable environment, including but not limited to conference rooms, offices, small meeting rooms, theaters, arenas, music venues, etc. System 500 may include Figure 6 502, such as one or more speakers, desktop microphones, display screens, and / or computing devices. In an embodiment, one or more of the components in system 500 may include a digital signal processor, a wireless receiver, a wireless transceiver, etc. In addition, environment 50 may also include one or more other persons other than speaker 502 and / or other objects not shown (e.g., musical instruments, phones, tablet computers, computers, HVAC equipment, etc.). It should be understood that Figure 6 The components shown in are merely exemplary, and any number, type, and placement of various components in environment 50 are contemplated and possible.

[0076] According to an embodiment, environment 50 may be substantially similar to Figure 1 environment 10, and the conference system 500 may be substantially similar to Figure 1 The conference system 100 and / or may be used Figure 4 For example, as shown, the conference system 500 includes a microphone 504, which can be substantially similar to Figure 1 Microphone 104, Figure 4 Microphones 302a, ..., z and / or Figure 3 The microphone array 200, and at least one camera 506, which may be substantially similar to Figure 1 The camera 106 and / or Figure 4 Therefore, for the sake of brevity, the camera 506 and the microphone 504 will not be described in detail. The components of the conference system 500 can communicate with each other and / or with a remote device (eg, a cloud computing server, etc.) by wire or wireless communication.

[0077] In some embodiments, the system 500 further includes a camera controller (e.g., Figure 4506) receives the position information from the microphone 504 and positions the camera 506 toward the received position. In other embodiments, the camera 506 may include a camera controller. The camera controller may point the camera at a specific position (e.g., the speaker position) or position the camera toward a specific position by adjusting one or more of the camera's angle, tilt, zoom, and framing, or any other suitable settings.

[0078] In some embodiments, system 500 further includes an aggregator (e.g., Figure 4 The aggregator unit 304 is an aggregator unit 304 that converts the position information received from the microphone 504 into a common coordinate system before providing the position information to the camera 506 for camera positioning. In other embodiments, the aggregator may be included in the microphone 504 or the camera 506. The common coordinate system may be a coordinate system centered on the microphone 504, the camera 506, and / or the environment 50, or any other previously determined coordinate system.

[0079] Microphone 504 can detect sounds or audio activity in environment 50 and determine the location of the sound or the audio source that produces the sound (e.g., speaker 502) relative to microphone 504 or in the coordinate system of microphone 504. For example, microphone 504 can include an audio activity locator (e.g., Figure 3 208) or other audio source localization algorithm, which is configured to localize the audio source based on the microphone elements (e.g., Figure 3 The audio signals received by microphone elements 202a, b, ..., zz) are used to determine a set of coordinates (or "positioning coordinates") representing the locations of audio activities (or "detected talker locations") in the environment 50.

[0080] The microphone 504 may also deploy a lobe toward the detected speaker location for capturing audio produced by the active speaker 502. For example, the microphone 504 may include a lobe selector (e.g., Figure 3 The microphone 504 may also include a beamformer (e.g., a beamformer). Figure 3 The beamformer 206 is used to deploy the audio pickup beam towards the position of the selected lobe, which can be provided as a set of coordinates in the coordinate system of the microphone 504.

[0081] As described herein, the position obtained by the audio source localization algorithm may represent the perceived position of the audio activity and may actually be an estimate of the audio source position, which may not coincide with the actual or true position of the audio activity. For example, when the localization coordinates are provided as spherical coordinates (az,elev,R), the radius component R may not be accurate enough, so that the localization point (e.g., Figure 6 The point Pa in ) does not correspond to the actual speaker position (e.g. Figure 6 Thus, a lobe deployed toward the position of a speaker detected by microphone 504 may only partially capture the sound produced by speaker 502, or not capture it at all, e.g. Figure 6 As shown in .

[0082] In embodiments, the conferencing system 500 may be configured to use the camera 506 to identify the actual location of an active speaker, or otherwise at least improve the radial coordinates of the detected speaker location obtained by the microphone 504. For example, the microphone 504 may transmit the detected speaker location and / or the selected lobe location to the camera 506 directly or via an aggregator and / or camera controller, as described herein. The position information (e.g., a set of coordinates representing the lobe location or the detected speaker location) may be relative to the coordinate system of the microphone 504, and thus may be converted to position information relative to the coordinate system of the camera 506. The conversion may be performed by the microphone 504, the camera 506, or any other suitable component of the system 500. In some embodiments, the position information may be converted before being transmitted to the camera 506, for example by the microphone 504 or other component of the system 500.

[0083] Upon receiving the location information, the camera 506 may point the image capture component of the camera 506 to the received location (e.g., the detected speaker location or lobe location). The image capture component may be configured to capture static images, moving images, and / or videos. In an embodiment, the camera 506 may include a speaker detection component that uses a facial detection algorithm, a human head detection algorithm, or any other suitable image processing algorithm to identify the face, head, or any other identifiable part of the speaker 502, or otherwise determine the camera angle that best corresponds to the face of a person or the front of a person near the received location information. For example, the speaker detection component may track a motion vector and scan the face or head of the speaker 502 while the camera 506 moves the image capture component along a straight line extending from the received location (e.g., point Pa) at the same azimuth and elevation angle as the coordinates of the received location. In other cases, the camera 506 may scan a nearby area or area of ​​the environment 50 in a grid-like manner until the face or head of the speaker 502 is identified. Other known techniques for obtaining a more accurate location of the speaker 502 may also be used.

[0084] like Figure 6 , camera 506 may identify the face of speaker 502 at a second location (e.g., point Pb) at a distance from the received location. In embodiments where the azimuth and elevation components remain the same, the coordinates of the second location (or "second speaker location") may differ from the coordinates of the initial location only in terms of the radius or distance component. In other embodiments, the second speaker location may differ from the received location (e.g., the first speaker location or lobe location) in terms of the azimuth, elevation, and / or distance components. In either case, the lobe position of microphone 504 may be adjusted to a new lobe position based on the second speaker location, thus enabling the microphone lobe to fully capture the audio generated by speaker 502. In particular, camera 506 may provide the second speaker location (or coordinates representing the second speaker location) to microphone 504. Microphone 504 may adjust the lobe position determined by microphone 504 by changing one or more coordinates of the lobe position based on the received coordinates. For example, microphone 504 may change or adjust the distance coordinates of the lobe position based on the distance coordinates of the second speaker location. If the lobe is already deployed toward the active talker 502, the beamformer of the microphone 504 may steer the lobe toward the adjusted lobe position. If the lobe is not deployed, the beamformer may deploy or steer the lobe toward the adjusted lobe position.

[0085] In some cases, the second talker position may be located in the coordinate system of camera 506. In this case, camera 506 may convert the second talker position to the coordinate system of microphone 504 before transmitting the position information to microphone 504. In other cases, the second talker position may be transmitted to microphone 504 as coordinates in the coordinate system of camera 506. In this case, microphone 504 may convert the received coordinates to the coordinate system of microphone 504 before adjusting the lobe position. In some embodiments, the conversion step may be performed by one or more other components of system 500 (e.g., an aggregator).

[0086] It should be noted that the techniques described herein for refining the lobe positions of microphones using the talker coordinates obtained by a camera can also be used in conferencing systems that include multiple microphone arrays, although not in Figure 6 506. In this case, the speaker coordinates determined using the face detection algorithm of camera 506 may be used to guide the selection of a lobe for one of the microphones (or microphone arrays). In some cases, the lobe may be selected based on which lobe and / or microphone array is best suited to pick up audio from the speaker position detected by camera 506. In other cases, the lobe may be pre-selected based on the positioning coordinates obtained by the microphone array based on the detected audio activity.

[0087] Figure 7An exemplary method or process 600 is described for refining the lobe position of a microphone (or microphone array) based on a talker position determined by a camera, according to an embodiment. A camera (e.g., Figure 6 camera 506) and microphone (e.g., Figure 6 The microphone 504) may be formed in an environment (e.g., Figure 6 environment 50) in a conference system (e.g., Figure 6 The environment can be a conference room, an event space, or a room containing one or more speakers (e.g., Figure 6 The method 600 may be performed by one or more processors of the conferencing system, such as a processor of a computing device included in the system and communicatively coupled to the camera and microphone array. In some cases, the method 600 may be performed by an aggregator (e.g., Figure 4 aggregator unit 304) and / or a camera controller (e.g., Figure 4 Executed by camera controller 308).

[0088] like Figure 7 , method 600 may include, at step 602, using a microphone (or microphone array) and determining a first speaker position of the microphone in a first coordinate system relative to a first microphone based on audio associated with the speaker. For example, the first coordinate system may be a coordinate system in which the first microphone is located at an origin, or any other suitable coordinate system. In an embodiment, determining the first speaker position includes detecting a sound generated near the first microphone (e.g., audio associated with a speaker) using the first microphone, and using an audio activity locator (e.g., Figure 3 An audio localization algorithm executed by the audio activity locator 208) determines the location of the detected sound.

[0089] Step 604 includes using at least one processor to convert the first speaker position from the first coordinate system to a second coordinate system relative to the camera. For example, the second coordinate system can be a coordinate system in which the camera is located at the origin, or any other suitable coordinate system.

[0090] Step 606 includes transmitting the first speaker position in the second coordinate system from the at least one processor to the camera so that the camera points the image capture component of the camera at the first speaker position. Figure 6, if the first speaker location is not the actual location of the speaker, the camera may be pointing away from the active speaker and / or may not be capturing the speaker correctly. In this case, the camera may use a speaker detection component of the camera to identify a second speaker location in the camera's second coordinate system that is proximal to the first speaker location and includes or points to the location of the speaker's face. For example, the speaker detection component may use a facial detection algorithm or the like to identify an image that resembles a human face, or otherwise detect the face of the active speaker, as described herein.

[0091] Step 608 includes receiving from the camera a second speaker position in a second coordinate system identified by the camera using a speaker detection component of the camera. Step 610 includes adjusting, using the microphone, a lobe position of the microphone based on the second speaker position received from the camera. For example, the distance coordinates of an initial lobe position of the microphone may be adjusted or refined based on the distance coordinates of the second speaker position determined by the camera. In some cases, the microphone may adjust the lobe position by deploying a new lobe toward the second speaker position. In other cases, the microphone may adjust the lobe position by directing an existing lobe toward the second speaker position.

[0092] In some embodiments, the second talker position may be converted to the first coordinate system of the microphone before adjusting the lobe position of the microphone. For example, in this case, step 610 may include: converting the second talker position from the second coordinate system to the first coordinate system using at least one processor; and adjusting the distance coordinate of the lobe position in the first coordinate system based on the distance coordinate of the second talker position in the first coordinate system using at least one processor.

[0093] In other embodiments, the position of the microphone lobe may be converted to the first coordinate system after adjusting the lobe position in the second coordinate system based on the second talker position. For example, in this case, step 610 includes adjusting, using at least one processor, the distance coordinate of the lobe position in the second coordinate system based on the distance coordinate of the second talker position in the second coordinate system; and converting, using at least one processor, the adjusted lobe position from the second coordinate system to the first coordinate system.

[0094] Figures 8 to 10 An exemplary environment 70 is depicted in which one or more of the systems and methods disclosed herein may be used. As shown, according to an embodiment, the environment 70 includes a conference system 700 that can be used to improve camera positioning by using lobe positions selected based on speaker coordinates obtained by one or more microphones 704 to determine which of a plurality of cameras 706 is best suited to capture images and / or video of each speaker 702. It should be understood that although Figures 8 to 10One potential environment is described, but the systems and methods disclosed herein may be used in any suitable environment, including but not limited to offices, small conference rooms, theaters, arenas, music venues, etc. System 700 may include Figures 8 to 10 700, such as one or more speakers, desktop microphones, display screens, and / or computing devices. In an embodiment, one or more of the components in system 700 may include a digital signal processor, a wireless receiver, a wireless transceiver, etc. In addition, environment 70 may also include one or more other persons other than speaker 702 and / or other objects not shown (e.g., musical instruments, phones, tablet computers, computers, HVAC equipment, etc.). It should be understood that Figures 8 to 10 The components shown in are merely exemplary, and any number, type, and placement of various components in environment 70 are contemplated and possible.

[0095] According to an embodiment, the environment 70 may be substantially similar to Figure 1 environment 10, and the conference system 700 can be substantially similar to Figure 1 The conference system 100 and / or may be used Figure 4 For example, as shown, the conference system 700 includes a plurality of microphones 704, each of which may be substantially similar to Figure 3 Microphone array 200, Figure 1 The microphone 104 and / or Figure 4 The conference system 700 also includes a plurality of cameras 706, each of which can be substantially similar to Figure 1 The camera 106 and / or Figure 4 In addition, the system 700 includes an aggregator 708, which can be substantially similar to Figure 4 The aggregator unit 304 and / or Figure 1 Therefore, for the sake of brevity, the microphone 704, the camera 706 and the aggregator 708 will not be described in detail.

[0096] In some embodiments, system 700 further includes a camera controller (e.g., Figure 4 The camera controller 308 is configured to receive position information from the microphone 704 and / or the aggregator 708 and position the camera 706 toward the received position. The camera controller may be a separate device integrated with the aggregator 708 or included in one of the cameras 706. The camera controller may point the camera at a specific position (e.g., the speaker position) or position the camera toward a specific position by adjusting one or more of the angle, tilt, zoom, and framing of the camera or any other suitable setting.

[0097] The components of the conference system 700 may communicate with each other and / or with remote devices (e.g., cloud computing servers, etc.) by wire or wireless communication. In some embodiments, one or more components of the system 700 may be integrated together or integrated into a single device. For example, the aggregator 708 may include one of the microphones 704 or one of the cameras 706.

[0098] Each microphone 704 can detect sounds or audio activity in environment 70 and determine the location of the sound or the audio source that produces the sound (e.g., speaker 702) relative to microphone 704, or in the coordinate system of microphone 704. For example, each microphone 704 can include an audio activity locator (e.g., Figure 3 208) or other audio source localization algorithms for locating audio sources based on the audio activity locator 208 of the microphone 704 (e.g., Figure 3 The audio signals received by the microphone elements 202a, b, ..., zz) are used to determine a set of coordinates (or "positioning coordinates") representing the locations of audio activities (or "detected talker locations") in the environment 70.

[0099] Each microphone 704 (or microphone array) may also deploy a lobe toward the detected talker location for capturing audio produced by the active talker 702. For example, each microphone 704 may include a lobe selector (e.g., Figure 3 The microphone 704 may also include a beamformer (e.g., a beamformer) that is configured to determine which of the multiple lobes of the microphone 704 is best suited to pick up or capture the audio at the location coordinates determined by the audio activity locator. The audio activity locator may send the detected talker location or the corresponding audio location coordinates to the lobe selector to select the optimal lobe based thereon. The microphone 704 may also include a beamformer (e.g., Figure 3 The beamformer 206 of the microphone 704 is used to deploy an audio pickup beam (or lobe) towards a selected lobe position, which can be provided as a set of coordinates in the coordinate system of the microphone 704.

[0100] As shown, the environment 70 includes multiple microphones 704 located in different areas of a room or space. Each microphone 704 can be configured to deploy multiple lobes to various locations in the environment 70 using, for example, automatic lobe tracking techniques, static lobe techniques, and / or other suitable techniques. The use of multiple microphones 704 and lobes can improve the sensing and capture of sounds from audio sources in the environment 70 and provide more accurate estimates of the locations of active speakers in the environment 70, as described herein. The environment 70 also includes multiple cameras 706 located in different areas of the room or space. The use of multiple cameras 706 can enable the capture of more and different types of images and / or videos of the environment 70. For example, a camera 706 located at the front of the environment 70 can be used to capture a wider view of the room, while a camera 706 located on a wall of the room can be used to capture a close-up of a speaker 702 in the environment 70.

[0101] The presence of multiple microphones and multiple cameras in a given environment can also complicate camera selection and / or microphone or lobe selection for active speakers. For example, when there are multiple microphone arrays, there may be more than one possible audio beam or lobe position for covering an active speaker, which may make it difficult to identify a unique speaker and / or lobe position for camera positioning purposes. As another example, when there are multiple cameras, it may be difficult to determine which camera should be oriented or pointed toward each speaker due to overlapping coverage areas and / or conflicting camera angles.

[0102] In an embodiment, Figures 8 to 10 The conference system 700 can be configured to optimize the active audio sources (e.g., Figure 8 The speaker 702 in the figure selects an appropriate microphone from the plurality of microphones 704 and / or selects an appropriate camera from the plurality of cameras 706. Figure 8 , the environment 70 may be divided into a number of areas 72 and 74, such that each area 72, 74 includes only one of the cameras 706. In various embodiments, the areas 72 and 74 may be assigned to respective cameras 706, depending on which camera 706 has the best field of view or coverage of the area, or otherwise based on, for example, the angle or orientation of the camera 706 that is best suited to capture a speaker located in that area of ​​the environment 70. The areas 72 and 74 may be preconfigured by a technician or installer of the system 700 based on various aspects of the environment 70, including, for example, the size and shape of the room or space, the number of cameras 706 present, the location or placement of the cameras 706, and the presence and / or location of other known objects (e.g., tables, chairs, podiums or podiums, display screens and / or whiteboards, microphones 704, speakers, etc.). Although Figures 8 to 10Environment 70 is shown as including only two regions, but it should be appreciated that any number of regions may be created in a given environment, eg, depending on the physical characteristics of the environment, the total number of cameras present, and / or other relevant considerations.

[0103] In some embodiments, the installer may manually determine the position of the microphone 704 relative to the environment 70, the placement and orientation of each of the cameras 706 relative to the microphone 704, and / or the position of other objects in the environment 70. In other embodiments, the installer may use a graphical tool or other software of the system 700 that is configured to automatically determine the position of each microphone 704, the position and orientation information of each of the cameras 706, and / or the position or placement information of other objects in the environment 70. For example, the graphical tool may be configured to scan one or more objects of the environment 70 using sound excitation, audio localization, and / or triangulation techniques, etc. to identify the position of the cameras 706, microphones 704, and / or speakers (not shown) relative to a common coordinate system (e.g., the coordinate system of the environment 70 or one of the components of the system 700).

[0104] In an embodiment, the known positions of the microphones 704 and cameras 706 and the parameters of the areas 72 and 74 (including the camera assignments) may be provided to the aggregator 708 and stored in a memory communicatively coupled to the aggregator 708. In some cases, the aggregator 708 may receive the parameters of the areas 72 and 74 from a separate computing device internal or external to the system 700. In some cases, for example, the aggregator 708 may also receive the known microphone and camera positions from a separate computing device that may be used to operate a graphical tool to automatically determine the array and camera positions. In other cases, the aggregator 708 may receive the microphone positions from each of the microphones 704 and the camera positions and orientations from each of the cameras 706.

[0105] During operation, aggregator 708 may receive position information from multiple microphones 704, such as position location coordinates generated when audio activity associated with speaker 702 is detected. For example, aggregator 708 may combine time-synchronized audio position location coordinates (or detected speaker locations) received from two or more of microphones 704 for the same audio activity to determine or estimate the active speaker location in environment 70 with greater accuracy than the individual coordinates. Position location coordinates may be combined by determining, for example, a common point or intersection of vectors formed by the position location coordinates, or the closest point between detected speaker locations (as described herein), or using any other suitable technique.

[0106] Prior to calculating the estimated speaker position, the aggregator 708 may first convert the received coordinates (or detected speaker positions) to a common coordinate system so that the estimated speaker position is provided in the common coordinate system. The common coordinate system may be a coordinate system centered on one of the microphones 704, one of the cameras 706, and / or the environment 70, or any other previously determined coordinate system. In some cases, the aggregator 708 may convert the estimated speaker position from the common coordinate system to a second coordinate system, such as the coordinate system of the camera 706 that will receive the converted information, so that the estimated speaker position can be readily used by that camera 706. In other cases, a given camera 706 and / or camera controller may be configured to convert the position information received from the aggregator 708 to the second coordinate system of the receiving camera 706.

[0107] Aggregator 708 may use the estimated speaker location to identify or select a lobe among multiple microphones 704 that will optimally capture an active speaker and / or an estimated speaker location. For example, aggregator 708 may select a lobe that has a lobe position that corresponds to or overlaps with the estimated speaker location. If multiple lobe positions correspond to a speaker location, aggregator 708 may determine the distance between the speaker location and each of the microphones 704 in a common coordinate system (e.g., one of the microphones 704 or the coordinate system of environment 70), and may determine or identify the microphone 704 that is closest to the identified speaker location and / or has an available lobe position that is closest to the speaker location in the common coordinate system. In some cases, aggregator 708 may determine the direction that speaker 702 is facing (e.g., using facial detection technology of camera 706), and may identify the microphone 704 that is closest to the face of speaker 702 and / or has an available lobe that can be directed toward the speaker's face. Aggregator 708 may be configured to calculate the distance between the identified talker location and each of microphones 704 using the sound localization and triangulation techniques described herein or any other known techniques.

[0108] In other embodiments, the aggregator 708 may be configured to use networked automatic mixer technology, etc. to identify a unique speaker position in the environment 70, and select the optimal lobe across multiple microphones 704 to capture the identified speaker position. In this case, the microphones 704 may be connected together to form a network, and may receive a common gating control signal from the aggregator 708, which indicates which of the microphone lobes across the network is gated open and / or which lobes are gated closed. As an example, the network automatic mixer may generate a common gating control signal by determining which lobe detects the strongest voice signal (or other audio signal) for a given audio event and by selecting the lobe and the corresponding microphone 704 for the common gating control signal. In the case where the microphone 704 is a microphone array, the microphone 704 may be configured to generate a beamformed audio signal based on the audio detected by the microphone elements in the microphone 704 and the common gating control signal. Then, the aggregator 708 may generate a final mixed audio signal for the detected audio event by aggregating the beamformed audio signals received from the microphones 704. Thus, the final mixed audio signal may reflect a desired audio mix, where audio from certain channels of microphone 704 is emphasized, while audio from other channels is weakened or suppressed.

[0109] The common gating control signal may also be used by the aggregator 708 to determine the estimated position and / or orientation of the speaker 702. For example, the aggregator 708 may use the common gating control signal and / or other gating decision information to determine which lobe of the microphone 704 is available (e.g., gated). In addition, given that the common gating control signal is derived based on the strongest voice signal detected by the networked automatic mixer, the aggregator 708 may use the placement or position of the gated open lobe to determine the estimated speaker orientation. For example, the aggregator 708 may use the networked automatic mixer signal to determine the direction the speaker 702 is facing, and may use the gating decision to identify the active lobe and / or microphone 704 directed toward the speaker's face and the estimated speaker position, or otherwise better able to capture the audio in the direction the speaker 702 is facing. As an example, the gating decision from the networked automatic mixer may be used to determine that the speaker 702 is physically closer to the first microphone 704, but the second microphone 704 is more suitable for picking up the voice of the speaker 702 because the speaker 702 is facing the second microphone 704.

[0110] In some cases, the network automatic mixer may receive position information directly from one or more microphones. For example, in embodiments where the microphone 704 includes one or more directional microphones or other non-array microphones (e.g., lavalier microphones, handheld microphones, boundary microphones, etc.), the positions of such microphones and / or the placement and / or direction of the audio pickup pattern or other type of microphone coverage used by such microphones (collectively referred to herein as "position information") may be previously known or predetermined. In this case, the aggregator 708 may receive the position information of the one or more directional microphones 704 from the network automatic mixer as previously stored data or data sent at runtime, and the aggregator 708 may use the known position information to determine (or triangulate) the estimated position of the speaker 702 and select the appropriate camera 706 based thereon. Therefore, in some cases, the aggregator 708 may make camera selections based not only on audio source positioning, but also on readily available or known position information obtained from multiple microphones 704.

[0111] like Figure 8 , once the aggregator 708 has selected a lobe 705 and / or microphone 704 to optimally capture the audio produced by the speaker 702, the selected lobe 705 may be deployed toward the speaker 702 by the corresponding microphone 704 (e.g., the second microphone 704) using one or more of the techniques described herein. Additionally, the position of the selected lobe 705 (or "lobe position") may be used by the aggregator 708 to determine or identify which of the multiple cameras 706 is best suited to capture images and / or video of the speaker 702 and / or the selected lobe position. In some cases, the gating decision of the networked automatic mixer may also drive which camera 706 is used. For example, because the networked automatic mixer has a global view of the environment 100, the gating decision described above may be used to identify the camera 706 that is pointed toward the face of the speaker 702, and thus avoid selecting a camera that is pointed toward the back of the speaker's head.

[0112] In some cases, the aggregator 708 may select an appropriate camera 706 by identifying an area 72, 74 in the environment 70 that includes or corresponds to the selected lobe location and by selecting a camera 706 that is assigned to or configured to capture the identified area 72, 74. Figure 8 , the aggregator 708 may select the first camera 706 to capture the speaker 702 because the selected lobe 705 is located within the first area 72 and the first camera 706 is assigned to the first area 72. The aggregator 708 may select the selected lobe position from the corresponding microphone 704 (e.g., Figure 8The first coordinate system of the second microphone 704 in the environment 70 is converted to a second coordinate system associated with the environment 70 in order to identify which of the areas 72 and 74 of the environment 70 contains the selected lobe position.

[0113] Aggregator 708 may send or transmit the coordinates of the selected lobe position to the selected camera 706 for positioning or pointing the image capture component of camera 706 toward speaker 702, or more specifically, the selected lobe position. In some embodiments, once the appropriate camera 706 is selected, aggregator 708 may convert the lobe position from the second coordinate system of environment 70 to a third coordinate system associated with the selected camera 706 so that the lobe position (or a set of coordinates) may be readily used by camera 706 when received. In other embodiments, aggregator 708 may transmit the lobe position in the first or second coordinate system to the selected camera 706, and camera 706 and / or camera controller may be configured to convert the received lobe position to the third coordinate system.

[0114] Although Figure 8 Only one speaker 702 is shown, but it should be appreciated that similar techniques may be applied where multiple speakers 702 are present in the environment 70. In particular, the aggregator 708 may determine the optimal lobe position for each of the speakers 702 one by one using one or more of the techniques described herein. Fig. 9 As shown in , the aggregator 708 may determine that the first lobe 705 of the first microphone 704 is optimal for capturing the sound generated by the first speaker 702, while the second lobe 705 of the second microphone 704 is optimal for capturing the sound generated by the second speaker 702.

[0115] Fig. 9 Also shown are two speakers 702 located at different locations in the same area 72 of the environment 70, and therefore cannot be captured by the same camera 706. In this case, the aggregator 708 can further analyze the relative positions of the cameras 706, microphones 704, and each of the areas 72 and 74 to identify the camera 706 that is best located or has the best angle for capturing each of the speakers 702. For example, in Fig. 9 , the aggregator 708 may determine that the first camera 706 is closest to the second speaker 702 and has the clearest view of the second speaker 702 , and therefore have the second camera 706 capture the first speaker 702 , even though both speakers 702 are located in the area 72 assigned to the first camera 706 .

[0116] In various embodiments, microphones 704 and / or cameras 706 may track one or more active speakers 702 in real time as the one or more active speakers 702 move around in environment 70 while speaking or otherwise producing sounds. For example, each of cameras 706 may be configured to scan its assigned area 72, 74 for images that resemble a human face, and may stay on the identified face as the corresponding speaker 702 moves around (e.g., sits, stands, gestures, changes position, etc.). As another example, microphones 704 may be configured to continuously or periodically determine speaker locations for newly detected sounds and provide the newly identified speaker locations to aggregator 708. Aggregator 708 may combine new speaker locations corresponding to the same audio activity from each of microphones 704 and determine a more accurate estimate of the speaker's current location based thereon.

[0117] In some cases, speaker 702 may move around so dramatically in environment 70, or move to a new location, that camera 706 and / or microphone 704 directed toward speaker 702 may no longer be able to capture speaker 702 and / or may not be suitable to do so. Fig.10 An exemplary scenario is shown in which a second speaker 702 speaks from a first location in a first area 72 (e.g., Fig. 9 ) moves to a second position in the second area 74, where the first camera 706 no longer has a clear line of sight to the second speaker 702 (e.g., because the first speaker 702 blocks the line of sight).

[0118] In this case, the aggregator 708 may change the camera and / or microphone assignments in the environment 70 based on the new speaker position or in order to accommodate the new speaker position. For example, the aggregator 708 may identify or estimate the new position of the second speaker 702 based on the new audio signals detected by one or more microphones 704 and using audio localization and / or triangulation techniques, as described herein. The aggregator 708 may determine that the new speaker position is now closer to the first microphone 704 than the second microphone 704, and that the audio produced at the new speaker position will be better captured by the second lobe 705 of the first microphone 704. Therefore, the first microphone 704 may be directed, for example, by the aggregator 708 to deploy and / or direct the second lobe 705 toward the new speaker position of the second speaker 702. In addition, the lobe of the second microphone 704 that was initially directed toward the second speaker 702 (e.g., as Fig. 9 ) can be gated off or deactivated.

[0119] Similarly, the aggregator 708 may determine that the new speaker location is closer to the second camera 706 than the first camera 706, and thus may direct the second camera 706 to point the image capture component of the second camera 706 away from the first speaker 702 (e.g., as shown in FIG. Figure 8 ) and towards the new speaker position, as shown in Fig.10 The aggregator 708 may also determine that the first camera 706 is now optimal for capturing the first speaker 702, and thus may move or direct the first camera 706 toward the first speaker 702 or an estimated speaker position corresponding to the first speaker 702, such as Fig.10 As shown in .

[0120] In some embodiments, Figures 8 to 10 The conference system 700 shown in FIG. 70 can be configured to enable speakers 702 to capture their images using one or more personal cameras (not shown) rather than cameras 706 located in the environment 70. For example, one or more of the speakers 702 may choose to join a conference call or other event using a camera installed on a laptop, tablet, or other personal device that can be brought into the environment 70 by the speakers 702. In this case, the speakers 702 can connect their personal devices to the aggregator 708 or other controller in the environment 70 and use their devices to contribute individual videos to the call. In some cases, personal cameras can be used in conjunction with one or more of the cameras 706. For example, a first camera 706 can be used to capture a first speaker 702 located in a first area 72 of the environment 70, while personal cameras can be used to individually capture a second speaker 702 and other speakers (not shown) located in a second area 74 of the environment 70. In any case, the aggregator 708 can be configured to stitch the received videos together or otherwise combine the captured images into a single video output. For example, the individual videos of each speaker may be presented as separate strips (eg, vertically or horizontally), in separate tiles, or in any other portion of the combined video.

[0121] In embodiments using personal cameras, one or more of the microphones 704 may still be used to provide sound localization information for estimating speaker locations using the techniques described herein, as well as capturing audio signals generated in the environment 70 (e.g., for transmission to remote participants of a conference call or other event). The estimated speaker locations may be used for purposes other than directing the appropriate (e.g., nearest) camera 706 toward the corresponding speaker 702. For example, when a given speaker 702 uses its own camera, the aggregator 708 may be configured to determine the estimated location of that speaker 702, for example, based on the sound localization provided by one or more of the microphones 704, and assign the estimated speaker location to the given speaker's personal camera or device. In this way, the output of each individual camera may be associated with a specific location in the environment 70. In addition, the aggregator 708 may be configured to use the estimated speaker location to pair each captured audio signal with an appropriate personal camera, i.e., a camera directed toward the speaker 702 that produced that audio (or a speaker 702 located at or near the estimated speaker location). This may prevent the aggregator 708 from enabling one of the cameras 706 installed in the environment 70 based on audio detected at the location of the individual camera user.

[0122] In various cases, the aggregator 708 may be configured to identify a speaker 702 (or its device) located in the environment 70 based on an identifier or other identifying information associated with the speaker 702 (or its device). For example, the aggregator 708 may receive the identifier from the speaker's personal device or camera as the speaker 702 enters the environment 70, or may have been previously provided as part of an ongoing conference call or other event. The identifier may be a device identifier, or other information uniquely associated with the speaker's personal camera (or device), a user identifier, or other information uniquely associated with a given speaker, or any other type of identifier. In some cases, the aggregator 708 may be further configured to use the identifier to assign the estimated speaker location to correspond to a specific speaker, personal camera, and / or personal device. For example, the aggregator 708 may assign an appropriate identifier (i.e., an identifier corresponding to a specific speaker) to the estimated speaker location.

[0123] Fig.11 An exemplary method or process 800 is described for determining camera positioning in an environment based on lobe positions selected based on talker coordinates obtained by one or more microphones (or microphone arrays) according to an embodiment. Figure 8 camera 706) and one or more microphones (e.g., Figure 8 The microphone 704) may be formed in an environment (e.g., Figure 8 environment 70) in a conference system (e.g., Figure 8The environment can be a conference room, an event space, or a room containing one or more speakers (e.g., Figure 8 The method 800 may be performed by one or more processors of the conferencing system, such as a processor of a computing device included in the system and communicatively coupled to the camera and microphone. In some cases, the method 800 may be performed by an aggregator (e.g., Figure 4 aggregator unit 304) and / or a camera controller (e.g., Figure 4 Executed by camera controller 308).

[0124] like Fig.11 , method 800 may include, at step 802, using a plurality of microphones (or microphone arrays) and at least one processor and determining a speaker position in a first coordinate system relative to a first microphone of the plurality of microphones based on audio associated with the speaker. For example, the first coordinate system may be a coordinate system in which the first microphone is located at an origin, or any other suitable coordinate system. In an embodiment, determining the speaker position may include using the first microphone to detect sounds generated near the first microphone, such as audio associated with the speaker, and using an audio activity locator (e.g., Figure 3 An audio localization algorithm executed by the audio activity locator 208) determines the location of the detected sound.

[0125] Step 804 includes selecting, using at least one processor and based on a speaker position in the first coordinate system, a lobe position of a selected one of the plurality of microphones in the first coordinate system. In some embodiments, selecting the lobe position may include: determining a distance between the speaker position in the first coordinate system and each of the plurality of microphones; and identifying the selected one of the plurality of microphones as being closest to the speaker position in the first coordinate system.

[0126] Step 806 includes selecting, using at least one processor, a first camera of the plurality of cameras based on the lobe position. In some embodiments, selecting the first camera may include: converting, using at least one processor, the lobe position from a first coordinate system to a common coordinate system; identifying a first region of a plurality of regions in the common coordinate system as including the lobe position in the common coordinate system, each region being assigned to one or more of the plurality of cameras; and identifying the first camera as being assigned to the first region. As an example, the common coordinate system may be an environment (e.g., Figure 8 environment 70) or any other coordinate system agreed upon by both parties.

[0127] Step 808 includes using at least one processor to convert the lobe positions to a second coordinate system relative to the first camera so that the lobe positions can be readily used by the first camera. For example, the second coordinate system can be a coordinate system in which the first camera is located at an origin, or any other suitable coordinate system. In some cases, the lobe positions can be converted from the first coordinate system of the first microphone array to the second coordinate system of the first camera. In other cases, the lobe positions can be converted from a third coordinate system of the environment to the second coordinate system of the first camera.

[0128] Step 810 includes transmitting the lobe position in the second coordinate system from at least one processor to the first camera so that the first camera directs the image capture component of the first camera to the lobe position in the second coordinate system. The first camera may direct the image capture component to the lobe position in the second coordinate system by adjusting one or more of the camera's angle, tilt, zoom, and framing, or any other relevant parameters of the camera.

[0129] As described herein, in some embodiments, the positions of cameras and / or microphones included in the conferencing systems described herein relative to each other and / or a given environment are previously known, for example, by another component of the system or by an external device in communication with the system. However, in other embodiments, the relative positions of the cameras and / or microphones are not initially known. In such cases, the conferencing system may use audio localization algorithms, triangulation techniques, and / or other tools to automatically determine the positions of cameras, microphones, and / or other audio devices relative to each other in the environment.

[0130] In some cases, the image may be captured based on images captured by cameras in the same environment (e.g. Figure 4 The position and / or orientation information obtained by the camera 306a) further improves or refines the given microphone (e.g., Figure 4 The invention also provides position and / or orientation information of microphone 302a), which in turn can improve the coordinate information provided to the camera to position it towards the active speaker.

[0131] More specifically, conferencing systems (e.g. Figure 4 of the conference system 300) (e.g., Figure 4 aggregator unit 304) and / or a camera controller (e.g., Figure 4 The camera controller 308 of the aggregator may be configured to point the camera at the microphone using the previously determined position of the array in a first coordinate system relative to the camera, and identify the first position as the center or "0" point of a second coordinate system relative to the microphone. Next, the aggregator and / or camera controller may point the camera at a second position in the first coordinate system that is known or intended to be the center (0,0,0) of the second coordinate system for the microphone.

[0132] Based on the first and second positions in the first coordinate system, the difference between the actual center of the second coordinate system of the microphone and the expected center of the coordinate system of the microphone can be calculated. The difference can be sent to the microphone to correct the deviation in the center of the second coordinate system. The difference can also be used to improve the coordinate transposition or conversion from the second coordinate system to the first coordinate system, and vice versa. In addition, the difference identified by the camera can be used to correct or refine the information indicating the orientation of the camera relative to the microphone. In some embodiments, the aggregator and / or the camera controller or other components of the conference system can be configured to automatically calculate the difference between the expected center and the true center of the second coordinate system. In other embodiments, the installer or other user can manually calculate the difference between the two positions and correct the center coordinate of the second coordinate system accordingly, or otherwise enter this value into the microphone array, for example, using an appropriate user interface.

[0133] Fig.12 Another exemplary environment 90 in which one or more of the systems and methods disclosed herein may be used is described. As shown, the environment 90 includes a conference system 900 that can be used to improve camera positioning of a speaker 902 located in different but adjacent areas 92 and 94 of the environment 90 based on speaker coordinates obtained by a microphone 904 located in different areas 92 and 94 of the environment 90. In an embodiment, the conference system 900 includes first and second microphones 904, first and second cameras 906, and first and second aggregators 908. The components of the conference system 900 may be generally similar to Figures 8 to 10 . For example, microphone 904 may be similar to microphone 704, camera 906 may be similar to camera 706, and aggregator 908 may be similar to aggregator 708. Therefore, for the sake of brevity, the individual components of conferencing system 900 will not be described in detail.

[0134] In the illustrated embodiment, environment 90 includes a first area 92 and a second area 94 adjacent to the first. Areas 92 and 94 may be side-by-side areas (as shown), or have a gap or space between them (not shown). In some cases, areas 92 and 94 may be physically separated workspaces or areas (e.g., using one or more dividers, walls, etc.). In other cases, areas 92 and 94 may be other areas of a "virtual" workspace or shared space that have designated boundaries known to aggregators 908 and speakers 902, but there may be no physical walls or other structures between the areas. As shown, first area 92 includes or encompasses first speaker 902, first microphone 904, first camera 906, and first aggregator 908. Similarly, second area 94 includes or encompasses second speaker 902, second microphone 904, second camera 906, and second aggregator 908. According to an embodiment, the areas 92 and 94 may be configured to allow speakers 902 to work or otherwise operate individually within their respective areas 92 and 94 despite being part of a shared space. For example, the virtual workspace may enable speakers 902 to individually participate in different video conference calls or other audio-visual events simultaneously without interfering with each other.

[0135] In an embodiment, to improve camera positioning or speaker tracking of the camera 906, the aggregator 908 may be configured to calculate an estimated speaker position of a given speaker 902 using speaker coordinates obtained by various microphones 904 in the environment 90, including one or more microphones 904 located in different areas 92, 94 than the given speaker 902. For example, a first microphone 904 may send a first set of speaker coordinates (e.g., x1, y1, z1) of a first estimated position p1 of the first speaker 902 to the first aggregator 908 using the positioning techniques described herein. For the same event, a second microphone 904 may also send a second set of speaker coordinates (e.g., x2, y2, z2) of a second estimated position p2 of the same first speaker 902 to the same first aggregator 908 using the positioning techniques. Using one or more techniques described herein, the first aggregator 908 may combine the two sets of coordinates to determine a more accurate estimated speaker position of the first speaker 902. Thus, the conferencing system 900 may be configured to obtain (or triangulate) estimated speaker locations with greater accuracy by using microphones 904 located outside of given areas 92, 94 of the environment 90 to increase the number of time synchronized position fixes that can be used to estimate the speaker locations.

[0136] Thus, the techniques described herein can help reduce the manual measurements typically performed by installers or integrators during configuration of a conferencing system, such as measurements of the distance and position between cameras and microphones. Thus, the amount of time and effort spent by installers, integrators, and users can be reduced, resulting in increased satisfaction with the installation and use of the conferencing system.

[0137] The components of the microphone array 200 and / or any of the conference systems 100, 300, 500, 700, and 900 may be implemented in hardware (e.g., discrete logic circuits, application specific integrated circuits (ASICs), programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), microprocessors, etc.), using software that can be executed by one or more computers (e.g., computing devices with processors and memory (e.g., personal computers (PCs), laptop computers, tablet computers, mobile devices, smart devices, thin clients, etc.), or by a combination of hardware and software. For example, some or all of the components of the microphone array 200 and / or any of the systems 100, 300, 500, 700, and 900 may be implemented using discrete circuit system devices and / or using one or more processors (e.g., audio processors and / or digital signal processors) executing program code stored in a memory (not shown), the program code being configured to perform one or more processes or operations described herein, such as, Figure 5 , 7 Thus, in embodiments, the microphone array 200 and / or any of the systems 100, 300, 500, 700, and 900 may include one or more processors, memory devices, computing devices, and / or other hardware components not shown in the figures.

[0138] All or part of the processes described herein (including Figure 5 Method 400, Figure 7 Method 600 and Fig.11 The method 800 of may be executed by one or more processing devices or processors (eg, analog / digital converters, encryption chips, etc.) located in a corresponding conference system (eg, Figure 4In addition, one or more other types of components (e.g., memory, input and / or output devices, transmitters, receivers, buffers, drivers, discrete components, logic circuits, etc.) may also be used in conjunction with a processor and / or other processing components to perform any, some, or all of the steps of methods 400, 600, and / or 800. As an example, in some embodiments, each of the methods described herein may be performed by a processor executing software stored in a memory. The software may include, for example, program code or computer program modules that include software instructions that can be executed by a processor. In some embodiments, the program code may be a computer program that is stored on a non-transitory computer-readable medium and can be executed by a processor of an associated device.

[0139] The terms "non-transitory computer-readable medium" and "computer-readable medium" include a single medium or multiple media, such as centralized or distributed databases, and / or associated caches and servers that store one or more sets of instructions. In addition, the terms "non-transitory computer-readable medium" and "computer-readable medium" include any tangible medium that is capable of storing, encoding, or carrying a set of instructions for execution by a processor or causing a system to perform one or more of the methods or operations disclosed herein. As used herein, the term "computer-readable medium" is expressly defined to include any type of computer-readable storage device and / or storage disk and excludes propagating signals.

[0140] Any process descriptions or blocks in the figures should be understood to represent modules, segments or portions of code, which include one or more executable instructions for implementing specific logical functions or steps in the process, and alternative implementations are included within the scope of the embodiments of the present invention, in which functions may be performed in an order different from that shown or discussed, including substantially simultaneously or in reverse order, depending on the functions involved, as will be understood by a person of ordinary skill in the art.

[0141] It should be understood that the examples disclosed herein may refer to computing devices and / or systems having components that may or may not be in physical proximity to one another. Certain embodiments may take the form of cloud-based systems or devices, and the term "computing device" should be understood to include distributed systems and devices (such as those that are cloud-based), as well as software, firmware, and other components configured to perform one or more of the functions described herein. In addition, as mentioned above, one or more features of the computing device may be physically remote (e.g., a separate microphone) and may be communicatively coupled to the computing device.

[0142] It should be noted that in the description and drawings, similar or substantially similar elements may be marked with the same reference numerals. However, sometimes these elements may be marked with different numbers, for example, in cases where such markings facilitate a clearer description. In addition, the drawings set forth herein are not necessarily drawn to scale, and in some examples, the proportions may be exaggerated to more clearly depict certain features. Such markings and drawing practices do not necessarily imply a potential substantive purpose. As stated above, the specification is intended to be understood as a whole and is interpreted according to the principles of the present invention as taught herein and understood by a person of ordinary skill in the art.

[0143] In this disclosure, the use of disjunctives is intended to include conjunctions. The use of definite or indefinite articles is not intended to indicate cardinality. In particular, reference to "the" object or "a" or "an" object is also intended to mean one of a plurality of possible such objects.

[0144] The present disclosure describes, illustrates and illustrates one or more specific embodiments of the present invention according to its principles. The present disclosure is intended to explain how to design and use various embodiments according to the technology, rather than to limit its true, expected and fair scope and spirit. That is, the foregoing description is not intended to be exhaustive or limited to the precise form disclosed herein, but to explain and teach the principles of the present invention in such a way that a person of ordinary skill in the art can understand these principles, and in the case of understanding the principles, it can be applied to not only practice the embodiments described herein, but also practice other embodiments that may be thought of according to these principles. The embodiments provided herein are selected and described to provide the best description of the principles of the described technology and its practical application, and enable a person of ordinary skill in the art to utilize the technology in various embodiments and various modifications suitable for the specific purpose under consideration. All such modifications and changes are within the scope of the embodiments determined by the attached claims and all equivalents thereof that may be modified during the pending period of this patent application, and the scope is determined when interpreted according to the breadth that the claims are fairly, legally and fairly enjoyed.

Claims

1. A method performed by one or more processors in communication with each of a first microphone, a second microphone, and a camera, the method comprising: determining, using a first microphone array and based on audio associated with the speaker, a first talker position in a first coordinate system relative to the first microphone array; determining, using a second microphone array and based on the audio associated with the speaker, a second talker position in a second coordinate system relative to the second microphone array; determining an estimated speaker position in a third coordinate system relative to the camera based on the first speaker position and the second speaker position; and The estimated speaker position in the third coordinate system is transmitted to the camera so that the camera points an image capture component of the camera toward the estimated speaker position.

2. The method according to claim 1, further comprising: transforming the first speaker position from the first coordinate system to the second coordinate system; determining an estimated speaker position in the second coordinate system based on the first speaker position and the second speaker position in the second coordinate system; and The estimated speaker position is transformed from the second coordinate system to the third coordinate system.

3. The method according to claim 1, further comprising: transforming the first speaker position from the first coordinate system to the third coordinate system; transforming the second speaker position from the second coordinate system to the third coordinate system; and The estimated speaker position in the third coordinate system is determined based on the first speaker position and the second speaker position in the third coordinate system.

4. The method of claim 1 , wherein determining the estimated speaker location comprises: A common point is identified based on the first speaker location and the second speaker location.

5. The method of claim 1 , wherein determining the estimated speaker location comprises: A closest point is identified based on the first speaker location and the second speaker location.

6. The method of claim 1 , wherein determining the first speaker location comprises: The location of sounds generated in the vicinity of the first microphone array is determined using an audio localization algorithm executed by an audio activity locator.

7. A system comprising: a first microphone array configured to determine a first talker position in a first coordinate system relative to the first microphone array based on audio associated with the talker; a second microphone array configured to determine a second talker position in a second coordinate system relative to the second microphone array based on the audio associated with the talker; a camera including an image capture component; and one or more processors communicatively coupled to each of the first microphone array, the second microphone array, and the camera, the one or more processors configured to: determining an estimated speaker position in a third coordinate system relative to the camera based on the first speaker position and the second speaker position; and transmitting the estimated speaker position in the third coordinate system to the camera, Wherein the camera is configured to point the image capture component at the estimated speaker location received from the one or more processors.

8. The system of claim 7, wherein the one or more processors are further configured to: transforming the first speaker position from the first coordinate system to the second coordinate system; determining an estimated speaker position in the second coordinate system based on the first speaker position and the second speaker position in the second coordinate system; and The estimated speaker position is transformed from the second coordinate system to the third coordinate system.

9. The system of claim 7, wherein the one or more processors are further configured to: transforming the first speaker position from the first coordinate system to the third coordinate system; transforming the second speaker position from the second coordinate system to the third coordinate system; and The estimated speaker position in the third coordinate system is determined based on the first speaker position and the second speaker position in the third coordinate system.

10. The system of claim 7, wherein determining the estimated speaker location comprises identifying a common point based on the first speaker location and the second speaker location.

11. The system of claim 7, wherein determining the estimated speaker location comprises identifying a closest point based on the first speaker location and the second speaker location.

12. The system of claim 7, further comprising an audio activity locator, wherein determining the first talker location comprises determining a location of a sound generated proximate to the first microphone array using an audio localization algorithm executed by the audio activity locator.

13. The system of claim 7, wherein the camera is configured to point the image capture component toward the estimated speaker location by adjusting one or more of the angle, tilt, zoom, and framing of the camera.

14. A non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processors in communication with each of a first microphone array, a second microphone array, and a camera, cause the one or more processors to perform: determining, using the first microphone array and based on audio associated with the speaker, a first talker position in a first coordinate system relative to the first microphone array; determining, using the second microphone array and based on the audio associated with the speaker, a second talker position in a second coordinate system relative to the second microphone array; determining an estimated speaker position in a third coordinate system relative to a camera based on the first speaker position and the second speaker position; and The estimated speaker position in the third coordinate system is transmitted to the camera to direct the camera to point an image capture component of the camera toward the estimated speaker position.

15. The non-transitory computer-readable storage medium of claim 14, further comprising instructions that cause the one or more processors to execute: transforming the first speaker position from the first coordinate system to the second coordinate system; determining an estimated speaker position in the second coordinate system based on the first speaker position and the second speaker position in the second coordinate system; and The estimated speaker position is transformed from the second coordinate system to the third coordinate system.

16. The non-transitory computer-readable storage medium of claim 14, further comprising instructions that cause the one or more processors to execute: transforming the first speaker position from the first coordinate system to the third coordinate system; transforming the second speaker position from the second coordinate system to the third coordinate system; and The estimated speaker position in the third coordinate system is determined based on the first speaker position and the second speaker position in the third coordinate system.

17. The non-transitory computer-readable storage medium of claim 14, wherein determining the estimated speaker location comprises identifying a common point based on the first speaker location and the second speaker location.

18. The non-transitory computer-readable storage medium of claim 14, wherein determining the estimated speaker location comprises identifying a closest point based on the first speaker location and the second speaker location.

19. The non-transitory computer-readable storage medium of claim 14, wherein determining the first talker location comprises determining a location of a sound generated proximate to the first microphone array using an audio localization algorithm executed by an audio activity locator.

20. The non-transitory computer-readable storage medium of claim 14, wherein the camera points the image capture component toward the estimated speaker location by adjusting one or more of an angle, tilt, zoom, and framing of the camera.

Images