Acoustic echo cancellation for multi-microphone conferencing systems
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-14
- Publication Date
- 2026-08-14
Smart Images
Figure CN122575389A_ABST
Abstract
Description
Technical Field
[0001] This application generally relates to video conferencing, and more specifically to acoustic echo cancellation for multi-microphone conferencing systems. Summary of the Invention
[0002] In one or more embodiments, a method is disclosed, comprising: detecting the location of a currently speaking participant in a video conference; selecting one or more audio signals from one or more microphones in a multi-microphone conferencing system based on the location of the currently speaking participant; and providing audio output based on the selected one or more audio signals. Attached Figure Description
[0003] The accompanying drawings, which are incorporated in and form part of this specification, illustrate one or more specific examples and, together with the description of the examples, serve to explain the principles and implementation of the specific examples.
[0004] Figure 1 This demonstrates an example system that provides chat and video conferencing capabilities to various client devices.
[0005] Figure 2 This section showcases example systems where chat and video conferencing providers offer chat and video conferencing capabilities to various client devices.
[0006] Figure 3 This demonstrates an example system that can establish virtual communication sessions.
[0007] Figure 4 Demonstrate the signal flow associated with a multi-microphone conferencing system when remote participants in a video conference are speaking.
[0008] Figure 5 This document presents an example flowchart for adaptive microphone selection (AEC) in a multi-microphone conferencing system.
[0009] Figure 6 This demonstrates an example method for AEC (Automatic Encoding and Conferencing) in multi-microphone conferencing systems.
[0010] Figure 7 An example computing device is shown that is suitable for use in an example system or method of AEC in a multi-microphone conferencing system according to the present disclosure. Detailed Implementation
[0011] This document describes examples in the context of acoustic echo cancellation for multi-microphone conferencing systems. Those skilled in the art will recognize that the following description is illustrative only and is not intended to be limiting in any way. Implementations of the examples illustrated in the accompanying drawings will now be discussed in detail. Throughout the drawings and the following description, the same reference numerals will be used to refer to the same or similar items.
[0012] For clarity, not all the conventional features of the examples described herein are shown or described. Of course, it should be understood that in the development of any such practical implementation, many implementation-specific decisions must be made to achieve the developer’s specific goals, such as complying with constraints related to the application and business, and these specific goals will vary from implementation to implementation and from developer to developer.
[0013] Multi-microphone conferencing systems utilize multiple microphones, located within a single device or distributed across multiple devices, to capture audio signals from local participants in a conference room. These microphones work together to enhance audio quality and extend coverage, ensuring better sound delivery for remote participants. However, acoustic echo cancellation (AEC) presents challenges in multi-microphone systems. Echoes occur when sound played from a speaker unit is picked up by a microphone and transmitted back to remote participants, creating unwanted feedback loops. Traditional AEC techniques typically rely on the audio from the speaker unit as a reference signal to cancel echoes. For these AEC techniques to function effectively, the reference signal needs to precede the signal captured by the microphone. However, this requirement may not always be met in multi-microphone systems. Therefore, traditional AEC techniques are ineffective for multi-microphone conferencing systems.
[0014] To address the AEC challenge in multi-microphone conferencing systems, particularly when the reference signal lags behind the signal captured by the microphones, multi-microphone systems need to intelligently select audio signals from different microphones to mitigate echo.
[0015] For example, a video conferencing setup is installed in a conference room where multiple local participants are present. These participants connect their user devices (e.g., laptops or mobile phones) to the video conferencing setup. The user devices are connected to the setup via a local network. The video conferencing setup includes a built-in microphone. The user devices include their own microphones, which are considered extended microphones for use with the video conferencing setup. The built-in and extended microphones form a multi-microphone system. In other words, the microphones on the user devices in the conference room and the video conferencing setup together form a multi-microphone conferencing system. The video conferencing setup collects audio signals from all the microphones to produce high-quality audio output and delivers this high-quality audio output to remote participants in the video conference. A speaker unit associated with the video conferencing setup plays the audio signals from the remote participants so that local participants in the conference room can hear them. The audio played by the speaker unit is captured as an echo signal by the built-in microphones of the video conferencing setup and the extended microphones of the user devices in the conference room. All the microphones can send the echo signal back to the remote participants, thus creating feedback.
[0016] The video conferencing device includes an AEC module configured to cancel echo signals captured by a built-in microphone. The AEC module uses an audio signal received from a remote participant as a reference signal, which is received before the echo signal captured by the built-in microphone. The AEC module then correlates the echo signal with the reference signal, for example using the Pearson correlation formula, to determine whether the echo signal and the reference signal are linearly related. If so, the AEC module cancels the echo signal.
[0017] However, if the user equipment or its microphones in a multi-microphone conferencing system include an AEC module, such an AEC module may not be as effective at canceling echo signals as the AEC module in a video conferencing device. The reference signal required for echo cancellation by the user equipment is provided by the video conferencing device via the local network. The AEC module on the user equipment can receive the reference signal after the extended microphones have received the echo signal. Therefore, the AEC module in a user equipment using traditional AEC technology will not function properly.
[0018] This disclosure provides microphone selection technology to dynamically adjust microphone usage based on voice activity detection to prevent echo signals from extended microphones. If the video conferencing device detects voice signals from a remote participant, it exclusively uses its built-in microphone to prevent echo signals from extended microphones. That is, the video conferencing device only provides audio signals from the built-in microphone to the remote device. If the video conferencing device does not detect voice signals from a remote participant, it evaluates the previous microphone usage state. If the previous state was that all microphones were in use, the video conferencing device maintains the state of using all microphones. That is, the video conferencing device collects audio signals from all microphones for processing and provides them to the remote participant. If the previous state was that only the built-in microphone was in use, the video conferencing device determines that voice from a local participant is present. If the video conferencing device does not detect voice from a local participant, it maintains the state of using only the built-in microphone. If the video conferencing device detects voice from a local participant, it switches to using all microphones for better audio coverage.
[0019] Therefore, the video conferencing equipment in the conference room dynamically selects which microphone to use to prevent echo signals from the extended microphone, while providing high-quality voice audio from the local participants to remote participants by using the extended microphone.
[0020] This illustrative example is given to introduce the reader to the general topics discussed herein, and this disclosure is not limited to this example. The following sections describe various additional, non-limiting examples and illustrations of acoustic echo cancellation for multi-microphone conferencing systems.
[0021] Now for reference Figure 1 , Figure 1 An example system 100 is shown that provides video conferencing capabilities to various client devices. System 100 includes a chat and video conferencing provider 110 connected to multiple communication networks 120, 130, through which various client devices 140-180 can participate in video conferences hosted by the chat and video conferencing provider 110. For example, the chat and video conferencing provider 110 may be located within a private network to provide video conferencing services to devices within that private network, or it may be connected to a public network, such as the Internet, so that anyone can access it. Some instances may even provide a hybrid model, where the chat and video conferencing provider 110 can provide components that enable private organizations to host private internal video conferences or connect their systems to the chat and video conferencing provider 110 via a public network.
[0022] The system also selectively incorporates one or more authentication and authorization providers, such as authentication and authorization provider 115, which can provide authentication and authorization services to users of client devices 140-160. Authentication and authorization provider 115 can authenticate users of chat and video conferencing provider 110 and manage user authorization for various services provided by chat and video conferencing provider 110. In this example, authentication and authorization provider 115 is operated by a different entity than chat and video conferencing provider 110, but in some instances, they can be the same entity.
[0023] Chat and video conferencing provider 110 allows customers to create video conferences (or “conferences”) and invite others to these conferences, as well as perform other related functions such as recording conferences, generating transcripts of conference audio, generating summaries and translations of conference audio, managing user functions in conferences, enabling SMS during conferences, creating and managing breakout rooms from virtual conferences, and more. Figure 2 The following provides a more detailed description of the architecture and functionality of chat and video conferencing provider 110. It should be understood that the term "conference" includes the meaning of the term "webinars" as used herein.
[0024] This example of a chat and video conferencing provider 110's meeting takes place in a virtual room that the participants connect to. In this case, the room is a structure provided by a server that serves as a common point for receiving various video and audio data, which is then multiplexed and provided to different participants. While "room" is a conceptual label in this disclosure, any suitable functionality that allows multiple people to participate in a shared video conference can be used.
[0025] To create a meeting with chat and video conferencing provider 110, a user can contact the provider using client devices 140-180 and select the option to create a new meeting. This option may be available in a webpage accessed by client devices 140-160 or through a client application executed by client devices 140-160. For telephone devices, an audio menu may be provided to the user, who can navigate by pressing number buttons on their telephone device. To create a meeting, chat and video conferencing provider 110 may prompt the user for information such as the meeting date, time, and duration, the number of participants, the type of encryption to be used, and whether the meeting is confidential or public. After receiving the various meeting settings, the chat and video conferencing provider may create a meeting record and generate a meeting identifier, and in some examples, generate a corresponding meeting password or code (or other authentication information), all of which is provided to the meeting host.
[0026] Upon receiving a meeting message, a user can distribute the message to one or more users to invite them to the meeting. To start the meeting at the scheduled time (or immediately if the meeting is set to start immediately), the host provides a meeting identifier and, where applicable, the appropriate authentication information (e.g., a password or code). The video conferencing system then starts the meeting and allows users to join. Depending on the options set for the meeting, users may join immediately after providing the appropriate meeting identifier (and authentication information, as applicable), even if the host has not yet arrived, or if a message informing users that the meeting has not yet started is presented to them, or if the host may require special permission from one or more users.
[0027] During the meeting, participants can use their client devices 140-180 to capture audio or video information and stream it to the chat and video conferencing provider 110. They can also receive audio or video information displayed on their respective client devices 140 from the chat and video conferencing provider 110, enabling various users to participate in the meeting.
[0028] At the end of the meeting, the host can choose to terminate the meeting, or the meeting can be automatically terminated at the scheduled end time or after a predetermined duration. When the meeting terminates, each participant is disconnected from the meeting and will no longer receive the audio or video stream (and the transmission of audio or video streams will cease). Chat and video conferencing providers can also invalidate meeting information such as meeting identifiers or passwords.
[0029] To provide this functionality, one or more client devices 140-180 may communicate with the chat and video conferencing provider 110 using one or more communication networks, such as network 120 or the public switched telephone network (“PSTN”) 130. Client devices 140-180 may be any suitable computing or communication device with audio or video capabilities. For example, client devices 140-160 may be conventional computing devices connected to the chat and video conferencing provider 110 using the Internet or other suitable computer networks, such as desktop or laptop computers with processors and computer-readable media. Suitable networks include the Internet, any local area network (“LAN”), metropolitan area network (“MAN”), wide area network (“WAN”), cellular networks (e.g., 3G, 4G, 4G LTE, 5G, etc.), or any combination of these networks. Alternatively, other types of computing devices may be used, such as tablets, smartphones, and dedicated video conferencing equipment. These devices may all provide audio and video capabilities and allow one or more users to participate in video conferences hosted by the chat and video conferencing provider 110.
[0030] In addition to the computing devices discussed above, client devices 140-180 may also include one or more telephone devices, such as cellular phones (e.g., cellular phone 170), Internet Protocol (“IP”) phones (e.g., phone 180), or conventional phones. Such telephone devices can allow users to make conventional telephone calls to other telephone devices using the PSTN (including chat and video conferencing provider 110). It should be understood that some computing devices may also provide telephone functionality and can operate as telephone devices. For example, smartphones typically provide cellular phone functionality and therefore can act as telephone devices. Figure 1 The telephone device in the example system 100 shown operates as follows. Additionally, a conventional computing device can execute software to enable telephone functionality, allowing a user to make and receive calls using a headset and microphone, etc. Such software can communicate with a PSTN gateway to route calls from the computer network to the PSTN. Therefore, the telephone device includes any device capable of making conventional telephone calls, and is not limited to dedicated telephone devices like conventional telephones.
[0031] Referring again to client devices 140-160, these devices 140-160 use network 120 to contact chat and video conferencing provider 110 and can provide information to chat and video conferencing provider 110 to access functions provided by chat and video conferencing provider 110, such as accessing to create new meetings or join existing meetings. For this purpose, client devices 140-160 can provide user authentication information, meeting identifiers, meeting passwords or codes, etc. In the example employing authentication and authorization provider 115, client devices, such as client devices 140-160, can operate in conjunction with authentication and authorization provider 115 to provide authentication and authorization information or other user information to chat and video conferencing provider 110.
[0032] The authentication and authorization provider 115 can be any entity trusted by the chat and video conferencing provider 110, which can help authenticate users of the chat and video conferencing provider 110 and authorize users to access the services provided by the chat and video conferencing provider 110. For example, a trusted entity could be a server run by a business or other organization with which the user has created an account (containing authentication and authorization information), such as an employer or a trusted third party. Users can log in to the authentication and authorization provider 115, for example, by providing a username and password, to access their account information at the authentication and authorization provider 115. Account information includes information established and maintained at the authentication and authorization provider 115 that can be used to authenticate and facilitate authorization for a specific user, regardless of what client device they may be using. An example of account information could be an email account created by a user at the authentication and authorization provider 115 and protected by a password or additional security features such as single sign-on, hardware tagging, two-factor authentication, etc. However, such account information may differ from functionalities such as email. For example, a healthcare provider might create accounts for its patients. While the relevant account information may have associated email accounts, the account information is different from those email accounts.
[0033] Therefore, a user's account information involves a secure, verified set of information that can be used for authentication and authorization services for a specific user and can only be accessed by that user. Through proper authentication, the associated user can then verify their identity on other computing devices or services (such as chat and video conferencing provider 110). Authentication and authorization provider 115 may require the user's prior explicit consent to allow chat and video conferencing provider 110 to access the user's account information for authentication and authorization purposes.
[0034] Once user authentication is successful, the authentication and authorization provider 115 can provide the chat and video conferencing provider 110 with information about the services the user is authorized to access. For example, the authentication and authorization provider 115 can store information about user roles associated with a user. User roles can include a set of services provided by the chat and video conferencing provider 110, and users assigned to those user roles are authorized to use those services. Alternatively, more or less granular user authorization methods can be used.
[0035] When a user accesses chat and video conferencing provider 110 using a client device, chat and video conferencing provider 110 communicates with authentication and authorization provider 115 using information provided by the user to verify the user's account information. For example, the user can provide a username or cryptographic signature associated with authentication and authorization provider 115. Authentication and authorization provider 115 then confirms the information provided by the user or rejects the request. Based on this response, chat and video conferencing provider 110 accordingly grants or denies access to its services.
[0036] For telephone devices such as client devices 170-180, users can dial the telephone chat and video conferencing provider 110 to access video conferencing services. After answering the call, the user can provide information about the video conference, such as a conference identifier (“ID”), password, or passcode, to allow the telephone device to join the conference and participate using the telephone device’s audio devices (e.g., microphone and speaker), even if the telephone device does not have video capabilities.
[0037] Because telephone devices are typically more limited in functionality than regular computing devices, they may not be able to provide certain information to the chat and video conferencing provider 110. For example, a telephone device may not be able to provide authentication information to the chat and video conferencing provider 110 to authenticate the telephone device or user. Therefore, the chat and video conferencing provider 110 can offer more limited functionality to such telephone devices. For example, a user may be allowed to join a meeting after providing meeting information, such as a meeting identifier and password, but only as an anonymous participant. In some examples, this might limit their ability to interact with the meeting, such as limiting their ability to speak, hear or view certain content shared during the meeting, or access other meeting features, such as joining breakout rooms or text chatting with other participants in the meeting.
[0038] It should be understood that even when a user is able to authenticate and uses a client device capable of authenticating with the chat and video conferencing provider 110, the user may choose to participate in meetings anonymously and refuse to provide account information to the chat and video conferencing provider 110. The chat and video conferencing provider 110 may determine whether to allow such anonymous users to use the services provided by the chat and video conferencing provider 110. Anonymous users using telephone devices, as discussed above, will be restricted regardless of the reason for their anonymity, and in some cases, may be prevented from accessing certain meetings or other services, or may be completely prevented from accessing the chat and video conferencing provider 110.
[0039] Referring again to chat and video conferencing provider 110, in some examples, it can allow client devices 140-160 to encrypt their respective video and audio streams to help improve their privacy during meetings. Encryption can be provided between client devices 140-160 and chat and video conferencing provider 110, or it can be provided in an end-to-end configuration, where multimedia streams (e.g., audio or video streams) transmitted by client devices 140-160 are not decrypted until they are received by another client device 140-160 participating in the meeting. Encryption can also be provided only for portions of the communication; for example, unencrypted communication across borders can be encrypted.
[0040] Client-to-server encryption can be used to protect communication between client devices 140-160 and the chat and video conferencing provider 110, while allowing the provider to access the decrypted multimedia stream to perform certain processes, such as recording the meeting for participants or generating meeting minutes. End-to-end encryption can be used to ensure complete confidentiality of the meeting to participants without worrying about the provider gaining access to sensitive content. Any suitable encryption method can be used, including key-pair encryption of the stream. For example, to provide end-to-end encryption, the meeting host's client device can obtain the public key of every other client device participating in the meeting and securely exchange a set of keys to encrypt and decrypt multimedia content transmitted during the meeting. Thus, client devices 140-160 can communicate securely with each other during the meeting. Furthermore, in some examples, certain types of encryption may be limited by the types of devices participating in the meeting. For example, telephone devices may lack the capability to encrypt and decrypt multimedia streams. Therefore, while encryption of multimedia streams may be desirable in many cases, it is not necessary, as it may prevent some users from participating in the meeting.
[0041] By using Figure 1The example system shown allows users to create and participate in meetings via chat and video conferencing provider 110 using their respective client devices 140-180. Furthermore, this system enables users to utilize a wide variety of client devices 140-180, ranging from traditional standards-based video conferencing hardware to dedicated video conferencing equipment, laptops or desktops, handheld devices, and traditional telephone devices.
[0042] Now for reference Figure 2 , Figure 2 In the example system 200 shown, chat and video conferencing provider 210 provides video conferencing functionality to various client devices 220-250. Client devices 220-250 include two conventional computing devices 220-230, a dedicated device for the video conferencing room 240, and a telephone device 250. Each client device 220-250 communicates with chat and video conferencing provider 210 via a communication network, such as the Internet for client devices 220-240 or the PSTN for client device 250, as described above. Figure 1 As described above. Chat and video conferencing provider 210 also communicates with one or more authentication and authorization providers 215, which can authenticate various users of chat and video conferencing provider 210, roughly as described above. Figure 1 As stated above.
[0043] In this example, chat and video conferencing provider 210 utilizes multiple different servers (or groups of servers) to provide various video conferencing functionality examples, enabling various client devices to create and participate in video conferences. Chat and video conferencing provider 210 uses one or more real-time media servers 212, one or more network service servers 214, one or more video room gateways 216, one or more messaging and online status gateways 217, and one or more telephone gateways 218. Each of these servers 212-218 is connected to one or more communication networks, enabling them to collectively provide client devices 220-250 with access to and participation rights in one or more video conferences.
[0044] Real-time media server 212 provides multiplexed multimedia streams to conference participants, such as Figure 2The client devices 220-250 are shown. Although video and audio streams typically originate from the respective client devices, they are transmitted from client devices 220-250 to chat and video conferencing provider 210 via one or more networks, where they are received by real-time media server 212. Real-time media server 212 determines which protocol is optimal based on factors such as proxy settings and the presence of firewalls. For example, client devices can choose audio and video over UDP, TCP, TLS, or HTTPS, and screen sharing over UDP.
[0045] Real-time media server 212 then multiplexes various video and audio streams based on the target client devices and delivers the multiplexed streams to each client device. For example, real-time media server 212 receives audio and video streams from client devices 220-240, while receiving only the audio stream from client device 250. Real-time media server 212 then multiplexes the streams received from devices 230-250 and provides the multiplexed streams to client device 220. Real-time media server 212 is adaptive in how it provides these streams, for example, reacting to real-time network and client changes. For example, real-time media server 212 can monitor parameters such as client bandwidth, CPU usage, memory, and network I / O, as well as network parameters such as packet loss, latency, and jitter, to determine how to modify the way the streams are provided.
[0046] Client device 220 receives the stream, performs any decryption, decoding, and demultiplexing on the received stream, and then outputs audio and video using the client device's video and audio devices. In this example, the real-time media server does not multiplex its own video and audio signals when sending the stream to client device 220. Instead, each client device 220-250 only receives multimedia streams from other client devices 220-250. For telephone devices lacking video capabilities, such as client device 250, the real-time media server 212 only transmits the multiplexed audio stream. Client device 220 can receive multiple streams for a specific communication, allowing client device 220 to switch between streams to provide a higher quality of service.
[0047] In addition to multiplexing multimedia streams, in some examples, the real-time media server 212 can also decrypt incoming multimedia streams. As described above, multimedia streams can be encrypted between client devices 220-250 and chat and video conferencing provider 210. In some such examples, the real-time media server 212 can decrypt incoming multimedia streams, appropriately multiplex multimedia streams for various clients, and encrypt the multiplexed streams for transmission.
[0048] As mentioned above Figure 1The chat and video conferencing provider 210 can provide certain functions regarding unencrypted multimedia streams upon user request. For example, a conference host can request to record the meeting or prepare to transcribe an audio stream, which can then be performed by the real-time media server 212 using the decrypted multimedia stream. Alternatively, the recording or transcription function can be offloaded to a dedicated server (or multiple servers), such as a cloud recording server, for recording audio and video streams. In some examples, the chat and video conferencing provider 210 can allow conference participants to notify it of inappropriate behavior or content during the meeting. Such a notification can trigger the real-time media server 212 to record portions of the meeting for review by the chat and video conferencing provider 210. Other functions can also be implemented to take action based on the decrypted multimedia stream at the chat and video conferencing provider, such as monitoring video or audio quality, adjusting or changing media encoding mechanisms, etc.
[0049] It should be understood that multiple real-time media servers 212 can participate in data communication for a single conference, and multimedia streams can be routed through multiple different real-time media servers 212. Furthermore, the various real-time media servers 212 may not be located in the same location, but rather in multiple different geographical locations, which enables high-quality communication between clients distributed across a wide geographical area (e.g., in different countries or continents). Additionally, in some examples, one or more of these servers may co-located at the client's premises, such as a business or other organization. For example, different geographical regions may each have one or more real-time media servers 212, enabling client devices in the same geographical region to have a high-quality connection with the chat and video conferencing provider 210 via the local server 212 to send and receive multimedia streams, rather than connecting to real-time media servers located in different countries or continents. The local real-time media server 212 can then use high-speed network infrastructure (e.g., the internet backbone) to communicate with physically distant servers that might otherwise be unavailable directly to client devices 220-250. Therefore, routing multimedia streams can be distributed throughout the video conferencing system and across many different real-time media servers 212.
[0050] Regarding the network service servers 214, these servers 214 provide management functions to enable client devices to create or participate in meetings, send meeting invitations, create or manage user accounts or subscriptions, and other related functions. Furthermore, these servers can be configured to perform different functions or operate at different tiers; for example, they can manage certain functions of the chat and video conferencing provider with the help of a supervisory server set for a specific region or location. When client devices 220-250 access the chat and video conferencing provider 210, they typically communicate with one or more network service servers 214 to access their accounts or participate in meetings.
[0051] In this example, when client devices 220-250 first contact chat and video conferencing provider 210, the client devices are routed to network service server 214. The client devices can then provide access credentials, such as a username and password or single sign-on credentials, to gain authenticated access to chat and video conferencing provider 210. This process may involve network service server 214 contacting authentication and authorization provider 215 to verify the provided credentials. Once the user's credentials are accepted and the user has consented, network service server 214 can perform administrative functions by interacting with network service server 214, such as updating user account information if the user has account information stored in chat and video conferencing provider 210, or scheduling new meetings. Authentication and authorization provider 215 can be used to determine which administrative functions a specific user can access based on assigned roles, permissions, groups, etc.
[0052] In some examples, users can anonymously access chat and video conferencing provider 210. When communicating anonymously, client devices 220-250 can communicate with one or more network service servers 214, but only provide information for creating or joining a meeting, depending on which features the chat and video conferencing provider allows anonymous users to use. For example, an anonymous user can use client device 220 to access the chat and video conferencing provider and provide a meeting ID and password. Network service server 214 can use the meeting ID to identify an upcoming or ongoing meeting and verify that the password for the meeting ID is correct. After verification, network service server 214 can then transmit information to client device 220 so that client device 220 can join the meeting and communicate with the appropriate real-time media server 212.
[0053] When a user wishes to schedule a meeting, the user (anonymous or authenticated) can choose to schedule a new meeting and then select various meeting options, such as the meeting date and time, meeting duration, encryption type to use, one or more users to invite, privacy controls (e.g., disallowing anonymous users, blocking screen sharing, manual authorization to join the meeting, etc.), meeting recording options, etc. The network service server 214 can then create and store the meeting records for the scheduled meeting. When the scheduled meeting time arrives (or within a pre-defined threshold time period), the network service server 214 can accept requests from various users to join the meeting.
[0054] To process requests to join a meeting, network service server 214 can receive meeting information (such as a meeting ID and password) from one or more client devices 220-250. Network service server 214 locates the meeting record corresponding to the provided meeting ID, then verifies whether the scheduled start time of the meeting has arrived, whether the meeting host has started the meeting, and whether the password matches the password in the meeting record. If the request was made by the host, network service server 214 activates the meeting and connects the host to real-time media server 212 so that the host can begin sending and receiving multimedia streams.
[0055] Once the host starts the meeting, if the meeting record is found and the password matches the password provided by the requesting client devices 220-250, subsequent users requesting access are allowed to join the meeting. In some examples, additional access control may also be used. However, if the network service server 214 determines that the requesting client devices 220-250 are allowed to participate in the meeting, the network service server 214 identifies the live media server 212 to process multimedia streams entering and leaving the requesting client devices 220-250 and provides the client devices 220-250 with information to connect to the identified live media server 212. When additional client devices 220-250 request access through the network service server 214, they can be added to the meeting.
[0056] After joining the meeting, client devices will send and receive multimedia streams via real-time media server 212, but they can also communicate with network service server 214 as needed during the meeting. For example, if the meeting host leaves the meeting, network service server 214 can designate another user as the new meeting host and assign host management permissions to that user. The host may have management permissions that allow them to manage the meeting, such as enabling or disabling screen sharing, muting or removing users from the meeting, assigning or moving users to or from the main meeting room or breakout rooms (if any), recording the meeting, etc. Such functions can be managed by network service server 214.
[0057] For example, if a host wants to remove a user from the meeting, they can select the user and issue the command through the user interface on their client device. The command can be sent to the network service server 214, which can then disconnect the selected user from the corresponding live media server 212. If the host wants to remove one or more participants from the meeting, such a command can also be processed by the network service server 214, which can terminate the authorization for one or more participants to join the meeting.
[0058] In addition to creating and managing ongoing meetings, network service server 214 can also be responsible for closing and canceling meetings after they end. For example, the meeting host can send a command to network service server 214 to end an ongoing meeting. Network service server 214 can then remove any remaining participants from the meeting, communicate with one or more real-time media servers 212 to stop streaming audio and video to the meeting, and deactivate (e.g., by deleting the corresponding password from the meeting records) or delete the meeting records associated with the meeting. Therefore, if a user later attempts to access the meeting, network service server 214 can refuse that request.
[0059] Depending on the features offered by the chat and video conferencing provider, the network service server 214 can provide additional functionalities, such as providing private meeting capabilities for organizations, special types of meetings (e.g., webinars), etc. Such functionalities can be provided based on various examples of video conferencing providers according to this specification.
[0060] Referring now to video room gateway servers 216, these servers 216 provide interfaces between dedicated video conferencing hardware, such as that used in dedicated video conference rooms. Such video conferencing hardware may include one or more cameras and microphones, and a computing device for receiving video and audio streams from each camera and microphone and connecting to chat and video conferencing provider 210. For example, the video conferencing hardware may be provided by the chat and video conferencing provider to one or more subscribers who may provide access credentials to the video conferencing hardware for connecting to chat and video conferencing provider 210.
[0061] Video room gateway server 216 provides dedicated authentication and communication with dedicated video conferencing hardware, which may not be applicable to other client devices 220-230, 250. For example, the video conferencing hardware may register with the chat and video conferencing provider upon initial installation, and the video room gateway can use this registration, along with information provided to the video room gateway server when the dedicated video conferencing hardware connects to the video room gateway server 216 (such as device ID information, subscriber information, hardware capabilities, hardware version information, etc.), to authenticate the video conferencing hardware. After receiving such information and authenticating the dedicated video conferencing hardware, the video room gateway server 216 can interact with network service server 214 and real-time media server 212 to allow the video conferencing hardware to create or join meetings hosted by chat and video conferencing provider 210.
[0062] Now refer to telephone gateway server 218, which supports and facilitates telephone devices' participation in conferences hosted by chat and video conferencing provider 210. Because telephone devices use PSTN communication instead of computer networking protocols such as TCP / IP, telephone gateway server 218 acts as an interface for conversion between PSTN and the network system used by chat and video conferencing provider 210.
[0063] For example, if a user connects to a conference using a telephone device, they can dial a telephone gateway server 218 corresponding to a chat and video conferencing provider. The telephone gateway server 218 will answer the call and generate audio information requesting the user to provide information such as a conference ID and password. The user can enter this information using buttons on their telephone device, for example, by sending a dual-tone multi-frequency (“DTMF”) audio stream to the telephone gateway server 218. The telephone gateway server 218 determines the number or letters entered by the user and provides the conference ID and password information along with a request to join or start the conference to the network service server 214, roughly as described above. Once the telephone client device 250 is accepted to join the conference, the telephone gateway server joins the conference on behalf of the telephone device.
[0064] After joining the conference, the telephone gateway server 218 receives the audio stream from the telephone device and provides it to the corresponding real-time media server 212. It then receives the audio stream from the real-time media server 212, decodes it, and provides the decoded audio to the telephone device. Therefore, the telephone gateway server 218 essentially operates as a client device, while the telephone device primarily functions as an input / output device for the corresponding telephone gateway server 218, such as a microphone and speaker, enabling users of the telephone device to participate in the conference without using a computing device or video.
[0065] It should be understood that the components of the chat and video conferencing provider 210 discussed above are merely examples and example architectures of such devices. Some video conferencing providers may offer more or fewer features than described above, and may not distribute features across the different types of servers mentioned. Instead, any suitable server and network architecture can be used depending on the specific example.
[0066] Now for reference Figure 3 , Figure 3 An example system 300 is shown that can establish virtual communication sessions. In this example system 300, a communication platform 310 and multiple client devices 340A-340N (which may be individually referred to as client devices 340 or collectively referred to as client devices 340 herein) are connected via a network 320. The communication platform 310 may be... Figure 1 Chat and video conferencing providers in China 110 or Figure 2The chat and video conferencing provider 210. The network 320 may be the Internet or may employ any suitable communication network or combination of communication networks, including LAN (e.g., within a corporate private LAN), WAN, MAN, cellular network (e.g., 3G, 4G, 4G LTE, 5G, etc.) or any combination thereof.
[0067] Client device 340 can be any suitable computing or communication device. Figure 1 Client devices (e.g., 140, 150, 160, or 170) or Figure 2 The client devices (e.g., 220, 230, or 250) are connected to the communication platform 310 via the Internet or other suitable computer networks. For example, client device 340 may be a desktop computer, laptop computer, tablet computer, or smartphone with a processor and computer-readable media connected to the communication platform 310. Client devices 340 are equipped with communication software that enables them to connect to the communication platform 310 for chat, video conferencing, email, and any other suitable communication. For example, during a chat session, a user associated with a client device (e.g., client device 340A) can interact with other users associated with other client devices (e.g., client devices 340B-340N) via the communication platform 310 by sending and receiving chat messages and responding to received chat messages.
[0068] Now for reference Figure 4 , Figure 4 This demonstrates the signal flow associated with a multi-microphone conferencing system when a remote participant in a video conference is speaking. A meeting room is equipped with a video conferencing setup that includes a built-in microphone. One or more local participants and their user devices, such as laptops or mobile devices, are present in the meeting room. The user devices are connected to the video conferencing setup via a local network. One or more user devices may also include microphones. These microphones included in the user devices are extension microphones of the video conferencing setup. Therefore, the video conferencing setup and the extension microphones from the user devices form a multi-microphone conferencing system in the meeting room, which can be referred to as a multi-microphone conference room.
[0069] exist Figure 4In the diagram, solid arrows represent signal transmission through a communication network used for video conferencing or a local area network. Dashed arrows indicate sound captured by the built-in microphone at the video conferencing device in the conference room or by the extended microphone at the user device. Audio signals from remote participants (referred to as far signals or far speech) are sent to the video conferencing device. Audio signals from local participants in the conference room are referred to as near signals or near speech. After receiving the far signal, the video conferencing device plays the audio through its speakers, allowing local participants to hear the remote participants. Simultaneously, the played audio is captured by the built-in and extended microphones in a multi-microphone conference room.
[0070] Without any AEC at the user device or video conferencing device, the far-field signals captured by all microphones will be sent back to the remote participants, causing them to hear their own voices as echoes. This echo interferes with communication and degrades the user experience.
[0071] Traditional AEC (Asynchronous Echo Control) techniques require the reference signal to arrive before the corresponding echo is canceled. This requirement is always met in single-microphone systems because the system only plays audio after it has been received, ensuring that any echo captured by the microphone is later than the received audio (i.e., the reference signal). Such systems are called causal systems. However, in multi-microphone systems, this requirement may not always be met. For example, the user equipment may receive the reference signal from the video conferencing device after an extended microphone at the user equipment has captured an echo from the speaker at the video conferencing device. Therefore, traditional AEC techniques do not work for multi-microphone conferencing systems.
[0072] like Figure 4 As illustrated, in step 1, the remote signal is transmitted to the video conferencing device. In step 2, the video conferencing device receives the remote signal and plays audio through a speaker device, allowing local participants to hear remote participants. In step 3, the played audio is captured by the built-in microphone of the video conferencing device and an extended microphone in the conference room. In step 4, the video conferencing device transmits a reference signal to the extended microphone. Figure 4 In the example audio follow-up, the video conferencing device itself operates as a causal system because step 3 always occurs after step 2. However, the extended microphone can operate as a non-causal system because the order of steps 3 and 4 is not fixed but depends on network conditions. In some examples, the extended microphone captures the echo before receiving the reference signal, making it a non-causal system where conventional AEC techniques fail.
[0073] See now Figure 5 , Figure 5This example flowchart illustrates adaptive microphone selection for AEC in a multi-microphone conferencing system. The video conferencing apparatus of the multi-microphone conferencing system includes an AEC module configured to cancel echo signals captured by the built-in microphones of the video conferencing apparatus. In this example, there is no AEC module on the user equipment in the meeting room. However, in some other examples, user equipment connected to the video conferencing apparatus may include an AEC module. The video conferencing apparatus also includes one or more Voice Activity Detection (VAD) modules. In some examples, the video conferencing apparatus includes a near-speech VAD module configured to detect speech signals from a local participant and a far-speech VAD module configured to detect speech signals from a remote participant. In some examples, the video conferencing apparatus includes a single VAD module to determine whether the audio signal originates from a remote or local participant. In some examples, one or more VAD modules use or implement deep learning algorithms, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), or transformer models, to detect whether the speech originates from a local or remote participant. In some examples, the VAD module uses feature extraction or statistical methods.
[0074] Video conferencing equipment includes a microphone selection module configured for dynamic microphone selection. In some examples, a selected microphone means that audio signals from the selected microphone are collected for processing and provided to other devices, while audio signals from unselected microphones are not collected. In some examples, a selected microphone indicates that the selected microphone is enabled and not muted, while unselected microphones are disabled or muted. Figure 5 As shown, if one or more VAD modules detect far-field speech, the microphone selection module selects the built-in microphone of the video conferencing device. That is, the microphone selection module selects only the audio signal from the built-in microphone to prevent far-field echo problems. Alternatively or additionally, the microphone selection module disables or mutes the extended microphone. If no far-field speech is detected, the microphone selection module evaluates the previous state of microphone usage. If the previous state of microphone usage was not that only the built-in microphone was in use—in other words, all microphones were in use—the microphone selection module maintains the previous state, i.e., all microphones are in use. If the previous state was that only the built-in microphone was in use, the microphone selection module checks if one or more VAD modules detect near-field speech. If near-field speech is detected, the microphone selection module switches to using all microphones. If no near-field speech is detected, the microphone selection module maintains the previous state of only the built-in microphone being in use.
[0075] The microphone selection module provides the audio signal from the selected microphone to the audio processing module, which then delivers the processed audio signal to the remote participants. In some examples, the audio processing module includes an AEC module for eliminating echo signals captured by the built-in microphone of the video conferencing device. The audio processing module uses or implements a smoothing function, such as a cosine function, to smooth the audio signal from the selected microphone, ensuring the audio signal is natural and does not disrupt the user experience. For example, during a microphone switching process from using only the built-in microphone to using all microphones, including the built-in and extended microphones, the smoothing region includes the time period before and after the switch. The smoothing function blends the audio signals from the built-in microphone and the audio signals from the extended microphones together.
[0076] See now Figure 6 , Figure 6 This demonstrates an example method for AEC (Active Electronic Conferencing) in a multi-microphone conferencing system. At block 602, the video conferencing device detects the location of the currently speaking participant during a video conference. For example, the device determines whether the currently speaking participant is located at a remote location or locally on the video conferencing device, as described above regarding... Figure 5 The discussion focuses on video conferencing. In some examples, a video conferencing device is installed in a conference room and connected to one or more user devices associated with one or more local participants in the conference room. The video conferencing device includes a built-in microphone, and the one or more user devices include one or more user microphones. One or more remote client devices associated with one or more remote participants and the video conferencing device are connected to the video conference. At block 604, the video conferencing device selects one or more audio signals from one or more microphones of a multi-microphone conferencing system for processing based on the location of the currently speaking participant. The multi-microphone conferencing system includes a built-in microphone in the video conferencing device or one or more user microphones associated with user devices connected to the video conferencing device in the conference room. At block 606, the video conferencing device provides audio output based on the selected one or more audio signals. As in the example above, the audio output is sent to one or more remote client devices in the video conference.
[0077] See now Figure 7 , Figure 7 An example computing device 700 suitable for use in an AEC example system or method of a multi-microphone conferencing system according to this disclosure is shown. The example computing device 700 includes a processor 710 that communicates with a memory 720 and other components of the computing device 700 using one or more communication buses 702. The processor 710 is configured to execute processor-executable instructions stored in the memory 720 to perform one or more methods of AEC in a multi-microphone conferencing system, such as those described above, according to various examples. Figure 6 This describes part or all of the example method 600. In some embodiments, the computing device may include software 760 for performing one or more methods described herein, such as one or more steps of method 600, for example. In this example, the computing device 700 also includes one or more user input devices 750, such as a keyboard, mouse, touchscreen, microphone, etc., to accept user input. The computing device 700 also includes a display 740 to provide visual output to the user.
[0078] In addition, the computing device 700 includes software 760 that enables users to join and participate in one or more virtual spaces or one or more meetings (such as traditional meetings or webinars) by: receiving multimedia streams from a virtual meeting provider, sending multimedia streams to a virtual meeting provider, joining and leaving breakout rooms, creating video conference presentations, etc., as described in this disclosure.
[0079] The computing device 700 also includes a communication interface 630. In some examples, the communication interface 730 may use one or more networks including: a local area network (“LAN”); a wide area network (“WAN”), such as the Internet; a metropolitan area network (“MAN”); a point-to-point or peer-to-peer connection; and so on. Any suitable network protocol may be used to achieve communication with other devices. For example, a suitable network protocol may include the Internet Protocol (“IP”), Transmission Control Protocol (“TCP”), User Datagram Protocol (“UDP”), or combinations thereof, such as TCP / IP or UDP / IP.
[0080] While some examples of the methods and systems described herein are as software executed on various machines, these methods and systems can also be implemented as specially configured hardware, such as field-programmable gate arrays (FPGAs) specifically designed to perform the various methods according to this disclosure. For example, examples can be implemented in digital electronic circuits or in computer hardware, firmware, software, or a combination thereof. In one example, the apparatus may include one or more processors. The processor includes computer-readable media, such as random access memory (RAM) coupled to the processor. The processor executes computer-executable program instructions stored in the memory, such as executing one or more computer programs. Such processors may include microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and state machines. Such processors may also include programmable electronic devices, such as PLCs, programmable interrupt controllers (PICs), programmable logic devices (PLDs), programmable read-only memory (PROMs), electronically programmable read-only memory (EPROMs or EEPROMs), or other similar devices.
[0081] Such processors may include or be able to communicate with media, such as one or more non-transitory computer-readable media that may store processor-executable instructions that, when executed by the processor, may cause the processor to perform or assist in the methods according to this disclosure. Examples of non-transitory computer-readable media may include, but are not limited to, electrical, optical, magnetic, or other storage devices capable of providing processor-executable instructions to a processor (such as a processor in a web server). Other examples of non-transitory computer-readable media include, but are not limited to, floppy disks, CD-ROMs, magnetic disks, memory chips, ROMs, RAMs, ASICs, configured processors, all optical media, all magnetic tapes or other magnetic media, or any other media from which a computer processor may read. The described processor and processing may reside in one or more structures and may be distributed across one or more structures. The processor may include code to perform methods (or portions thereof) according to this disclosure.
[0082] The foregoing description of some examples is presented for illustrative and descriptive purposes only and is not intended to be exhaustive or to limit this disclosure to its precise form. Many modifications and adjustments will be apparent to those skilled in the art without departing from the spirit and scope of this disclosure.
[0083] References to examples or implementations herein imply that a particular feature, structure, operation, or other characteristic described in connection with an example may be included in at least one implementation of this disclosure. This disclosure is not limited to the particular example or implementation described in this way. The phrases “in an example,” “in one example,” “in one implementation,” or “in one implementation,” or variations thereof, appearing throughout the specification, do not necessarily refer to the same example or implementation. Any particular feature, structure, operation, or other characteristic described in this specification with respect to an example or implementation may be combined with other features, structures, operations, or other characteristics described with respect to any other example or implementation.
[0084] The word “or” as used in this article is intended to encompass both inclusive and exclusive “or” conditions. In other words, A or B or C includes any or all of the following alternative combinations suitable for a particular purpose: A only; B only; C only; A and B only; A and C only; B and C only; and A and B and C.
Claims
1. A method comprising: Detect the position of the participant currently speaking in a video conference; Select one or more audio signals from one or more microphones in a multi-microphone conferencing system based on the location of the currently speaking participant; as well as Audio output is provided based on the selected one or more audio signals.