Audio control method, terminal, storage medium, and computer program product

Abstract

The application provides an audio control method, a terminal, a storage medium and a computer program product. The method comprises the following steps: acquiring a first distance between a current sound source and each target terminal in a multimedia session; the target terminal is an access terminal located in the same physical space as the sound source, and the target terminal comprises a first terminal; determining a state of the first terminal in the multimedia session based on the first distance, wherein the state comprises a collection state and a mute state; if the first terminal is in the collection state, processing a collected audio signal, and uploading the processed collected audio signal to the multimedia session; if the first terminal is in the mute state, stopping uploading the collected audio signal to the multimedia session. The application adopts multi-target terminal cooperative audio control, dynamically allocates a collection role according to the distance between the sound source and each terminal, makes the terminal close to the sound source responsible for collecting and uploading the audio signal, and makes other terminals stop uploading, thereby avoiding the mutual interference and howling risk of the audio signal caused by simultaneous collection and uploading of multiple terminals.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to an audio control method, terminal, storage medium, and computer program product. Background Technology

[0002] With the widespread adoption of remote work and video conferencing, multi-person online meetings have become a common work mode. In these meetings, each terminal independently captures and plays audio. However, in practice, it's common for multiple terminals in the same physical space (such as a conference room) to simultaneously access the same audio / video conference. This simultaneous audio capture and playback by multiple terminals in the same physical space can cause echo and feedback problems, severely impacting the meeting and user experience.

[0003] To suppress echo and feedback issues in multi-person online conferences, related technologies typically employ Acoustic Echo Cancellation (AEC) for audio processing at each participating terminal. However, echo and feedback in multi-person online conferences are usually caused by complex coupling between the microphones and speakers of multiple terminals, which AEC methods struggle to handle. Furthermore, poor synchronization of audio signals between multiple terminals makes it difficult for AEC methods to accurately identify and cancel duplicate signals, resulting in poor echo and feedback suppression. Summary of the Invention

[0004] In view of this, embodiments of this application provide an audio control method, a terminal, a storage medium, and a computer program product.

[0005] The technical solution of this application embodiment is implemented as follows: In a first aspect, embodiments of this application provide an audio control method applied to a first terminal, the method comprising: Obtain the first distance between the current sound source and each target terminal in the multimedia session; the target terminal is an access terminal located in the same physical space as the sound source, including the first terminal; Based on the first distance, the state of the first terminal in the multimedia session is determined, and the state includes a collection state and a muted state. If the first terminal is in acquisition mode, the acquired audio signal is processed and uploaded to the multimedia session; If the first terminal is muted, then the uploading of the acquired audio signal to the multimedia session will stop.

[0006] In the above scheme, if the first terminal is in a data collection state, the method further includes: Based on the first distance between the current sound source and the first terminal, the audio processing parameters of the first terminal are adjusted.

[0007] In the above scheme, the audio processing parameters include one or more of the following parameters: filter step size, residual echo suppression strength, and acquisition gain.

[0008] In the above scheme, obtaining the first distance between the current sound source and each target terminal in the multimedia session includes: Obtain the second distance between each of the target terminals; Based on the second distance, a multi-dimensional scale analysis is performed on the spatial distribution of the target terminal to generate a spatial device distribution map, which includes two-dimensional coordinates representing the physical location of the target terminal. Based on the spatial device distribution map, the first distance between the current sound source and each target terminal in the multimedia session is determined.

[0009] In the above scheme, the step of performing multi-dimensional scale analysis on the spatial distribution of the target terminal based on the second distance to generate a spatial device distribution map includes: Based on the second distance, construct a squared distance matrix; The squared distance matrix is ​​centered to obtain the inner product matrix; The inner product matrix is ​​subjected to eigenvalue decomposition and two-dimensional coordinate extraction to obtain the two-dimensional coordinates of the target terminal, and the spatial device distribution map is generated based on the two-dimensional coordinates.

[0010] The method in the above scheme further includes: Obtain the audio sampling rate and acquisition volume of each target terminal; Determining the state of the first terminal in the multimedia session based on the first distance includes: A score is determined for each target terminal based on its first distance, audio sampling rate, and acquisition volume. Based on the scores, the target terminals are sorted, and the status of the first terminal in the multimedia session is determined based on the sorting results.

[0011] In the above scheme, after determining the state of the first terminal in the multimedia session, the method further includes: If the change in the score of the target terminal exceeds a set threshold or a new target terminal is added to the multimedia session, the status of the first terminal in the multimedia session is redefined.

[0012] Secondly, embodiments of this application provide a terminal, the terminal comprising: a processor and a memory for storing a computer program capable of running on the processor, wherein the processor, when running the computer program, performs the steps of the method described in the first aspect.

[0013] Thirdly, embodiments of this application provide a storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0014] Fourthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0015] In the technical solution of this application, a first distance is obtained between the current sound source and each target terminal in the multimedia session; the target terminal is an access terminal located in the same physical space as the sound source, including a first terminal; based on the first distance, the state of the first terminal in the multimedia session is determined, including a collection state and a muted state; if the first terminal is in the collection state, the collected audio signal is processed and uploaded to the multimedia session; if the first terminal is in the muted state, the uploading of the collected audio signal to the multimedia session is stopped. Thus, this embodiment of the application adopts multi-target terminal collaborative audio control, which can dynamically allocate collection roles according to the distance between the sound source and each terminal, so that the terminal closer to the sound source is responsible for collecting and uploading the audio signal, while other terminals stop uploading, thereby avoiding the risk of mutual interference and howling caused by multiple terminals collecting and uploading at the same time, and significantly improving the audio quality and user experience of the multimedia session. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the audio control method according to an embodiment of this application; Figure 2 This is a schematic diagram of the spatial device distribution in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of the control device of the first terminal in the embodiment of this application; Figure 4 This is a schematic diagram of the structure of the audio control device according to an embodiment of this application; Figure 5 This is a schematic diagram of the terminal structure according to an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] With the widespread adoption of remote work and video conferencing, multi-person online meetings have become a common work mode. Participants connect to the audio and video conference through their respective terminals to conduct discussions. Each terminal connected to the audio and video conference independently captures and plays audio to ensure that participants' speeches are uploaded to the conference in real time and that they can receive and play back the speeches of other participants in real time.

[0019] In real-world work scenarios, it's common to encounter situations where multiple terminals simultaneously access the same audio / video conference within the same physical space. For example, members of a project team might be in the same meeting room, each using their own laptops to access the same conference, or multiple employees in the same office area might be using their own desktop computers or mobile phones to access the same conference. In these scenarios, because each terminal independently captures and plays audio, when a participant speaks, their voice is not only picked up by their own terminal but also by microphones on other terminals in the same physical space. This results in the same audio signal being repeatedly captured and uploaded to the audio / video conference by multiple terminals, creating an echo. Simultaneously, the sound output from any terminal's speaker (such as a participant speaking from a distance) can easily be picked up by microphones on nearby terminals, forming a positive feedback loop that generates high-frequency, sharp noise, causing feedback. Clearly, these echo and feedback problems severely impact the meeting process and the participant experience.

[0020] To suppress echo and feedback issues common in multi-person online conferences, related technologies typically employ the AEC (Average Echo Control) method for audio processing after a terminal joins an audio / video conference. In this method, the terminal plays a remote audio signal through a speaker while simultaneously capturing a local audio signal through a microphone. The AEC method uses the played remote audio signal as a reference and subtracts the echo component from the captured local audio signal, thus preserving the local speech signal.

[0021] However, the AEC method is designed for a single terminal, with each terminal executing independently. Echo and feedback in multi-person online conferences are typically caused by complex coupling between the microphones and speakers of multiple terminals, which the AEC method struggles to accurately handle. Furthermore, audio signals between multiple terminals often suffer from poor synchronization, making it difficult for the AEC method to accurately identify and cancel duplicate signals. Therefore, echo and feedback suppression is ineffective in multi-person online conference scenarios.

[0022] To address the aforementioned issues, this application provides an audio control method aimed at improving the echo and feedback suppression effect of multimedia sessions.

[0023] Exemplarily, the audio control method of this application embodiment is as follows: Figure 1 As shown, the method includes: Step 101: Obtain the first distance between the current sound source and each target terminal in the multimedia session.

[0024] Step 102: Based on the first distance, determine the state of the first terminal in the multimedia session, including the acquisition state and the mute state.

[0025] Step 103: If the first terminal is in acquisition mode, the acquired audio signal is processed and uploaded to the multimedia session.

[0026] Step 104: If the first terminal is muted, then stop uploading the acquired audio signal to the multimedia session.

[0027] Here, the audio control method of this application embodiment is applied to a first terminal, which is any target terminal in a multimedia session, that is, the target terminal includes the first terminal.

[0028] Here, multimedia sessions refer to real-time communication sessions that support audio transmission, including but not limited to audio and video conferencing, online classrooms, live streaming, multi-person voice calls, and online game voice communication.

[0029] Here, participants join a multimedia session based on an access terminal; the access terminal is an electronic device that supports multimedia sessions and has audio acquisition and playback functions, including but not limited to mobile phones, tablets, laptops, desktop computers, game consoles and conferencing terminals.

[0030] Here, the access terminal is equipped with an audio acquisition device, such as a microphone, for acquiring audio signals, and an audio playback device, such as a speaker, for playing audio signals. In a multimedia session, the access terminal uploads the acquired audio signals to the multimedia session, which then mixes the received audio signals and sends them to other access terminals, which then play them through the audio playback device to enable real-time voice interaction between participants.

[0031] Here, the access terminal can access the multimedia session through connection methods such as Wi-Fi, mobile data network, Ethernet, etc., and this application embodiment does not specifically limit this.

[0032] Here, the sound source in a multimedia session is usually the speaker among the participants. The speaker sends voice signals to nearby access terminals, which then collect the voice signals and upload them to the multimedia session.

[0033] The target terminal is the access terminal located in the same physical space as the sound source.

[0034] Here, the target terminal includes at least two access terminals.

[0035] Here, physical space refers to enclosed or semi-enclosed physical spaces, including but not limited to meeting rooms, offices, open-plan office areas, classrooms, training rooms, and living rooms.

[0036] It is understood that the audio control method of this application embodiment is applied to scenarios where multiple terminals access the same multimedia session within the same physical space. For example, online meeting scenarios where team members access the session using their laptops in the same meeting room, remote training scenarios where multiple employees access the session using their mobile phones in the same office area, and real-time voice scenarios in online games are all typical application scenarios of this application embodiment.

[0037] Here, the first distance is the distance between the sound source and the target terminal located in the same physical space.

[0038] It should be noted that, in order for the first terminal to obtain the first distance corresponding to each target terminal, the method further includes: discovering other target terminals before obtaining the first distance between the current sound source and each target terminal in the multimedia session.

[0039] In some embodiments, after the first terminal accesses the multimedia session, it enables terminal discovery detection via Wi-Fi, LAN, or Bluetooth to detect whether there are other terminals in the current physical space. After discovering other terminals in the same physical space, the first terminal sends a multimedia session identifier (e.g., session ID) to the other terminals and confirms whether the other terminals have accessed the same multimedia session based on the feedback results from the other terminals, thereby completing the discovery process of other target terminals.

[0040] In some embodiments, after other target terminals are discovered, a transmission channel is established between the target terminals. The transmission channel is used to transmit information carrying a first distance or information used to determine the first distance. The transmission channel can be established based on Web Real-Time Communication (WebRTC) to meet the low latency requirements of multimedia sessions.

[0041] It should be noted that the first terminal can obtain the first distance corresponding to other target terminals through the above-mentioned method of establishing a transmission channel, or it can obtain the first distance corresponding to other target terminals based on the communication mechanism of multimedia sessions. This application embodiment does not specifically limit this.

[0042] It should be noted that, in order to suppress the echo and howling problems that are easily caused when multiple terminals access the same multimedia session in the same physical space, this application adopts a multi-target terminal collaborative audio control mechanism. Under this mechanism, not all target terminals independently perform audio signal acquisition and uploading. Instead, different working states are assigned to each target terminal, and only the target terminal assigned to the acquisition state uploads the acquired audio signal to the multimedia session.

[0043] In some embodiments, determining the state of the first terminal in a multimedia session based on a first distance includes: sorting target terminals based on the first distance, and determining the state of the first terminal in a multimedia session based on the sorting result.

[0044] Specifically, the main cause of echo and feedback in multimedia sessions is that the speaker's voice is captured, uploaded, and played by multiple access terminals located in the same physical space. Due to varying distances from the speaker, each access terminal exhibits significantly different voice signal acquisition effects. Typically, access terminals used by or near the speaker have excellent voice acquisition effects, while those far from the speaker have poorer voice acquisition effects and noticeable delays. Therefore, in this embodiment, after the first terminal accesses the multimedia session and determines that other terminals in the current physical space are also accessing the session, it obtains the first distance between each target terminal and the current sound source. Then, based on its ranking among all target terminals according to its distance from the current sound source, it determines the corresponding working state.

[0045] Based on the sorting result of the first distance, one or more target terminals that are closer to the current sound source are assigned to the acquisition state, and the remaining target terminals are assigned to the mute state. If the first terminal is assigned to the acquisition state, it acquires and uploads the audio signal; if it is assigned to the mute state, it stops uploading the acquired audio signal.

[0046] Here, stopping the upload of acquired audio signals can mean stopping the acquisition of audio signals altogether, or stopping the upload of the acquired audio signals to the multimedia session after acquisition.

[0047] It is understood that this application embodiment, through a multi-terminal collaborative audio control mechanism, assigns the target terminal closer to the sound source to be responsible for high-quality audio acquisition and uploading, ensuring the clarity and coherence of uploaded voice in multimedia sessions; simultaneously, terminals farther from the sound source are muted, prohibiting remote terminals from uploading acquired audio signals, thereby cutting off potential echo paths and positive feedback loops generated by remote terminals, avoiding the risk of mutual interference and feedback caused by simultaneous acquisition and uploading by multiple terminals, and suppressing the generation of echoes and feedback from the source. Furthermore, the multi-terminal collaborative audio control mechanism of this application embodiment does not require manual intervention from session participants; it is automatically completed by the access terminals, significantly improving the audio quality and user experience of multimedia sessions.

[0048] For example, obtaining the first distance between the current sound source and each target terminal in a multimedia session includes: obtaining the second distance between each target terminal; performing multi-dimensional scale analysis on the spatial distribution of the target terminals based on the second distance to generate a spatial device distribution map, the spatial device distribution map including two-dimensional coordinates representing the physical location of the target terminals; and determining the first distance between the current sound source and each target terminal in the multimedia session based on the spatial device distribution map.

[0049] Here, the second distance is the distance between any two target terminals in the same physical space. In some embodiments, the method further includes: calculating the second distance based on a sound source localization algorithm.

[0050] In some embodiments, the target terminal may calculate the second distance based on sound source localization algorithms such as Time Difference of Arrival (TDOA).

[0051] The steps for the target terminal to calculate the second distance based on the TDOA algorithm include: each target terminal collecting environmental audio signals and recording the timing information of the environmental audio signals; each target terminal exchanging timing information and determining the time difference of the same sound event arriving at each target terminal based on the timing information; and determining the corresponding second distance based on the time difference.

[0052] Here, sound events are identifiable events in the ambient audio signal, such as test sounds actively emitted by the target terminal, or the starting point of a speaker's voice.

[0053] Here, the target terminal collects the ambient audio signal and records the corresponding timestamp; the timing information of the ambient audio signal can be the timestamp corresponding to the sound event, or it can be the ambient audio signal carrying the timestamp.

[0054] Here, the target terminal can obtain the timing information and / or second distance of other target terminals based on the established transmission channel.

[0055] Here, the second distance between target terminal A and target terminal B can be calculated based on the above formula:

[0056] Where Δd is the second distance between target terminal A and target terminal B, and T B -T A The time difference between the arrival of the sound event at target terminal A and target terminal B is given by v, which is the speed of sound, usually taken as the speed of sound in air, 343 m / s.

[0057] It is understood that the first terminal can directly obtain the second distance calculated by other target terminals, or it can obtain the timing information corresponding to other target terminals and calculate the second distance between the corresponding target terminals based on the timing information. This application embodiment does not specifically limit this.

[0058] Here, after the first terminal obtains the second distance between each pair of all target terminals, it uses a spatial scale distribution preprocessing algorithm to calculate the two-dimensional relative position of each target terminal in the physical space and generate the corresponding spatial device distribution map; the spatial scale distribution preprocessing algorithm includes a multidimensional scaling algorithm.

[0059] Accordingly, based on the second distance, a multi-dimensional scale analysis is performed on the spatial distribution of the target terminal to generate a spatial device distribution map, including: constructing a squared distance matrix based on the second distance; centering the squared distance matrix to obtain an inner product matrix; performing feature decomposition and two-dimensional coordinate extraction on the inner product matrix to obtain the two-dimensional coordinates of the target terminal, and generating the spatial device distribution map based on the two-dimensional coordinates.

[0060] The process of constructing a squared distance matrix based on the second distance includes: constructing a distance matrix based on the second distance between each target terminal; and determining the squared distance matrix based on the distance matrix.

[0061] Here, the first terminal obtains the second distance between each target terminal and constructs a distance matrix D=[Δd] containing all the second distances. ij ], where Δd ij Let D be the second distance between target terminal i and target terminal j; after constructing the distance matrix D, process the distance matrix D to obtain the squared distance matrix. Squared distance matrix D 2 Used for dual-centralization processing.

[0062] The process of centering the squared distance matrix to obtain the inner product matrix includes: based on the centering matrix, performing double centering on the squared distance matrix to obtain the inner product matrix.

[0063] Here, the central speech matrix J is , where I is an n×n identity matrix and 1 is a column vector of all 1s.

[0064] Accordingly, based on the centralization matrix J, the squared distance matrix D is... 2 By performing double centering, the inner product matrix is ​​obtained. .

[0065] It should be noted that the centering matrix is ​​used to translate the data points to a coordinate system with the center as the origin, eliminating the influence of absolute position on coordinate recovery; by performing double centering on the squared distance matrix, the squared distance matrix can be converted into an inner product matrix in dot product form, so that the coordinate matrix obtained by subsequent feature decomposition can correctly reflect the relative positional relationship between each target terminal.

[0066] The process involves performing eigenvalue decomposition and two-dimensional coordinate extraction on the inner product matrix to obtain the two-dimensional coordinates of the target terminal. This includes: performing eigenvalue decomposition on the inner product matrix to obtain an eigenvalue diagonal matrix and an eigenvector matrix; extracting two-dimensional coordinates from the eigenvalue diagonal matrix and the eigenvector matrix; and constructing a coordinate matrix based on the extracted eigenvalues ​​and eigenvectors.

[0067] Here, the eigenvalue diagonal matrix The eigenvalues ​​in the matrix are arranged in descending order and used to extract coordinates; the eigenvector matrix V is composed of eigenvectors corresponding to the eigenvalues ​​and is an orthogonal matrix.

[0068] Here, based on the eigenvalue decomposition property, the inner product matrix B is decomposed using the following formula:

[0069] Here, after eigenvalue decomposition, we obtain an n×n eigenvalue diagonal matrix. And an n×n eigenvector matrix V; in the two-dimensional coordinate recovery stage, from the eigenvalue diagonal matrix Extract the first two positive feature values And extract the corresponding n×2 feature vectors from the feature vector matrix V. Based on the extracted feature values ​​and corresponding feature vectors, a coordinate matrix X is constructed, where the coordinate matrix X is... ,Right now .

[0070] It is understandable that the coordinate matrix X is an n×2 matrix, where each row of elements corresponds to the two-dimensional coordinates of the target terminal.

[0071] Here, after constructing a coordinate matrix to obtain the two-dimensional coordinates of each target terminal, the coordinate points corresponding to each target terminal are marked in a Cartesian coordinate system to generate a space device distribution map; in some embodiments, the space device distribution map is as follows: Figure 2 As shown, this includes the coordinates of the target terminals D1-D6.

[0072] It is understandable that the spatial device distribution map represents the relative position of the target terminal in physical space. After the first terminal obtains the spatial device distribution map, it can calculate the corresponding first distance based on the coordinate position of the target terminal.

[0073] In some embodiments, the method further includes: acquiring the audio sampling rate and acquisition volume of each target terminal.

[0074] Accordingly, based on the first distance, the state of the first terminal in the multimedia session is determined, including: determining the score of each target terminal based on the first distance, audio sampling rate and acquisition volume of each target terminal; sorting the target terminals based on the score, and determining the state of the first terminal in the multimedia session based on the sorting result.

[0075] Here, the audio sampling rate is the frequency at which the audio signal is sampled, used to determine the upper limit of the sound frequency that can be effectively recorded; the sampling volume is the sampling amplitude parameter of the audio signal.

[0076] Specifically, the score for each target terminal is determined based on its first distance, audio sampling rate, and acquisition volume. This includes: calculating a distance score component for each target terminal based on its first distance; calculating a sampling rate score component for each target terminal based on its audio sampling rate; calculating a volume score component for each target terminal based on its acquisition volume; and determining the score for each target terminal based on its distance score component, sampling rate score component, and volume score component.

[0077] It should be noted that, in order to further improve audio acquisition commands while suppressing echo and feedback, this application embodiment also provides a state determination mechanism based on multi-dimensional scoring. Under this mechanism, the allocation of the working state of the target terminal not only considers the distance between the target terminal and the current sound source, but also comprehensively considers the audio sampling rate, which represents the audio processing capability of each target terminal, and the acquisition volume, which represents the sound pickup effect. The audio acquisition capability of each target terminal is comprehensively scored through a weighted scoring method, and the working state is assigned to each target terminal according to the scoring results, thereby further improving the audio acquisition quality and user experience of multimedia sessions.

[0078] In some embodiments, the distance scoring component of the target terminal is determined based on the maximum first distance corresponding to the target terminal, as shown in the following formula:

[0079] in, For the distance scoring component of the target terminal, D i D represents the first distance corresponding to the target terminal. max The first maximum distance.

[0080] Here, the distance score component is negatively correlated with the first distance corresponding to the target terminal, that is, the larger the first distance, the lower the distance score component.

[0081] In some embodiments, the sampling rate scoring component of the target terminal is specifically determined based on the following formula:

[0082] in, F is the sampling rate scoring component of the target terminal. i This represents the audio sampling rate corresponding to the target terminal.

[0083] Here, the sampling rate score component is negatively correlated with the audio sampling rate corresponding to the target terminal; that is, the higher the audio sampling rate, the lower the sampling rate score component.

[0084] In some embodiments, the volume score component of the target terminal is specifically determined based on the following formula:

[0085] in, V is the volume score component for the target terminal. i The volume level corresponding to the target terminal.

[0086] Here, the volume score component is negatively correlated with the collection volume of the target terminal, that is, the higher the collection volume, the lower the volume score component.

[0087] In some embodiments, the target terminal's score S i Specifically determined based on the following formula:

[0088] in, These are, in order, the distance rating components. Sampling rate scoring components and volume rating components The corresponding weighting factors can be adjusted according to the priority of each scoring dimension.

[0089] In some embodiments, each weighting factor is configured as follows: , , .

[0090] Understandably, the lower the target terminal's score, the stronger its audio acquisition capability for the current sound source.

[0091] It should be noted that in a multimedia session, after the first terminal discovers other target terminals, it determines the number of terminals to be collected. The collected terminals are the target terminals assigned to the collection status. The number of collected terminals can be a pre-configured number, such as 2, or it can be determined based on the number of target terminals accessing the multimedia session. For example, if the number of target terminals currently accessing the session exceeds 10, it can be configured as 2.

[0092] It is understandable that after sorting the target terminals in order of score from low to high, the target terminals to be collected are selected sequentially from the sorting results, and the selected target terminals are assigned to the collection state, while the remaining unselected target terminals are assigned to the mute state.

[0093] In some embodiments, the number of acquisition terminals is configured to be 2. After determining the score of each target terminal, the target terminal with the lowest score is selected as the main terminal of the acquisition terminal, which is responsible for picking up the audio signal of the current sound source, and the target terminal with the second lowest score is selected as the auxiliary terminal of the acquisition terminal, which is used to provide supplementary audio signals or assist in noise reduction and echo suppression.

[0094] In some embodiments, after determining the state of the first terminal in a multimedia session, the method further includes: if the change in the score of the target terminal exceeds a set threshold or a new terminal is connected to the multimedia session, then the state of the first terminal in the multimedia session is re-determined.

[0095] It should be noted that, in order to avoid the first terminal frequently switching between the acquisition state and the mute state, this application embodiment also sets up a state reassignment triggering mechanism. The state reassignment of each target terminal is triggered only when the score of any target terminal changes significantly beyond a set threshold or when a new terminal accesses the multimedia session in the same physical space. After the state is reassigned, the first terminal performs corresponding audio control according to the assigned state.

[0096] Here, the conditions that cause the change in the target terminal's score to exceed the set threshold include, but are not limited to: speaker switching, change in the physical location of the target terminal, and change in the audio sampling rate or acquisition volume of the target terminal.

[0097] If a new terminal is connected to the multimedia session, the first terminal obtains the second distance between the new terminal and other target terminals before re-determining the state of the first terminal in the multimedia session, updates the spatial device distribution map according to the newly obtained second distance, and then obtains the first distance between the new terminal and the current sound source according to the spatial device distribution map.

[0098] In some embodiments, if the first terminal is in a data acquisition state, the method further includes: adjusting the audio processing parameters of the first terminal based on a first distance between the current sound source and the first terminal.

[0099] The audio processing parameters include one or more of the following: filter step size, residual echo suppression strength, and acquisition gain.

[0100] Here, the acquired audio signal is processed, including: processing the acquired audio signal based on determined audio processing parameters.

[0101] The filter step size, also known as the adaptive step size, is the step size by which the coefficients of an adaptive filter used for audio processing are updated in each iteration. It is used to control the balance between the convergence speed and steady-state error of echo cancellation. The larger the step size, the faster the filter converges, but the larger the steady-state error; the smaller the step size, the slower the convergence, but the higher the accuracy.

[0102] The Residual Echo Suppression Strength (RES Level) is the degree of nonlinear attenuation of the echo signal remaining after echo cancellation, used to further suppress echo components that were not completely eliminated by the linear filter. A stronger RES Level results in fewer residual echo components, but also a higher risk of speech distortion.

[0103] Capture gain (or microphone gain) is the amplification factor of the audio signal during the preamplifier stage, used to adjust the amplitude of the input audio signal. A higher capture gain results in greater microphone sensitivity and a larger amplitude audio signal, but excessively high capture gain can lead to signal clipping distortion or increased feedback risk.

[0104] It should be noted that, in addition to multiple terminals located in the same physical space acquiring and playing audio signals from the same sound source, forming a positive feedback loop, the direct cause of howling in multimedia sessions also includes the fact that each terminal usually uses the same or similar audio processing parameters to process the acquired audio signals. For example, each terminal is configured with the same acquisition gain. The same or similar parameter configuration causes the audio signal to gain continuously with each cycle of processing and playback in the positive feedback loop, resulting in a rapid increase in signal amplitude that exceeds the system's linear response range, thus triggering howling.

[0105] It should be noted that, in order to further suppress the howling problem in multimedia sessions, the multi-terminal collaborative audio control mechanism of this application embodiment, in addition to allocating working states for access terminals in the same physical space, also provides an audio processing parameter adjustment mechanism. The terminal in the acquisition state adjusts its own audio processing parameters based on the first distance between itself and the current sound source, so that each terminal in the acquisition state uses different audio processing parameters to process the acquired audio signal. This avoids the accumulation of gain in a positive feedback loop when multiple terminals in the acquisition state acquire and play audio signals due to consistent parameters. Thus, while ensuring the sound pickup quality of the near-end terminal, it effectively suppresses the howling risk caused by the far-end terminal.

[0106] In some embodiments, adjusting the audio processing parameters of the first terminal based on the first distance between the current sound source and the first terminal includes: normalizing the first distance between the current sound source and the first terminal to obtain a normalization result; and determining the audio processing parameters of the first terminal based on the normalization result.

[0107] In some embodiments, the first distance is normalized using a Sigmoid mapping function; wherein the normalization result Determined based on the following formula:

[0108] Where d is the first distance, d0 is the midpoint distance, and k is the sensitivity factor. The midpoint distance d0 and the sensitivity factor k are preset parameters.

[0109] In some embodiments, the midpoint distance d0 is determined based on the spatial device distribution map.

[0110] In some embodiments, the filter step size is specifically determined based on the following formula:

[0111] in, The filter step size, and This is the preset step size.

[0112] In some embodiments, , .

[0113] It is understandable that the filter step size is positively correlated with the first distance; the smaller the first distance, the smaller the filter step size.

[0114] It is understandable that setting a smaller filter step size for acquisition terminals that are relatively close to the current sound source (such as the main terminal) can avoid excessive cancellation of the audio signal, while setting a larger filter step size for acquisition terminals that are relatively far from the current sound source (such as the auxiliary terminal) can speed up the convergence of the echo path.

[0115] In some embodiments, the residual echo suppression intensity is specifically determined based on the following formula:

[0116] in, The residual echo suppression intensity, and This is the preset residual echo suppression intensity.

[0117] In some embodiments, , .

[0118] It is understandable that the residual echo suppression intensity is positively correlated with the first distance; the smaller the first distance, the weaker the residual echo suppression intensity.

[0119] It is understandable that setting a smaller residual echo suppression intensity for acquisition terminals that are relatively close to the current sound source (such as the main terminal) can fully preserve the audio signal components emitted by the current sound source, while setting a larger residual echo suppression intensity for acquisition terminals that are relatively far from the current sound source (such as the auxiliary terminal) can fully suppress echoes and howling.

[0120] In some embodiments, the acquisition gain is specifically determined based on the following formula:

[0121] in, For acquisition gain, and This is the preset acquisition gain.

[0122] In some embodiments, , .

[0123] It is understandable that the acquisition gain is negatively correlated with the first distance; the smaller the first distance, the greater the acquisition gain.

[0124] Understandably, acquisition terminals that are relatively close to the current sound source (such as the main terminal) maintain a high acquisition gain, while acquisition terminals that are relatively far from the current sound source (such as the auxiliary terminal) reduce the corresponding acquisition gain.

[0125] In some embodiments, if the first terminal is in a data acquisition state, the method further includes: adjusting the audio processing parameters of the first terminal based on the sorting result corresponding to the first terminal.

[0126] It is understandable that, in addition to adjusting the audio processing parameters based on the corresponding first distance, each target terminal in the acquisition state can also adjust the audio processing parameters based on the scoring and ranking results that characterize the audio acquisition capability.

[0127] In some embodiments, the method further includes: obtaining a state allocation result; determining the state of the first terminal in a multimedia session based on the state allocation result; if the first terminal is in a acquisition state, processing the acquired audio signal and uploading the processed acquired audio signal to the multimedia session; if the first terminal is in a muted state, stopping the uploading of the acquired audio signal to the multimedia session.

[0128] Among them, obtaining the state allocation result includes obtaining the state allocation result sent by other target terminals.

[0129] In some embodiments, the method further includes: determining the state of one or more other target terminals in a multimedia session based on a first distance, and generating a corresponding state allocation result; and sending the state allocation result to the corresponding target terminal.

[0130] It is understood that in the multi-terminal collaborative audio control mechanism of this application embodiment, after the target terminal discovery process is completed, the state allocation of the target terminal can be determined by any one or more target terminals; that is, the state of the target terminal in the multimedia session can be determined by itself or by other target terminals. Each target terminal performs corresponding audio control based on its own determined state or the received state allocation result. This application embodiment does not make specific limitations on this.

[0131] In one application example of this application, the first terminal includes a control device, such as... Figure 3 As shown, the system includes a device discovery module, a spatial positioning module, an audio coordination module, and an echo and feedback suppression module. The device discovery module executes the steps of the target terminal discovery process described above after the first terminal accesses the multimedia session; the spatial positioning module executes the steps of the first distance acquisition process described above; the audio coordination module executes the steps of the state allocation process; and the echo and feedback suppression module executes the steps of the audio processing signal adjustment process. Based on the coordinated operation of these modules, the first terminal can implement the audio control method of this application embodiment.

[0132] The audio control method of this application will be specifically explained below in the context of a multi-person online conference scenario in which multiple terminals located in the same conference room access the same audio and video conference.

[0133] In this example, terminals A, B, and C are connected to the same audio and video conference and located in the same conference room. When terminals A, B, and C connect to the audio and video conference, they sequentially perform the terminal discovery, spatial positioning, state allocation, and echo and howling processing stages.

[0134] During the terminal discovery phase, terminals A, B, and C discover each other and send and receive conference IDs using the Wi-Fi provided in the conference room. They confirm that they are all connected to the same audio and video conference based on the same conference ID. After confirming that they are all connected to the same audio and video conference, terminals A, B, and C establish a transmission channel based on WebRTC. This transmission channel is used for the real-time transmission of audio signals and control commands.

[0135] In the spatial positioning phase, each terminal's microphone detects that the user corresponding to terminal B is speaking, and simultaneously receives audio signals with timestamps sent by other terminals and / or the distance between terminals. Based on the TODA algorithm, the relative position of each terminal is determined, and a corresponding spatial device distribution map is generated.

[0136] During the state allocation phase, if each terminal detects that the user corresponding to terminal B is speaking, then based on the location of each terminal marked on the spatial device distribution map, combined with the audio sampling rate and acquisition volume of each terminal, it is determined that terminal B is the main terminal, terminal A is the auxiliary terminal, and terminal C is the muted terminal. According to the state allocation result, the microphones of terminal B and terminal A are turned on, and the microphone of terminal C is turned off, while the speakers of terminal A and terminal C remain on.

[0137] In the echo and howling processing stage, terminal B, which is assigned as the main terminal, reduces the filter step size, lowers the residual echo suppression intensity, and increases the acquisition gain, while terminal A, which is assigned as the auxiliary terminal, adopts a longer filter length, increases the filter step size, increases the residual echo suppression intensity, and lowers the acquisition gain.

[0138] When the user on terminal C starts speaking, the scores of each terminal change significantly. Then, the status of each terminal is redefined, terminal C is switched to the main terminal, and terminal B is switched to the mute terminal.

[0139] When it is discovered that terminal D, located in the same conference room, is connected to the audio and video conference, the steps of the spatial positioning stage, state allocation stage, and echo and howling processing stage mentioned above are re-executed.

[0140] In order to implement the method of the embodiments of this application, the embodiments of this application also provide an audio control device applied to a first terminal. The audio control device corresponds to the above-described audio control method, and the steps in the embodiments of the above-described audio control method are also fully applicable to the embodiments of this device.

[0141] like Figure 4As shown, the audio control device in this embodiment includes an acquisition module 401, a determination module 402, a first control module 403, and a second control module 404. The acquisition module 401 acquires a first distance between the current sound source and each target terminal in the multimedia session; the target terminal is an access terminal located in the same physical space as the sound source, including the first terminal; the determination module 402 determines the state of the first terminal in the multimedia session based on the first distance, the state including a acquisition state and a muted state; the first control module 403 processes the acquired audio signal and uploads the processed acquired audio signal to the multimedia session if the first terminal is in an acquisition state; the second control module 404 stops uploading the acquired audio signal to the multimedia session if the first terminal is in a muted state.

[0142] In some embodiments, the first control module 403 is further configured to: if the first terminal is in a data acquisition state, adjust the audio processing parameters of the first terminal based on the first distance between the current sound source and the first terminal.

[0143] In some embodiments, the audio processing parameters include one or more of the following parameters: filter step size, residual echo suppression strength, and acquisition gain.

[0144] In some embodiments, the acquisition module 401 is specifically used to: acquire a second distance between each target terminal; perform multi-dimensional scale analysis on the spatial distribution of the target terminals based on the second distance to generate a spatial device distribution map, the spatial device distribution map including two-dimensional coordinates representing the physical location of the target terminals; and determine a first distance between the current sound source and each target terminal in the multimedia session based on the spatial device distribution map.

[0145] In some embodiments, the acquisition module 401 is specifically used to: construct a squared distance matrix based on the second distance; center the squared distance matrix to obtain an inner product matrix; perform feature decomposition and two-dimensional coordinate extraction on the inner product matrix to obtain the two-dimensional coordinates of the target terminal, and generate the spatial device distribution map based on the two-dimensional coordinates.

[0146] In some embodiments, the acquisition module 401 is further configured to: acquire the audio sampling rate and acquisition volume of each target terminal.

[0147] In some embodiments, the determining module 402 is specifically configured to: determine a score for each target terminal based on a first distance, audio sampling rate, and acquisition volume; sort the target terminals based on the scores; and determine the status of the first terminal in the multimedia session based on the sorting results.

[0148] In some embodiments, the determining module 402 is specifically used to: if the change in the score of the target terminal exceeds a set threshold or a new target terminal is added to the multimedia session, then redetermine the state of the first terminal in the multimedia session.

[0149] It should be noted that the audio control device provided in the above embodiments is only illustrated by the division of the above-described program modules when performing audio control. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the audio control device and the audio control method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0150] Based on the hardware implementation of the above program modules, and in order to implement the audio control method of this application embodiment, this application embodiment also provides a terminal, such as... Figure 5 As shown, terminal 500 includes at least one processor 501, memory 502, user interface 503, and at least one network interface 504. The various components in terminal 500 are coupled together via bus system 505. It can be understood that bus system 505 is used to implement communication between these components. In addition to a data bus, bus system 505 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 5 The general designated all buses as Bus System 505.

[0151] The user interface 503 may include a monitor, keyboard, mouse, trackball, click wheel, buttons, touchpad, or touch screen.

[0152] The memory 502 in this embodiment is used to store various types of data to support the operation of the terminal 500. Examples of such data include any computer program used to operate on the terminal 500.

[0153] The audio control method disclosed in this application can be applied to or implemented by the processor 501. The processor 501 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the audio control method can be completed by the integrated logic circuitry in the hardware of the processor 501 or by instructions in software form. The processor 501 can be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 501 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules can be located in a storage medium, specifically in memory 502. The processor 501 reads information from memory 502 and, in conjunction with its hardware, completes the steps of the audio control method provided in the embodiments of this application.

[0154] In an exemplary embodiment, terminal 500 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), FPGAs, general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned audio control method.

[0155] It is understood that memory 502 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), EEPROM, ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Sync Link Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM). The memory 502 described in this application embodiment is intended to include, but is not limited to, these and any other suitable types of memory.

[0156] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory 502 storing a computer program. This computer program can be executed by the processor 501 of the terminal 500 to complete the steps described in the audio control method of this application embodiment. The computer-readable storage medium can be a ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM, etc.

[0157] In an exemplary embodiment, this application also provides a computer program product, including a computer program that can be executed by the processor 501 of a terminal 500 to complete the steps described in the method of this application embodiment.

[0158] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0159] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0160] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An audio control method, characterized in that, Applied to a first terminal, the method includes: Obtain the first distance between the current sound source and each target terminal in the multimedia session; the target terminal is an access terminal located in the same physical space as the sound source, including the first terminal; Based on the first distance, the state of the first terminal in the multimedia session is determined, and the state includes a collection state and a muted state. If the first terminal is in acquisition mode, the acquired audio signal is processed and uploaded to the multimedia session; If the first terminal is muted, then the uploading of the acquired audio signal to the multimedia session will stop.

2. The method according to claim 1, characterized in that, If the first terminal is in a data collection state, the method further includes: Based on the first distance between the current sound source and the first terminal, the audio processing parameters of the first terminal are adjusted.

3. The method according to claim 2, characterized in that, The audio processing parameters include one or more of the following parameters: filter step size, residual echo suppression strength, and acquisition gain.

4. The method according to claim 1, characterized in that, The step of obtaining the first distance between the current sound source and each target terminal in the multimedia session includes: Obtain the second distance between each of the target terminals; Based on the second distance, a multi-dimensional scale analysis is performed on the spatial distribution of the target terminal to generate a spatial device distribution map, which includes two-dimensional coordinates representing the physical location of the target terminal. Based on the spatial device distribution map, the first distance between the current sound source and each target terminal in the multimedia session is determined.

5. The method according to claim 4, characterized in that, The step of performing multi-dimensional scale analysis on the spatial distribution of the target terminal based on the second distance to generate a spatial device distribution map includes: Based on the second distance, construct a squared distance matrix; The squared distance matrix is ​​centered to obtain the inner product matrix; The inner product matrix is ​​subjected to eigenvalue decomposition and two-dimensional coordinate extraction to obtain the two-dimensional coordinates of the target terminal, and the spatial device distribution map is generated based on the two-dimensional coordinates.

6. The method according to claim 1, characterized in that, The method further includes: Obtain the audio sampling rate and acquisition volume of each target terminal; Determining the state of the first terminal in the multimedia session based on the first distance includes: A score is determined for each target terminal based on its first distance, audio sampling rate, and acquisition volume. Based on the scores, the target terminals are sorted, and the status of the first terminal in the multimedia session is determined based on the sorting results.

7. The method according to claim 6, characterized in that, After determining the state of the first terminal in the multimedia session, the method further includes: If the change in the score of the target terminal exceeds a set threshold or a new target terminal joins the multimedia session, the status of the first terminal in the multimedia session is redefined.

8. A terminal, characterized in that, The terminal includes: a processor and a memory for storing a computer program capable of running on the processor, wherein the processor, when running the computer program, performs the steps of the method according to any one of claims 1 to 7.

9. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images