Audio signal processing, reverberation detection and conference method, device and storage medium

Abstract

The embodiment of the present application provides an audio signal processing, reverberation detection and conference method, device and storage medium. In the embodiment of the present application, the reverberation parameters of the environment in which the terminal device is located are introduced, and the reverberation parameters and the signal causing the echo noise played by the speaker are used to perform echo suppression on the signal collected by the microphone, so as to obtain a signal after the echo noise is suppressed, which is conducive to achieving better echo cancellation performance and improving the quality of the signal collected by the microphone.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to an audio signal processing, reverberation detection and conference method, device and storage medium. Background Art

[0002] In an audio and video communication system, a local device usually has a speaker and a microphone. The sound played by the speaker may generate echo noise after passing through multiple propagation paths. The echo noise is eventually picked up again by the local microphone and transmitted to the other end, causing the speaker at the other end to hear the echo of his or her own speech.

[0003] The prior art generally adopts an adaptive filtering method to eliminate echo noise of audio and video communication systems, but the adaptive filtering method is not very effective in eliminating echo noise. Therefore, how to eliminate echo noise of audio and video communication systems is an urgent problem to be solved. Summary of the invention

[0004] Multiple aspects of the present application provide an audio signal processing, reverberation detection and conferencing method, device and storage medium to achieve better echo cancellation performance and improve the quality of signals collected by a microphone.

[0005] An embodiment of the present application provides a conference method, including: obtaining reverberation parameters in a conference scene where a local conference device is located, the local conference device including a speaker and a microphone; during the conference, using the microphone to collect a sound signal at the local end, the sound signal collected by the microphone is mixed with echo noise, and the echo noise is generated by the sound signal played by the speaker; according to the reverberation parameters and the sound signal played by the speaker, echo suppression is performed on the sound signal collected by the microphone, and the sound signal after echo suppression is transmitted to the opposite-end conference device.

[0006] An embodiment of the present application also provides a remote communication method, including: obtaining reverberation parameters in a scene where a local communication device is located, the local communication device including a speaker and a microphone; during the remote communication process, using the microphone to collect the local sound signal, the sound signal collected by the microphone is mixed with echo noise, and the echo noise is generated by the sound signal played by the speaker; according to the reverberation parameters and the sound signal played by the speaker, the sound signal collected by the microphone is echo suppressed, and the sound signal after echo suppression is transmitted to the opposite communication device.

[0007] An embodiment of the present application also provides an audio signal processing method, which is applicable to a terminal device, wherein the terminal device includes a speaker and a microphone, and the method includes: obtaining reverberation parameters in an environment in which the terminal device is located, wherein a signal played by the speaker in the terminal device may generate echo noise in the environment; obtaining a mixed signal collected by the microphone in the terminal device, wherein the mixed signal includes at least a target signal and a first echo noise, and the first echo noise is generated by a first signal played by the speaker; and performing echo suppression on the mixed signal according to the reverberation parameters and the first signal to obtain a target signal after suppressing the first echo noise.

[0008] An embodiment of the present application also provides a reverberation detection method, which is applicable to a terminal device, wherein the terminal device includes a speaker and a microphone, and the method includes: using the microphone to collect echo noise in the environment where the terminal device is located under echo noise collection conditions, the echo noise is generated in the environment by a signal played by the speaker; linearly filtering the echo noise to obtain residual noise of the echo noise; and calculating the reverberation parameters in the environment where the terminal device is located based on the echo noise and its residual noise.

[0009] The embodiment of the present application also provides a testing method, which is applicable to cloud devices, including: sending a test signal to a terminal device in an audio and video communication system, so that the terminal device uses its speaker to play the test signal in the environment; receiving echo noise collected by its microphone and returned by the terminal device, the echo noise is generated by the test signal in the environment; linearly filtering the echo noise to obtain residual noise of the echo noise, and calculating reverberation parameters in the environment based on the echo noise and its residual noise; sending the reverberation parameters to the terminal device, so that the terminal device can optimize its signal to be sent and / or signal to be played.

[0010] An embodiment of the present application also provides an audio signal processing method, which is applicable to a terminal device, wherein the terminal device includes a speaker and a microphone, and the method includes: obtaining reverberation parameters in the environment in which the terminal device is located; optimizing the playback parameters on the downlink playback link using the reverberation parameters; processing the signal to be played by the speaker using the optimized playback parameters; and playing the processed signal using the speaker.

[0011] An embodiment of the present application also provides a terminal device, including: a speaker, a microphone, a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program, so as to: obtain reverberation parameters in the environment in which the terminal device is located, wherein the signal played by the speaker can generate echo noise in the environment; and obtain a mixed signal collected by the microphone, the mixed signal at least including a target signal and a first echo noise, the first echo noise being generated by the first signal played by the speaker; according to the reverberation parameters and the first signal, echo suppression is performed on the mixed signal to obtain the target signal after suppressing the first echo noise.

[0012] An embodiment of the present application also provides a terminal device, including: a speaker, a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program, so as to: obtain reverberation parameters in the environment in which the terminal device is located; optimize the playback parameters on the downlink playback link using the reverberation parameters; process the signal to be played by the speaker using the optimized playback parameters; and play the processed signal using the speaker.

[0013] An embodiment of the present application also provides a conference device, including: a speaker, a microphone, a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program, so as to: obtain reverberation parameters in a conference scene where the conference device at this end is located; during the meeting, use the microphone to collect the sound signal at this end, the sound signal collected by the microphone is mixed with echo noise, and the echo noise is generated by the sound signal played by the speaker; according to the reverberation parameters and the sound signal played by the speaker, the sound signal collected by the microphone is echo suppressed, and the sound signal after echo suppression is transmitted to the conference device at the other end.

[0014] The embodiment of the present application also provides a terminal device, including: a speaker, a microphone, a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program, so as to: obtain reverberation parameters in the scene where the local communication device is located; in the process of remote communication, use the microphone to collect the sound signal of the local end, the sound signal collected by the microphone is mixed with echo noise, and the echo noise is generated by the sound signal played by the speaker; according to the reverberation parameters and the sound signal played by the speaker, the sound signal collected by the microphone is echo suppressed, and the sound signal after echo suppression is transmitted to the opposite communication device.

[0015] An embodiment of the present application also provides a cloud device, including: a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program, so as to: send a test signal to a terminal device in an audio and video communication system, so that the terminal device uses its speaker to play the test signal in the environment; receive echo noise collected by a microphone of the terminal device and returned by the terminal device, the echo noise is generated by the test signal in the environment; linearly filter the echo noise to obtain the residual noise of the echo noise, and calculate the reverberation parameters in the environment based on the echo noise and its residual noise; send the reverberation parameters to the terminal device, so that the terminal device can optimize the signal to be sent and / or the signal to be played.

[0016] An embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor implements the steps in the method provided in each embodiment of the present application.

[0017] In an embodiment of the present application, the reverberation parameters of the environment in which the terminal device is located are introduced, and the signal collected by the microphone is echo suppressed using the reverberation parameters and the signal causing the echo noise played by the speaker, thereby obtaining a signal with suppressed echo noise, which is conducive to achieving better echo cancellation performance and improving the quality of the signal collected by the microphone. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 A schematic flow chart of a reverberation detection method provided for an exemplary embodiment of the present application;

[0020] Figure 2a A schematic diagram of a transfer function of echo noise provided by an exemplary embodiment of the present application;

[0021] Figure 2b A schematic diagram of a process of linearly filtering echo noise based on an adaptive filter provided by an exemplary embodiment of the present application;

[0022] Figure 3 A flowchart of an audio signal processing method provided by an exemplary embodiment of the present application;

[0023] Figure 4 A schematic diagram of a process of performing echo suppression on a mixed signal provided by an exemplary embodiment of the present application;

[0024] Figure 5 A flowchart of another audio signal processing method provided for an exemplary embodiment of the present application;

[0025] Figure 6 A schematic diagram of the structure of a conference system provided for an exemplary embodiment of the present application;

[0026] Figure 7 A flowchart of a conference method provided for an exemplary embodiment of the present application;

[0027] Figure 8a A schematic diagram of a scenario in which user A and user B communicate with each other via a terminal device, provided by an exemplary embodiment of the present application;

[0028] Figure 8bA flowchart of a remote communication method provided for an exemplary embodiment of the present application;

[0029] Figure 8c A schematic diagram of the structure of an audio and video communication system provided for an exemplary embodiment of the present application;

[0030] Figure 8d A schematic diagram of the structure of another audio and video communication system provided for an exemplary embodiment of the present application;

[0031] Figure 8e A flow chart of a testing method provided for an exemplary embodiment of the present application;

[0032] Fig. 9 A schematic diagram of the structure of a terminal device provided for an exemplary embodiment of the present application;

[0033] Fig.10 A schematic diagram of the structure of another terminal device provided for an exemplary embodiment of the present application;

[0034] Fig.11 A schematic diagram of the structure of a conference device provided for an exemplary embodiment of the present application;

[0035] Fig.12 A schematic diagram of the structure of a cloud device provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0037] Before introducing the embodiments of the present application in detail, the following explanation is made: the various methods provided in the following embodiments of the present application are applicable to any terminal device having a speaker and a microphone, in other words, to any terminal device having an audio signal input and output function, such as an audio device, a video device, an audio and video all-in-one machine, or an audio and video conferencing device, etc. Specifically, the terminal device may be a television, a laptop, a tablet computer, a desktop computer, a mobile phone, a speaker, a self-service robot, and various conference terminals, etc., having an audio signal input and output function, but is not limited to these.

[0038] In each embodiment of the present application, the terminal device is usually in a certain environment, and the environment in which the terminal device is located can be an indoor environment or an outdoor environment. The indoor environment can be a conference room, a cinema, a room, a shopping mall, a supermarket, a stadium, a temporary recording studio, and other closed or semi-closed space environments. The outdoor environment is a space environment corresponding to the indoor environment, such as a stadium, a square, or a park.

[0039] Regardless of the environment in which the terminal device is located, during use, reflections will occur due to obstacles such as walls, ceilings, and the ground in the surrounding environment. This phenomenon is called reverberation. The degree of reverberation in the environment in which the terminal device is located has an important impact on the listening effect of people. Because of the reverberation phenomenon in the environment in which the terminal device is located, the sound signal played by the speaker in the terminal device may return to the terminal device side after passing through one or more propagation paths due to reflection, and be picked up again by the microphone in the terminal device, thereby causing certain interference to the target signal that the microphone really needs to pick up, that is, generating an echo phenomenon.

[0040] If the terminal device is used in a communication system, for example, the local user uses the terminal device to make a call with the other party, or the local user uses the terminal device to conduct audio or video communication with the other party (for example, voice or video chatting with the other party, or voice or video conferencing with the other party), then the sound signal played by the speaker comes from the other party. After the sound signal from the other party is played by the speaker, it will return to the terminal device side after passing through one or more propagation paths in the environment of the terminal device and be picked up again by the microphone in the terminal device. If it is not suppressed, the sound signal from the other party will be transmitted to the other party again, so that the other party can hear its own voice, which generates the so-called acoustic echo.

[0041] Regardless of the type of echo, if the echo exceeds a reasonable range, the frequency transmission characteristics in the environment (such as a conference room or room) where the terminal device is located will deteriorate, and at the same time, a large amount of regenerative reverberation interference will be generated, causing serious interference to the target signal that the microphone needs to collect, affecting the signal quality. In the following embodiments of the present application, various echoes (such as ordinary echoes or acoustic echoes) are collectively referred to as echo noise.

[0042] In an embodiment of the present application, an acoustic echo canceller (AEC) technology can be used to cancel the echo noise caused by the microphone receiving the signal played by the speaker during the audio, video or audio and video communication process and other application scenarios. Although AEC technology has been developed for decades, the performance of AEC (i.e., the effect of canceling echo) is not ideal, and how to improve the performance of AEC is still an industry problem. The inventors of the embodiments of the present application have found through continuous research and analysis that the performance of AEC depends to a large extent on the application scenario, for example, the reverberation degree of the room, and thus proposed to incorporate the reverberation degree factor into the AEC process, and perform AEC processing on the signal collected by the microphone in combination with the reverberation degree to achieve better echo cancellation performance. In view of this, in the following embodiments of the present application, a reverberation detection method for automatically detecting the reverberation degree in the environment where the terminal device is located is provided, and the application of the reverberation degree factor in the AEC process and / or in the downlink playback link optimization process is also given. In the following embodiments of the present application, the implementation process of AEC can be implemented by hardware chips or software solutions.

[0043] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0044] Figure 1 The following is a flow chart of a reverberation detection method provided by an exemplary embodiment of the present application. Figure 1 As shown, the method includes:

[0045] 11. Using a microphone in the terminal device to collect echo noise in the environment where the terminal device is located under echo noise collection conditions, the echo noise is generated by a signal played by a speaker in the terminal device;

[0046] 12. Perform linear filtering on the echo noise collected by the microphone to obtain the residual noise of the echo noise;

[0047] 13. Calculate the reverberation parameters in the environment where the terminal device is located based on the echo noise and its residual noise.

[0048] In this embodiment, the reverberation parameter in the environment where the terminal device is located can be detected by echo noise. Specifically, an echo noise collection condition is created in the environment where the terminal device is located. The echo noise collection condition is an environmental condition that allows the microphone to clearly collect the echo noise generated by the signal played by the speaker. It is usually a relatively quiet environmental condition. Under this environmental condition, there is no signal from other sound sources at the local end, and only the signal played by the speaker. Based on this, the microphone can be used to collect the echo noise in the environment where the terminal device is located under the echo noise collection condition. The echo noise is generated by the signal played by the speaker in the terminal device after multiple reflections in the environment. Further, the echo noise is linearly filtered to obtain the residual noise of the echo noise; according to the echo noise and its residual noise, the reverberation parameter in the environment where the terminal device is located is calculated. Among them, the reverberation parameter is used to describe the degree of attenuation of the sound signal in the environment where the terminal device is located, that is, the reverberation parameter is a reflection of the reverberation degree in the environment where the terminal device is located. Optionally, the reverberation parameter can be represented by the reverberation time (Reverberation Time 60dB, RT60), and RT60 represents the time taken for the sound to attenuate 60dB. The larger the RT60 is, the longer it takes for the sound signal generated in the current environment to disappear.

[0049] In this embodiment, the implementation method of using a microphone to collect echo noise in the environment where the terminal device is located under the echo noise collection condition is not limited. In an optional embodiment, in the environment where the terminal device is located, it is required to keep relatively quiet from the beginning of the terminal device and within a specified time after powering on to create echo noise collection conditions, wherein the speaker will play a power-on sound signal during the power-on period and within a specified time after powering on, and the power-on sound signal will reach the microphone after multiple reflections in the environment after being played to form echo noise. In this case, the noise signal collected by the microphone within the specified time after the terminal device is powered on is mainly echo noise, so the noise signal collected by the microphone within the specified time after the terminal device is powered on can be obtained as echo noise. Among them, the specified time can be 1s, 2s, 5s, etc. after powering on, which can be flexibly set according to application requirements. In another optional embodiment, there are some application scenarios in which a user can use a terminal device to interact with a peer user by voice, such as communicating with a peer user by voice or video through an instant messaging application on the terminal device, or calling a peer user through the terminal device. In these application scenarios, in order to ensure the quality of communication or calls, the terminal equipment is usually in a relatively quiet environment (equivalent to the echo noise collection condition), and the local user and the opposite user usually speak or make speeches alternately. During the time interval when the local user is not speaking, the opposite user is generally speaking. At this time, the voice signal of the opposite user will be transmitted through the network to reach the local end and played through the speaker. The voice signal of the opposite user played by the speaker will reach the microphone after multiple reflections in the environment where the terminal equipment is located to form echo noise. In this case, the noise signal collected by the microphone during the time interval when the local user is not speaking is mainly echo noise. Therefore, the noise signal collected by the microphone during the time interval when the local user is not speaking can be obtained as echo noise.

[0050] In this embodiment, by analyzing and studying the transfer function of the echo noise generated in the environment where the terminal device is located, it is found that the echo noise caused by the reverberation characteristics of the environment can be divided into three parts: direct path, early reverberation, and late reverberation. Among them, the transfer function of the echo noise generated in the environment where the terminal device is located is as follows: Figure 2aAs shown in the figure, the horizontal axis represents time; the vertical axis represents signal amplitude. Among these three parts, direct sound and early reverberation can be simulated by linear filtering. In practical applications, the signal collected by the microphone can be linearly filtered, which can basically suppress the direct sound and early reverberation; however, linear filtering cannot simulate the late reverberation in the environment, and the components of the late reverberation will affect the convergence effect of linear filtering. Therefore, due to the different reverberation levels in different environments, the performance of linear filtering is different. That is to say, by analyzing the performance of the linear filter, the reverberation process in the environment can be indirectly reflected, especially the degree of late reverberation. In view of the research results, in an embodiment of the present application, the linear filtering effect of the linear filter on the echo noise collected by the microphone in the environment where the terminal device is located is used to characterize the reverberation level in the environment where the terminal device is located.

[0051] Specifically, after acquiring the echo noise collected by the microphone under the echo noise collection condition, the echo noise is further linearly filtered to obtain the residual noise of the echo noise. In this embodiment, the implementation method of linear filtering of the echo noise is not limited. Optionally, an adaptive filter (or linear filter) can be used to linearly filter the echo noise. Figure 2b As shown, the process of linearly filtering the echo noise based on the adaptive filter includes: inputting the signal (such as the power-on sound signal or the sound signal of the opposite user) played by the speaker to the adaptive filter, adjusting the weight vector of the filter by using different adaptive filtering algorithms in the adaptive filter, and estimating an approximate echo noise; further, using the estimated approximate echo noise to offset the echo noise collected by the microphone, the residual noise of the echo noise can be obtained. Among them, the commonly used adaptive algorithms include: minimum mean square error algorithm, least squares algorithm, zero forcing algorithm and steepest descent algorithm.

[0052] In this embodiment, the linear filtering effect of the echo noise collected by the microphone in the environment where the terminal device is located is defined according to the echo noise and its residual noise, that is, the reverberation degree in the environment where the terminal device is located is defined, and the reverberation parameters representing the reverberation program are calculated, but the implementation method of calculating the reverberation parameters according to the echo noise and its residual noise is not limited. In some optional embodiments of the present application, the reverberation parameters in the environment are characterized by the echo return loss enhancement (ERLE) parameter. Specifically, the process of calculating the reverberation parameters of the environment where the terminal device is located according to the echo noise and its residual noise includes: calculating the ERLE parameter according to the power value of the echo noise and its residual noise; according to the ERLE parameter, querying the corresponding relationship between the pre-configured ERLE parameter and the reverberation parameter to obtain the reverberation parameters of the current environment. Further optionally, the ratio of the power of the echo noise to the power of the residual noise can be calculated as the ERLE parameter. Among them, the higher the ERLE parameter value, the smaller the residual noise of the echo noise, that is, the better the performance of the linear filtering, which means that the reverberation degree in the environment is relatively simpler.

[0053] In this embodiment, the correspondence between the ERLE parameter of the terminal device in the environment in which it is located and the reverberation parameter of the environment in which it is located can be preconfigured. In the same environment, the ERLE parameter obtained is different according to the different terminal devices, but the reverberation parameter (such as RT60) in the same environment is the same. Therefore, for the same environment, the correspondence between the ERLE parameter and the reverberation parameter can be preconfigured for different terminal devices. Specifically, taking the environment in which the above-mentioned terminal device is located as an example, the above-mentioned terminal device can be deployed in the environment, and the ERLE parameter is obtained by the above-mentioned method, that is, the echo noise in the environment in which the terminal device is located is collected under the echo noise collection condition by using a microphone, and the echo noise is linearly filtered to obtain the residual noise of the echo noise; the ratio of the power of the echo noise to the power of its residual noise is calculated as the ERLE parameter; at the same time, the reverberation parameter under the environmental condition is tested, and then the correspondence between the ERLE parameter and the reverberation parameter of the terminal device in the environment is configured. In particular, a dedicated test device, such as a sound level meter, can be used to test the reverberation parameters under environmental conditions, such as RT60.

[0054] It is explained here that the reverberation detection method provided in this embodiment can be executed once when the terminal device is turned on. After obtaining the reverberation parameters in the environment where the terminal device is located, the reverberation parameters can be used continuously in the subsequent process; or, it can also be executed intermittently. After obtaining the reverberation parameters in the environment where the terminal device is located each time, the reverberation parameters can be used continuously until the next new reverberation parameters are calculated. Regardless of the execution method, during the entire reverberation detection process, no offline calibration is required. The reverberation parameters of the environment where the terminal device is located can be adaptively calculated by turning on the terminal device. It is highly efficient and the user is not aware of it, which is conducive to improving the user experience.

[0055] In the embodiments of the present application, the application scenarios of the above-mentioned reverberation detection method are not limited. For example, it can be applied in the field of architecture to evaluate the acoustic characteristics of buildings such as concert halls, auditoriums or theaters through the calculated reverberation parameters. Of course, in addition to using reverberation parameters to evaluate the acoustic characteristics of various buildings, in the following embodiments of the present application, new application scenarios are provided for reverberation parameters, that is, applying reverberation parameters to the AEC process and / or using reverberation parameters to optimize the downlink playback link in the audio and video playback scenario. It should be noted that the reverberation parameters used in the following embodiments can be obtained by the reverberation detection method provided in the aforementioned embodiments, or by other methods, without limitation. The application of reverberation parameters in the AEC process and in the downlink playback link optimization process is described in detail below.

[0056] Figure 3 The flowchart of an audio signal processing method provided by the exemplary embodiment of the present application is applicable to any terminal device having a speaker and a microphone, such as Figure 3 As shown, the method includes:

[0057] 31. Obtain reverberation parameters in the environment where the terminal device is located, wherein a signal played by a speaker in the terminal device may generate echo noise in the environment where the terminal device is located;

[0058] 32. Acquire a mixed signal collected by a microphone in the terminal device, where the mixed signal includes at least a target signal and a first echo noise, where the first echo noise is generated in the environment by a first signal played by a loudspeaker;

[0059] 33. Perform echo suppression on the mixed signal according to the reverberation parameter and the first signal generating the first echo noise, so as to obtain the target signal after suppressing the first echo noise.

[0060] In this embodiment, the terminal device has a speaker and a microphone. During use, the microphone can be used to collect sound signals from external sound sources, the speaker can be used to play the received sound signals externally, or the speaker can be used to play some sound signals generated inside the terminal device, such as power-on sound signals or sound signals collected by the microphone.

[0061] In actual applications, due to the reverberation phenomenon in the environment where the terminal device is located, the signal collected by the microphone is generally a mixed signal, which includes not only the target signal that the microphone actually needs to collect, but also the echo noise generated by the signal played by the speaker in the environment. Among them, the target signal that the microphone really needs to collect may be a noise signal, such as white noise, or it may be a useful signal emitted by the sound source at the local end, such as the sound signal of the user at the local end, or the sound signal played by other devices at the local end. In this embodiment, for the convenience of description and distinction, the echo noise collected by the microphone in actual application is called the first echo noise, and the signal that generates the first echo noise played by the speaker is called the first signal.

[0062] In order to perform echo suppression on the mixed signal collected by the microphone, in this embodiment, the reverberation parameter in the environment where the terminal device is located is also obtained, and the reverberation parameter can represent the reverberation degree in the environment. Generally, the reverberation parameter in the environment where the terminal device is located can be obtained before the microphone collects the mixed signal.

[0063] In this embodiment, the implementation method of obtaining the reverberation parameters in the environment where the terminal device is located is not limited. In an optional embodiment, the reverberation parameters of the environment where the terminal device is currently located can be estimated based on cepstrum filtering. First, a 1000Hz excitation signal is emitted by a built-in speaker; secondly, the audio signal received by the microphone is subjected to fast Fourier transform, logarithmic operation and discrete cosine transform to obtain the cepstrum parameters of the reverberation signal; then, the cepstrum parameters are filtered to separate the cepstrum parameters of the room filter from the cepstrum parameters of the reverberation signal; finally, the reverberation parameters of the room are obtained by inverse discrete cosine transform, exponential transform and inverse fast Fourier transform. In another optional embodiment, the reverberation detection method provided in the aforementioned embodiment can be used to obtain the reverberation parameters in the environment where the terminal device is located. Regarding the implementation method of obtaining the reverberation parameters in the environment where the terminal device is located using the reverberation detection method, please refer to the aforementioned embodiment, which will not be repeated here.

[0064] In this embodiment, echo suppression can be performed on the mixed signal according to the reverberation parameters and the first signal generating the first echo noise, so as to obtain the target signal after suppressing the first echo noise. In this embodiment, the implementation method of performing echo suppression on the mixed signal according to the reverberation parameters and the first signal to obtain the target signal after suppressing the first echo noise is not limited. Optionally, a specific implementation method includes: linearly filtering the mixed signal according to the first signal to obtain an intermediate signal, the intermediate signal including the target signal and the residual noise of the first echo noise; nonlinearly filtering the intermediate signal according to the reverberation parameters and the first signal to filter out the residual noise of the first echo noise to obtain the target signal. Optionally, an adaptive filter can be used to linearly filter the mixed signal.

[0065] Further optionally, an implementation method of performing nonlinear filtering on the intermediate signal based on the reverberation parameters and the first signal includes: estimating, based on the reverberation parameters and the first signal, the late reverberation noise generated by the first signal in the environment in which the terminal device is located, the late reverberation noise being nonlinear noise; and performing nonlinear filtering on the intermediate signal using the estimated late reverberation noise to filter out the residual noise of the first echo noise to obtain the target signal.

[0066] Further optionally, the residual noise of the first echo noise obtained after linear filtering may include not only the late reverberation noise (i.e., the nonlinear part) but also the linear residual part of the first echo noise. Based on this, before the intermediate signal is nonlinearly filtered according to the reverberation parameter and the first signal, the intermediate signal may be de-estimated to filter out the linear residual noise in the residual noise of the first echo noise.

[0067] Combine the following Figure 4 , a process of performing echo suppression on a mixed signal according to a reverberation parameter and a first signal to obtain a target signal after suppressing the first echo noise is described in detail.

[0068] Assume that the first signal played by the loudspeaker is x(n), h(n) is the transfer function between the loudspeaker and the microphone, and the first signal x(n) becomes the first echo noise d(n) after passing through the loudspeaker and reaching the microphone, wherein d(n)=x(n)*h(n); the mixed signal y(n) collected by the microphone includes not only the first echo noise d(n), but also the noise signal n(n) and the target signal s(n), that is, y(n)=d(n)+n(n)+s(n). In this embodiment, the target signal s(n) is a non-noise signal.

[0069] like Figure 4As shown, the first signal x(n) is input into a linear filter (such as an adaptive filter) for linear estimation to obtain an estimated signal d^(n) of the first echo noise d(n), and d^(n) is used to offset the first echo noise d(n) to obtain an intermediate signal e(n)=e u (n)+n(n)+s(n), where e u (n) = d(n) - d^(n).

[0070] For the first echo noise, it is mainly divided into three parts: direct sound, early reverberation and late reverberation. Linear filters cannot simulate the late reverberation of the environment in which the terminal equipment is located. Therefore, the residual signal obtained after linear filtering of the first echo noise contains a nonlinear residual part (i.e., the late reverberation part), and optionally, it can also contain a linear residual part. That is, the residual noise of the first echo noise can be expressed as: e u (n) = e uLinear (n)+e uReverb (n), where e uReverb (n) is the late reverberation, e uLinear (n) is the linear residual part. For the linear residual part of the first echo noise mixed in the intermediate signal, it can be filtered by using a de-estimation method before nonlinear filtering. For the nonlinear residual part of the first echo noise mixed in the intermediate signal (i.e., late reverberation), the intermediate signal can be nonlinearly filtered according to the reverberation parameter and the first signal to filter out the residual noise of the first echo noise (mainly refers to the nonlinear residual part in the residual noise, i.e., late reverberation) to obtain the target signal.

[0071] Specifically, Figure 4 As shown, the reverberation parameters and the first signal can be input into a late reverberation prediction module, and the reverberation parameters can be used to predict the late reverberation noise generated by the first signal in the environment where the terminal device is located, and then the late reverberation noise estimated by the late reverberation prediction module can be input into a nonlinear filter, and the estimated late reverberation noise can be used in the nonlinear filter to perform nonlinear filtering on the intermediate signal to filter out the nonlinear residual part of the first echo noise to obtain the target signal. In the nonlinear filter, the late reverberation noise estimated by the late reverberation prediction module can be used to offset the nonlinear residual part of the first echo noise signal mixed in the intermediate signal, so as to achieve the purpose of suppressing the first echo noise, thereby obtaining the target signal.

[0072] Optionally, the above nonlinear filtering can be implemented by using Wiener filtering. The application principle of Wiener filtering in this embodiment is: the transfer function h of the nonlinear filtering Re (n), the frequency response function corresponding to the transfer function is: in, represents the power spectral density of the estimated residual noise, S ee (Ω, n) represents the power spectral density of the residual noise, the noise signal, and the target signal. If the late reverberation and the linear residual part in the residual noise are independent stationary random processes, the estimated power spectral density of the residual noise can be expressed as the sum of the power spectral density of the linear residual part and the power spectral density of the late reverberation, that is, Among them, the residual noise is estimated by According to the adaptive algorithm of Wiener filtering, such as the minimum mean square error algorithm, the least squares algorithm, the zero forcing algorithm and the steepest descent algorithm, the The value of makes the nonlinear filtering achieve the best suppression effect on residual noise.

[0073] After the target signal is obtained, the specific processing of the target signal may depend on the application requirements and is not limited thereto. Optionally, after the target signal is obtained, the target signal may be sent, that is, the target signal may be sent out, for example, to a peer device. Alternatively, after the target signal is obtained, the target signal may be sent to a speaker, and the speaker may be used to replay the target signal.

[0074] In this embodiment, the reverberation parameters of the environment in which the terminal device on this end is located are introduced, and the reverberation parameters and the signal played by the speaker are used to perform echo suppression on the mixed signal collected by the microphone, thereby obtaining a signal after the echo noise is suppressed, which is conducive to achieving better echo cancellation performance and improving the quality of the signal collected by the microphone. Furthermore, in the audio and video communication scenario, after obtaining the signal after the echo noise is suppressed, the signal can also be sent to the opposite end device, which can improve the communication quality between the local end and the opposite end and improve the user experience. Furthermore, this method does not require offline calibration and can be used by turning on the terminal device. The reverberation parameters can be used to dynamically and adaptively suppress the echo noise, thereby improving the user experience.

[0075] The inventors of the embodiments of the present application have found that the reverberation parameters in the environment (such as a room or conference room) where the terminal device is located not only affect the sound pickup effect of the microphone, but also affect the listening experience of the speaker. If the reverberation is severe, the sound played by the speaker will be reflected multiple times by the environment, and the different reflected signals will be superimposed in the human ear, making it difficult to hear clearly. Based on this, the present application also provides a downlink optimization method based on reverberation parameters. In the embodiments of the present application, the downlink refers to the signal link that the sound signal passes through from arriving at the terminal device to being played out through the speaker in the terminal device.

[0076] Figure 5 A flowchart of another audio signal processing method provided by an exemplary embodiment of the present application is shown in FIG. Figure 5 As shown, the method includes:

[0077] 51. Obtain reverberation parameters in the environment where the terminal device is located;

[0078] 52. Optimize the playback parameters on the downlink playback link using the reverberation parameters;

[0079] 53. Process the signal to be played by the speaker using the optimized playback parameters;

[0080] 54. Use the speaker in the terminal device to play the processed signal.

[0081] In this embodiment, the reverberation parameters in the environment where the terminal device is located can be obtained before optimizing the playback parameters on the downlink playback link. Optionally, if the terminal device is in some relatively small environment, considering that the reverberation parameters in the environment will not change or change very little (the change can be ignored), the operation of obtaining the reverberation parameters can be performed once, and the reverberation parameters can be reused to optimize the playback parameters on the downlink playback link in the future, but it is not limited to this. Optionally, if the terminal device is in some dynamically changing environment, considering that the reverberation parameters will also change dynamically due to changes in the environment, the operation of obtaining the reverberation parameters can be dynamically performed according to the changes in the environment, and after each re-acquisition of the reverberation parameters, the newly obtained reverberation parameters can be used to optimize the playback parameters on the downlink playback link.

[0082] In this embodiment, the implementation method of obtaining the reverberation parameters in the environment where the terminal device is located is not limited. The implementation method of obtaining the reverberation parameters each time can refer to the description in the above embodiment, which will not be repeated here.

[0083] In the embodiments of the present application, the specific implementation method of optimizing the playback parameters on the downlink playback link using the reverberation parameters is not limited, and it can be determined according to the specific playback parameters on the downlink playback link. In some optional embodiments, the downlink playback link includes at least one of an equalizer, a limiter, and a gain module. The equalizer, the limiter, and the gain amplifier can be implemented in software or in hardware, and there is no limitation on this. Among them, the equalizer can adjust the amplification of the electrical signals of various frequency components in the sound signal, and compensate for the defects of the speaker and the sound field by adjusting the electrical signals of various frequencies; the limiter can limit the output signal amplitude within a certain range, that is, when the input voltage exceeds or falls below a certain reference value, the output voltage will be limited to a certain level (called the limiting level) and will no longer change with the input voltage; the gain amplifier is used to amplify the voltage of the input signal.

[0084] Based on the above, in an optional embodiment, reverberation parameters are used to optimize playback parameters on the downlink playback link, including at least one of the following: using reverberation parameters to optimize equalization parameters of an equalizer on the downlink playback link; using reverberation parameters to optimize limiting parameters of a limiter on the downlink playback link; using reverberation parameters to perform dynamic range control (DRC) on gain parameters on the downlink playback link.

[0085] In this embodiment, optimizing the equalization parameters of the equalizer on the downlink playback link by using the reverberation parameters is a process of adaptively adjusting the equalization parameters of the equalizer on the downlink playback link according to the reverberation parameters. Optionally, the frequency band with more severe reverberation may be a high-frequency band. Based on this, an implementation method of optimizing the equalization parameters of the equalizer on the downlink playback link by using the reverberation parameters includes: when the reverberation parameter is greater than the set reverberation parameter threshold, it means that the reverberation level in the environment where the terminal device is located is more serious, then the amplitude equalization parameter in the first frequency band can be increased, and the amplitude equalization parameter in the second frequency band can be reduced; wherein the lower limit frequency value of the first frequency band is greater than the set first threshold, the upper limit frequency value of the second frequency band is less than the set second threshold, and the second threshold is less than or equal to the first threshold. In short, the first frequency band is a relatively high frequency range, and the second frequency band is a relatively low frequency range. In this way, for sound signals in the high frequency range, the equalizer will increase the amount of amplification of this part of the sound signal; for sound signals in the low frequency range, the equalizer will reduce the amount of amplification of this part of the sound signal. This can ensure the sound quality of high-frequency signals and improve the listening effect of users.

[0086] In this embodiment, optimizing the limiting parameters of the upper limiter of the downlink playback link by using the reverberation parameters is a process of adaptively adjusting the limiting parameters of the upper limiter of the downlink playback link according to the reverberation parameters. Optionally, an implementation method of optimizing the limiting parameters of the upper limiter of the downlink playback link by using the reverberation parameters includes: when the reverberation parameter is greater than the set reverberation parameter threshold, it means that the reverberation level in the environment where the terminal device is located is relatively serious, then the first voltage threshold in the limiter can be increased, and the second voltage threshold in the limiter can be reduced; wherein the first voltage threshold is greater than the second voltage threshold. The first voltage threshold and the second voltage threshold refer to reference values ​​for the limiter to adjust the output signal voltage. For example, when the voltage of the input sound signal is greater than the first voltage threshold or less than the second voltage threshold, the limiter can limit the output voltage to a certain voltage (called the limiting voltage). This can ensure the sound quality and improve the user's listening effect.

[0087] In this embodiment, using reverberation parameters to perform DRC on the gain parameters on the downlink playback link is a process of adaptively adjusting the gain parameters on the downlink playback link based on the reverberation parameters. Optionally, the frequency band with more severe reverberation may be a high-frequency band. Based on this, an implementation method of using reverberation parameters to perform DRC on the gain parameters on the downlink playback link includes: when the reverberation parameter is greater than a set reverberation parameter threshold, increasing the gain parameter in the first frequency band and reducing the gain parameter in the second frequency band; the definitions of the first frequency band and the second frequency band are as described above and will not be repeated here. Simply put, for sound signals in the high-frequency range, the voltage gain of this part of the sound signal can be increased; for sound signals in the low-frequency range, the voltage gain of this part of the sound signal can be reduced. This can ensure the sound quality of high-frequency signals and improve the user's listening experience.

[0088] It is hereby explained that the various audio signal processing methods provided in the above embodiments of the present application can be applied in various scenarios involving audio signal input and output. For example, it can be applied in an application scenario in which at least two users make a phone call through a terminal device, or it can be applied in an application scenario in which at least two users conduct an audio chat or a video chat through a terminal device, and it can also be applied in a teleconference scenario. Among them, the teleconference can be an audio conference with pure audio, or it can be an audio and video conference with audio and video. The following takes an audio or audio and video conference as an example to exemplify the method provided in the above embodiments of the present application.

[0089] In this embodiment, if Figure 6 As shown, a conference system 600 is provided, including a local conference room 601 and a remote conference room 602. At least one conference device 603 is arranged in the local conference room, and at least one conference device 604 is deployed in the remote conference room. The local conference device 603 and the remote conference device 604 can be connected through Internet communication. Among them, the local conference device 603 and the remote conference device 604 both include a microphone 605 and a speaker 606. Further optionally, the local conference device 603 and the remote conference device 604 can also include other components such as a display screen and an input device. In this embodiment, the implementation form of the conference device is not limited, for example, it can be a smart phone, a laptop computer, a desktop computer, a conference phone, a video conferencing terminal or an audio and video all-in-one machine, or it can be a conference subsystem including conference microphones, conference speakers, projectors and other devices.

[0090] Before the meeting starts, the above conference system needs to be deployed. Optionally, it can be deployed in advance or temporarily. In addition, before the meeting starts, the correspondence between the ERLE parameters of the conference equipment on this end and the reverberation parameters in the conference room on this end can also be pre-tested. Based on the above conference system, the embodiment of the present application also provides a conference method, which is mainly described from the local end, and the other end can participate in the conference in a similar or the same way as the local end. Figure 7 As shown, the method includes:

[0091] 71. Obtain reverberation parameters in a conference scene where a local conference device is located. The local conference device includes a speaker and a microphone.

[0092] Optionally, after the conference system is deployed and before the meeting starts, echo noise collection conditions can be created in the local conference scene (i.e., the local conference room), such as a relatively quiet environment, and the local conference equipment can be turned on. At this time, the speaker in the local conference equipment will play a power-on sound signal to the outside. The power-on sound signal is a system startup sound effect that all local conference equipment has. If the power-on sound signal is normal, it means that the local conference equipment is normal. At this time, the power-on sound signal will generate echo noise in the local conference room, and the echo noise can be collected by the microphone in the local conference equipment. Since the local conference room is relatively quiet during the startup period, the noise signal collected by the microphone is mainly the echo noise. The microphone can send the echo noise to the processor of the local conference device, and the processor performs linear filtering on the echo noise to obtain the residual noise of the echo noise; then, the ratio of the power of the echo noise to the power of its residual noise is calculated to obtain an ERLE parameter that can characterize the linear filtering performance; further, the correspondence between the ERLE parameters of the local conference device obtained in advance and the reverberation parameters in the local conference room is queried through the ERLE parameters to obtain the reverberation parameters in the local conference room.

[0093] or,

[0094] After the conference starts, the local user and the opposite user speak alternately. During the time interval when the local user does not speak but the opposite user speaks, the sound signal of the opposite end will be transmitted to the local conference device through the transmission link between the local conference device and the opposite conference device, and the speaker in the local conference device will play the sound signal from the opposite end; wherein, the sound signal from the opposite end played by the speaker will generate echo noise in the local conference room, and the echo noise will be collected by the microphone in the local conference device. Since the local user does not speak at this time, the signal collected by the microphone is mainly echo noise. At this time, the microphone can send the collected echo noise to the processor of the local conference device, and the processor performs linear filtering on the echo noise to obtain the residual noise of the echo noise; then, the ratio of the power of the echo noise to the power of its residual noise is calculated to obtain the ERLE parameter that can characterize the linear filtering performance; further, the corresponding relationship between the ERLE parameter of the local conference device obtained in advance and the reverberation parameter in the local conference room is queried through the ERLE parameter to obtain the reverberation parameter in the local conference room.

[0095] 72. During a conference, a microphone is used to collect sound signals at the local end. The sound signals collected by the microphone are mixed with echo noise, which is generated by the sound signals at the other end played by the loudspeaker.

[0096] During an actual conference, a microphone in a local conference device may be used to collect a local sound signal, wherein the sound signal collected by the microphone is mixed with echo noise, which is generated by the sound signal of the other end played by the speaker.

[0097] Optionally, before the microphone in the local conference device collects the local sound signal, it may receive the sound signal from the opposite end transmitted by the opposite end conference device. If the sound signal from the opposite end is received, the speaker in the local conference device will play the sound signal from the opposite end. In this case, the echo noise collected by the microphone is mainly generated in the local conference room by the sound signal from the opposite end played by the speaker.

[0098] 73. According to the reverberation parameters and the sound signal of the other end, echo suppression is performed on the sound signal collected by the microphone, and the sound signal after echo suppression is transmitted to the conference device of the other end.

[0099] After the microphone collects the sound signal, the local conference device needs to transmit the sound signal collected by the microphone to the opposite conference device, and then the opposite conference device plays the sound signal to the opposite user through its speaker. Before transmitting the sound signal collected by the microphone to the opposite end, the sound signal collected by the microphone can be echo suppressed according to the reverberation parameters and the sound signal of the opposite end, and the sound signal after echo suppression is transmitted to the opposite conference device, which can improve the sound quality transmitted to the opposite device, prevent the opposite user from hearing the echo of his own voice, improve the call quality in the meeting, and improve the user's meeting experience.

[0100] In an optional embodiment, both the local conference device and the opposite conference device are audio and video devices, and the audio and video device refers to a device that supports both audio and video, such as a mobile phone, a laptop computer, an audio and video all-in-one machine, or a conference device with a camera. In this embodiment, during the conference, the local conference device can collect the local sound signal through the microphone and the local video screen through its camera; further, in the process of transmitting the sound signal after echo suppression to the opposite conference device, the local video screen collected by the local audio and video device can be transmitted to the opposite conference device at the same time, and the opposite conference device will play the local sound signal and the video screen to the opposite user at the same time. Similarly, the opposite conference device will also collect the opposite sound signal and the opposite video screen at the same time, and transmit them to the local conference device at the same time, and the local conference device will play the opposite sound signal and the video screen to the local user at the same time.

[0101] Among them, the detailed process of performing echo suppression on the sound signal collected by the microphone according to the reverberation parameter and the sound signal of the other end can be referred to the aforementioned embodiment, which will not be repeated here.

[0102] In an alternative embodiment, if Figure 7 As shown, the method of this embodiment also includes:

[0103] 74. Use the reverberation parameters to optimize the playback parameters on the downlink playback link in the local conference device;

[0104] 75. Whenever receiving a sound signal from the opposite end sent by the opposite end conference device, optimizing the sound signal from the opposite end using the optimized playback parameters;

[0105] 76. Use the speaker in the conference equipment at this end to play the optimized sound signal from the other end.

[0106] In this embodiment, when the reverberation parameters in the conference room at this end are obtained, the reverberation parameters can also be used to optimize the playback parameters on the downlink playback link in the conference device at this end. Optionally, the playback parameters on the downlink playback link in the conference device at this end are optimized using the reverberation parameters, including at least one of the following: optimizing the equalization parameters of the equalizer (Equalizer) on the downlink playback link using the reverberation parameters; optimizing the limiting parameters of the limiter (Limiter) on the downlink playback link using the reverberation parameters; performing DRC on the gain parameters on the downlink playback link using the reverberation parameters. For a detailed description of each optimization method, please refer to the aforementioned embodiment, which will not be repeated here.

[0107] In this embodiment, the equalizer, limiter and other modules on the downlink playback link can be dynamically adjusted according to the reverberation parameters in the conference room at the local end, so that the frequency band with severe reverberation in the sound signal from the other end can be slightly suppressed or limited, thereby solving the problem that the sound signal in the frequency band with severe reverberation cannot be heard clearly after being played back and then reflected multiple times and superimposed in the human ear, which is beneficial to improving the hearing experience of the local user.

[0108] Of course, in addition to the above meeting scenarios, Figure 8a As shown, in a scenario where user A and user B make a call via a mobile phone or user A and user B conduct online video or voice communication via instant messaging software on terminal devices such as a mobile phone, a laptop computer or a tablet computer, especially when the hands-free function is turned on, on the one hand, reverberation parameters can be used in the device at each end to suppress the echo of the local sound signal collected by the microphone, and on the other hand, reverberation parameters can be used to optimize the playback parameters on the downlink playback link, thereby improving the sound quality of the other end user played, improving the call quality between the two parties, and enhancing the user experience.

[0109] Whether it is the above-mentioned conference scenario, or the scenario in which the user makes a call through a mobile phone or uses a terminal device such as a mobile phone, a laptop or a tablet computer to conduct online video or voice, it can be summarized as an audio and video communication scenario. Based on the audio and video communication scenario, the embodiment of the present application also provides a remote communication method, such as Figure 8b As shown, the method includes:

[0110] 81b. Obtain reverberation parameters in the scene where the local communication device is located, where the local communication device includes a speaker and a microphone;

[0111] 82b. During the remote communication process, a microphone is used to collect a sound signal of the local end, and the sound signal collected by the microphone is mixed with echo noise, which is generated by the sound signal played by the speaker;

[0112] 83b. Perform echo suppression on the sound signal collected by the microphone according to the reverberation parameter and the sound signal played by the speaker, and transmit the sound signal after echo suppression to the opposite communication device.

[0113] In addition to the above-mentioned conference scenarios, scenarios where users make calls through mobile phones or use mobile phones, laptops or tablet computers to conduct online video or voice communication, audio and video communication scenarios also include online education scenarios, online medical scenarios, online chat scenarios, etc. For example, in online education scenarios, teachers can use terminal devices to live broadcast lectures to students, and students can use terminal devices to answer questions raised by teachers. In this process, the terminal device on the teacher's side (referred to as the teacher terminal) can be used Figure 8b The remote communication method shown in the figure (the local communication device is the teacher terminal at this time) can also be applied to the terminal device on the student side (referred to as the student terminal) Figure 8b The remote communication method shown (in this case, the local communication device is the student terminal) can improve the call quality of the online education system and enhance the students' listening experience. For another example, in an online medical system, a doctor has opened an online voice or video consultation or diagnosis service. Patients can ask questions about their own problems, and the doctor will answer the patient's questions and give medical advice. During the whole process, the doctor and the patient need to communicate by voice or video through the terminal device. During this process, the terminal device on the doctor's side (referred to as the doctor's terminal) can be used Figure 8b The remote communication method shown in the figure (the local communication device is the doctor terminal in this case) can also be applied to the terminal device on the patient side (referred to as the patient terminal) Figure 8b The remote communication method shown (in this case, the local communication device is the patient terminal) can improve the call quality of the online medical system, improve the patient consultation experience, and promote the development and widespread application of online medical care. For example, in an online chat system, users can chat by voice or video through terminal devices. Figure 8b The remote communication method shown (in this case, the local communication device is the terminal of the chat user) is used to improve the chat quality and user experience.

[0114] In the above embodiments, the audio and video communication system obtains the reverberation parameters in the environment through the terminal device, and performs echo suppression on the signal to be sent according to the reverberation parameters to improve the signal quality of the signal to be sent, or the downlink playback link can be optimized according to the reverberation parameters. In addition, the reverberation parameters can be obtained by using the cloud device and provided to the terminal device so that the terminal device can suppress the echo of the signal to be sent according to the reverberation parameters, and / or, optimize the parameters according to the reverberation parameters, and use the optimized parameters to optimize the downlink. Based on this, this embodiment also provides an audio and video communication system, such as Figure 8c As shown, the audio and video communication system includes: a local terminal device 801c, a peer terminal device 802c and a cloud device 803c.

[0115] The local terminal device 801c and the opposite terminal device 802c both include a microphone 804c and a speaker 805c. Further, the local terminal device 801c and the opposite terminal device 802c may also include other components such as a display screen and an input device. In this embodiment, the cloud device 803c is not limited. For example, the cloud device 803c may be a server device in an audio and video communication system, such as Figure 8c Or the cloud device 803c can be a cloud device independent of the server device 806c in the audio and video communication system, such as Figure 8d In addition, the cloud device 803c may be one or more.

[0116] In the case where the cloud device 803c is a server device in the audio and video communication system, the cloud device 803c is connected to the local terminal device 801c and the opposite terminal device 802c for communication, that is, the local terminal device 801c and the opposite terminal device 802c can communicate based on the cloud device 803c. On the one hand, the cloud device 803c can provide services such as communication management and control, policy control, encoding and decoding, encryption and decryption or compression, and on the other hand, provide the audio and video content of the local terminal device 801c to the opposite terminal device 802c, or provide the audio and video content of the opposite terminal device 802c to the local terminal device 801c. In addition, the cloud device 803c can also calculate the reverberation parameters of the environment in which the corresponding terminal device is located based on the echo noise collected by the local terminal device 801c or the opposite terminal device 802c, and provide the reverberation parameters to the corresponding terminal device.

[0117] In the case where the cloud device 803c is independent of the server device 806c in the audio and video communication system, Figure 8dAs shown, the server device provides services such as communication management and control, policy control, encoding and decoding, encryption and decryption or compression, and is responsible for providing the audio and video content of the local terminal device 801c to the opposite terminal device 802c, or providing the audio and video content of the opposite terminal device 802c to the local terminal device 801c. The cloud device 803c is responsible for calculating the reverberation parameters of the environment in which the corresponding terminal device is located based on the echo noise collected by the local terminal device 801c or the opposite terminal device 802c, and providing the reverberation parameters to the corresponding terminal device.

[0118] Regardless of the above situation, the cloud device of this embodiment can provide a test method. Whether it is the local terminal device 801c or the opposite terminal device 802c, the role in the above test method is the same. Therefore, for the convenience of description, the local terminal device 801c and the opposite terminal device 802c are collectively referred to as terminal devices in the following test method. Figure 8e As shown, the method includes:

[0119] 81e. Sending a test signal to a terminal device in the audio and video communication system, so that the terminal device can play the test signal in the environment using its speaker;

[0120] 82e. receiving echo noise collected by a microphone of the terminal device and returned by the terminal device, where the echo noise is generated by the test signal in the environment;

[0121] 83e. Linearly filter the echo noise to obtain residual noise of the echo noise, and calculate reverberation parameters in the environment based on the echo noise and its residual noise;

[0122] 84e. Send the reverberation parameters to the terminal device so that the terminal device can optimize the signal to be sent and / or the signal to be played.

[0123] In this embodiment, the test signal may be an audio content that can be played on a speaker of the terminal device. In order to facilitate the terminal device to collect echo noise, in this embodiment, the terminal device may play the test signal under the echo noise collection condition, collect the echo noise and send it to the cloud device.

[0124] In this embodiment, after receiving the echo noise sent by the terminal device, the cloud device can perform linear filtering on the echo noise to obtain the residual noise of the echo noise, and calculate the reverberation parameters in the environment based on the echo noise and its residual noise. For the implementation method of calculating the reverberation parameters in the environment based on the echo noise, please refer to the above embodiment, which will not be repeated here.

[0125] In this embodiment, after the cloud device calculates the reverberation parameters, it provides the reverberation parameters to the terminal device so that the terminal device can optimize the signal to be sent and / or the signal to be played. Among them, the signal to be sent refers to the signal collected by the terminal device through its microphone and needs to be sent to another terminal device, for example, it can be the sound of the user speaking on the terminal device side. For example, the terminal device can use the reverberation parameters to suppress the echo of the signal to be sent, improve the signal quality of the signal to be sent, improve the sound quality heard by the other end, and improve the user experience. The signal to be played refers to the signal that the terminal device needs to play out through the speaker (it can be the signal of the terminal device itself or the signal from the other end). For example, before the speaker plays, the terminal device optimizes the signal to be played through the reverberation parameters, for example, the equalization parameters of the equalizer on the downlink playback link are optimized by the reverberation parameters; the limiter parameters of the limiter on the downlink playback link are optimized by the reverberation parameters; the gain parameters on the downlink playback link are DRC, etc.; then, the speaker is used to play the signal after equalization, limiter and / or gain processing to improve the sound quality. It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 11 to 13 can be device A; for another example, the execution subject of steps 11 and 12 can be device A, and the execution subject of step 13 can be device B; and so on.

[0126] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 11, 12, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0127] Fig. 9 The following is a schematic diagram of a terminal device provided by an exemplary embodiment of the present application. Fig. 9 As shown, the terminal device includes: a speaker 91, a microphone 92, a memory 94 and a processor 95.

[0128] The memory 94 is used to store computer programs and can be configured to store various other data to support operations on the terminal device. Examples of such data include instructions for any application or method operating on the terminal device, contact data, phone book data, messages, pictures, videos, etc.

[0129] The processor 95 is coupled to the memory 94 and is used to execute the computer program in the memory 94 to: obtain reverberation parameters in the environment in which the terminal device is located, wherein the signal played by the speaker 91 may generate echo noise in the environment; and obtain a mixed signal collected by the microphone 92, wherein the mixed signal includes at least a target signal and a first echo noise, and the first echo noise is generated by the first signal played by the speaker 91; and perform echo suppression on the mixed signal according to the reverberation parameters and the first signal to obtain a target signal after suppressing the first echo noise.

[0130] In an optional embodiment, the processor 95 is further used to: send a target signal; or use a speaker to play the target signal. For example, in some test scenarios, the local microphone needs to collect the target signal and then play it out through the local speaker. In this case, after obtaining the target signal, the local speaker can be used to play the target signal to complete the performance test of the entire audio loop.

[0131] In an optional embodiment, when the processor 95 obtains the reverberation parameters in the environment where the terminal device is located, it is specifically used to: obtain a second echo noise collected by the microphone under echo noise collection conditions, where the second echo noise is generated by a second signal played by the speaker; perform linear filtering on the second echo noise to obtain the residual noise of the second echo noise; and calculate the reverberation parameters in the environment where the terminal device is located based on the second echo noise and its residual noise.

[0132] In an optional embodiment, when the processor 95 obtains the second echo noise collected by the microphone under the echo noise collection condition, it is specifically used to: obtain the signal collected by the microphone within a specified time after the terminal device is turned on as the second echo noise; wherein the speaker plays the power-on sound signal within the specified time after the power-on; or, obtain the signal collected by the microphone during the time interval when the local user is not speaking as the second echo noise, wherein the speaker plays the sound signal of the opposite user during the time interval when the local user is not speaking.

[0133] In an optional embodiment, when the processor 95 calculates the reverberation parameters in the environment where the terminal device is located based on the second echo noise and its residual noise, it is specifically used to: calculate the ERLE parameters in the environment where the terminal device is located based on the power values ​​of the second echo noise and its residual noise; and query the correspondence between the pre-configured ERLE parameters and the reverberation parameters based on the ERLE parameters to obtain the reverberation parameters in the environment where the terminal device is located.

[0134] In an optional embodiment, when the processor 95 performs echo suppression on the mixed signal according to the reverberation parameters and the first signal to obtain a target signal after suppressing the first echo noise, the processor 95 is specifically used to: perform linear filtering on the mixed signal according to the first signal to obtain an intermediate signal, wherein the intermediate signal includes the target signal and the residual noise of the first echo noise; and perform nonlinear filtering on the intermediate signal according to the reverberation parameters and the first signal to filter out the residual noise of the first echo noise to obtain the target signal.

[0135] In an optional embodiment, when the processor 95 performs nonlinear filtering on the intermediate signal according to the reverberation parameters and the first signal to filter out the residual noise of the first echo noise to obtain the target signal, it is specifically used to: estimate, according to the reverberation parameters and the first signal, the late reverberation noise generated by the first signal in the environment where the terminal device is located, the late reverberation noise being nonlinear noise; and perform nonlinear filtering on the intermediate signal using the estimated late reverberation noise to filter out the residual noise of the first echo noise to obtain the target signal.

[0136] In an optional embodiment, before performing nonlinear filtering on the intermediate signal according to the reverberation parameter and the first signal, the processor 95 is further configured to: de-estimate the intermediate signal to filter out linear residual noise in the residual noise of the first echo noise.

[0137] In an optional embodiment, the processor 95 is further used to: optimize the playback parameters on the downlink playback link using the reverberation parameters; process the signal to be played by the speaker using the optimized playback parameters; and play the processed signal using the speaker 91.

[0138] In an optional embodiment, when the processor 95 optimizes the playback parameters on the downlink playback link using the reverberation parameters, it is specifically used to perform at least one of the following: optimizing the equalization parameters of the equalizer on the downlink playback link using the reverberation parameters; optimizing the limiting parameters of the limiter on the downlink playback link using the reverberation parameters; performing DRC on the gain parameters on the downlink playback link using the reverberation parameters.

[0139] In an optional embodiment, when the processor 95 uses the reverberation parameters to optimize the equalization parameters of the equalizer on the downlink playback link, it is specifically used to: when the reverberation parameters are greater than the set reverberation parameter threshold, increase the amplitude equalization parameters in the first frequency band and reduce the amplitude equalization parameters in the second frequency band; wherein the lower limit frequency value of the first frequency band is greater than the set first threshold, the upper limit frequency value of the second frequency band is less than the set second threshold, and the second threshold is less than or equal to the first threshold.

[0140] In an optional embodiment, when the processor 95 uses the reverberation parameter to optimize the limiting parameter of the limiter in the downlink playback link, it is specifically used to: when the reverberation parameter is greater than the set reverberation parameter threshold, increase the first voltage threshold in the limiter and reduce the second voltage threshold in the limiter; wherein the first voltage threshold is greater than the second voltage threshold.

[0141] In an optional embodiment, when the processor 95 uses the reverberation parameter to perform DRC on the gain parameter on the downlink playback link, it is specifically used to: when the reverberation parameter is greater than a set reverberation parameter threshold, increase the gain parameter in the first frequency band and reduce the gain parameter in the second frequency band; wherein the lower limit frequency value of the first frequency band is greater than the set first threshold, the upper limit frequency value of the second frequency band is less than the set second threshold, and the second threshold is less than or equal to the first threshold.

[0142] In an optional embodiment, the terminal device is an audio and video integrated machine, an audio device, a video device or an audio and video conferencing device.

[0143] In an optional embodiment, the environment where the terminal device is located is an indoor environment, which may be a conference room, a cinema, a room, a shopping mall, a supermarket, a stadium, a temporary recording studio, or any other closed or semi-closed space environment.

[0144] Further, if Fig. 9 As shown, the terminal device also includes: a communication component 96, a display 97, a power supply component 98 and other components. Fig. 9 Only some components are shown schematically, which does not mean that the terminal equipment only includes Fig. 9 Components shown.

[0145] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement the steps that can be executed by the terminal device in the above-mentioned audio signal processing method and reverberation detection method embodiments.

[0146] The embodiment of the present application also provides a terminal device, which can be implemented as a communication device in an audio and video communication system. When implemented as a communication device, its implementation structure is similar to Fig. 9 The implementation structure of the terminal device shown is the same or similar, and can be referred to Fig. 9 The communication device provided in this embodiment is implemented as shown in the structure of the terminal device. Fig. 9The difference between the terminal devices in the illustrated embodiments mainly lies in: the functions implemented by the processor executing the computer program stored in the memory are different. For the communication device provided in this embodiment, its processor executes the computer program stored in the memory, which can be used to: obtain the reverberation parameters in the scene where the local communication device is located, and the local communication device includes a speaker and a microphone; in the process of remote communication, use the microphone to collect the sound signal of the local end, and the sound signal collected by the microphone is mixed with echo noise, and the echo noise is generated by the sound signal played by the speaker; according to the reverberation parameters and the sound signal played by the speaker, the sound signal collected by the microphone is echo suppressed, and the sound signal after echo suppression is transmitted to the opposite communication device.

[0147] In an optional embodiment, the local communication device is a student terminal or a teacher terminal in an online education system; or, the local communication device is a patient terminal or a doctor terminal in an online medical system; or, the local communication device is a user terminal in an online chat system.

[0148] Accordingly, the embodiment of the present application further provides a computer-readable storage medium storing a computer program, which can achieve the above-mentioned Figure 8b The steps in the method embodiment shown may be executed by a terminal device.

[0149] Fig.10 This is a schematic diagram of the structure of another terminal device provided by an exemplary embodiment of the present application. Fig.10 As shown, the terminal device includes: a speaker 1001, a memory 1004 and a processor 1005.

[0150] The memory 1004 is used to store computer programs and can be configured to store various other data to support operations on the terminal device. Examples of such data include instructions for any application or method operating on the terminal device, contact data, phone book data, messages, pictures, videos, etc.

[0151] Processor 1005 is coupled to memory 1004 and is used to execute a computer program in memory 1004 to: obtain reverberation parameters in the environment in which the terminal device is located; optimize the playback parameters on the downlink playback link using the reverberation parameters; process the signal to be played by the speaker using the optimized playback parameters; and play the processed signal using speaker 1001.

[0152] In an optional embodiment, when the processor 1005 optimizes the playback parameters on the downlink playback link using the reverberation parameters, it is specifically used to perform at least one of the following: optimizing the equalization parameters of the equalizer on the downlink playback link using the reverberation parameters; optimizing the limiting parameters of the limiter on the downlink playback link using the reverberation parameters; performing DRC on the gain parameters on the downlink playback link using the reverberation parameters.

[0153] In an optional embodiment, when the processor 1005 uses the reverberation parameters to optimize the equalization parameters of the equalizer on the downlink playback link, it is specifically used to: when the reverberation parameters are greater than the set reverberation parameter threshold, increase the amplitude equalization parameters in the first frequency band and reduce the amplitude equalization parameters in the second frequency band; wherein the lower limit frequency value of the first frequency band is greater than the set first threshold, the upper limit frequency value of the second frequency band is less than the set second threshold, and the second threshold is less than or equal to the first threshold.

[0154] In an optional embodiment, when the processor 1005 uses the reverberation parameter to optimize the limiting parameter of the limiter in the downlink playback link, it is specifically used to: when the reverberation parameter is greater than the set reverberation parameter threshold, increase the first voltage threshold in the limiter and reduce the second voltage threshold in the limiter; wherein the first voltage threshold is greater than the second voltage threshold.

[0155] In an optional embodiment, when the processor 1005 uses the reverberation parameter to perform DRC on the gain parameter on the downlink playback link, it is specifically used to: when the reverberation parameter is greater than a set reverberation parameter threshold, increase the gain parameter in the first frequency band and reduce the gain parameter in the second frequency band; wherein the lower limit frequency value of the first frequency band is greater than the set first threshold, the upper limit frequency value of the second frequency band is less than the set second threshold, and the second threshold is less than or equal to the first threshold.

[0156] Further, if Fig.10 As shown, the terminal device also includes: a communication component 1006, a display 1007, a power component 1008, a microphone 1009 and other components. Fig.10 Only some components are shown schematically, which does not mean that the terminal equipment only includes Fig.10 Components shown.

[0157] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement the steps that can be executed by the terminal device in the above-mentioned audio signal processing method embodiment.

[0158] In an optional embodiment, the terminal device provided in the above embodiment of the present application can be implemented as a conference device. Fig.11 The following is a schematic diagram of a conference device provided by an exemplary embodiment of the present application. Fig.11As shown, the conference device includes: a speaker 1101 , a microphone 1102 , a memory 1104 and a processor 1105 .

[0159] The memory 1104 is used to store computer programs and can be configured to store various other data to support operations on the conference device. Examples of such data include instructions for any application or method used to operate on the conference device, contact data, phone book data, messages, pictures, videos, etc.

[0160] The processor 1105 is coupled to the memory 1104 and is used to execute the computer program in the memory 1104, so as to: obtain the reverberation parameters in the conference scene where the conference device at the local end is located; during the conference, use the microphone to collect the sound signal at the local end, the sound signal collected by the microphone is mixed with echo noise, and the echo noise is generated by the sound signal played by the speaker; according to the reverberation parameters and the sound signal played by the speaker, the sound signal collected by the microphone is echo suppressed, and the sound signal after echo suppression is transmitted to the conference device at the other end.

[0161] In an optional embodiment, before using a microphone to collect the sound signal of the local end, the processor 1105 is also used to: receive the sound signal from the other end transmitted by the conference device of the other end, and use a speaker to play the sound signal from the other end, and the echo noise is generated by the sound signal from the other end played by the speaker.

[0162] In an optional embodiment, the local conference device and the opposite conference device are audio and video devices, and the processor 1105 is also used to: collect the local video image during the process of collecting the local sound signal, and transmit the collected local video image to the opposite conference device during the process of transmitting the echo-suppressed sound signal to the opposite conference device.

[0163] Further, if Fig.11 As shown, the conference device also includes: a communication component 1106, a display 1107, a power supply component 1108 and other components. Fig.11 Only some components are shown schematically, which does not mean that the conference equipment only includes Fig.11 Components shown.

[0164] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement the various steps that can be executed by the conference device in the above-mentioned conference method embodiment.

[0165] Fig.12 A schematic diagram of the structure of a cloud device provided by an exemplary embodiment of the present application. Fig.12 As shown, the cloud device includes: a memory 1204 and a processor 1205 .

[0166] The memory 1204 is used to store computer programs and can be configured to store various other data to support operations on the cloud device. Examples of such data include instructions for any application or method operating on the cloud device, contact data, phone book data, messages, pictures, videos, etc.

[0167] The processor 1205 is coupled to the memory 1204 and is used to execute the computer program in the memory 1204, so as to: send a test signal to a terminal device in the audio and video communication system, so that the terminal device uses its speaker to play the test signal in the environment; receive the echo noise collected by the microphone of the terminal device and returned by the terminal device, where the echo noise is generated by the test signal in the environment; perform linear filtering on the echo noise to obtain the residual noise of the echo noise, and calculate the reverberation parameters in the environment based on the echo noise and its residual noise; send the reverberation parameters to the terminal device, so that the terminal device can optimize the signal to be sent and / or the signal to be played.

[0168] In an optional embodiment, the cloud device may be a server device in the audio and video communication system, or a cloud device independent of the server device in the audio and video communication system.

[0169] Further, if Fig.12 As shown, the cloud device also includes: a communication component 1206, a power supply component 1208 and other components. Fig.12 Only some components are shown schematically, which does not mean that the cloud device only includes Fig.12 Components shown.

[0170] Above Figure 9-12 The memory in can be implemented by any type of volatile or non-volatile storage device or a combination of them, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0171] Above Figure 9-12The communication component in is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0172] Above Figure 9-11 The display in the embodiment includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0173] Above Figure 9-12 The power supply component in the device provides power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0174] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0175] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0176] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0177] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0178] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0179] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0180] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0181] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0182] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A reverberation detection method, applicable to a terminal device, wherein the terminal device comprises a speaker and a microphone, characterized in that: The method comprises: Using a microphone to collect echo noise in an environment where the terminal device is located under echo noise collection conditions, the echo noise is generated in the environment by a signal played by a loudspeaker; Performing linear filtering on the echo noise to obtain residual noise of the echo noise; A linear filtering effect of the echo noise collected by the microphone in the environment where the terminal device is located is defined according to the echo noise and its residual noise, and a reverberation parameter of the environment where the terminal device is located is determined according to the linear filtering effect of the echo noise.

2. The method according to claim 1, characterized in that The echo noise in the environment where the terminal device is located is collected using a microphone under echo noise collection conditions, including: Acquire a noise signal collected by the microphone within a specified time after the terminal device is turned on as the echo noise; wherein the speaker plays a power-on sound signal within a specified time after the terminal device is turned on; or, A noise signal collected by the microphone during a time interval when the local user does not speak is obtained as echo noise, wherein the loudspeaker plays a voice signal of the opposite user during the time interval when the local user does not speak.

3. The method according to claim 2, characterized in that Defining a linear filtering effect of the echo noise collected by the microphone in the environment where the terminal device is located according to the echo noise and its residual noise, and determining a reverberation parameter of the environment where the terminal device is located according to the linear filtering effect of the echo noise, including: Calculating an echo round-trip loss enhancement ERLE parameter in the environment where the terminal device is located according to the power values ​​of the echo noise and its residual noise; According to the ERLE parameter, the correspondence between the pre-configured ERLE parameter and the reverberation parameter is queried to obtain the reverberation parameter of the environment in which the terminal device is located.

4. A conference method, characterized in that: include: Obtaining reverberation parameters in a conference scene where a local conference device is located according to the method of claim 1, wherein the local conference device includes a speaker and a microphone, the local conference device refers to a terminal device, and the conference scene where the local conference device is located refers to an environment where the terminal device is located; During the conference, a microphone is used to collect a sound signal of the local end, and the sound signal collected by the microphone is mixed with echo noise, and the echo noise is generated by the sound signal played by the speaker; According to the reverberation parameter and the sound signal played by the loudspeaker, echo suppression is performed on the sound signal collected by the microphone, and the sound signal after echo suppression is transmitted to the opposite end conference device.

5. The method according to claim 4, characterized in that Before using the microphone to collect the sound signal of the local end, it also includes: A sound signal from the opposite end transmitted by the opposite end conference device is received, and the sound signal from the opposite end is played by a loudspeaker, wherein the echo noise is generated by the sound signal from the opposite end played by the loudspeaker.

6. The method according to claim 4 or 5, characterized in that: If the local conference device and the opposite conference device are audio and video devices, the method further includes: The video image of the local end is collected in the process of collecting the sound signal of the local end, and the collected video image of the local end is transmitted to the conference device of the opposite end in the process of transmitting the sound signal after echo suppression to the conference device of the opposite end.

7. A remote communication method, characterized in that: include: Acquire the reverberation parameters in the scene where the local communication device is located according to the method of claim 1, the local communication device includes a speaker and a microphone, the local communication device refers to a terminal device, and the scene where the local communication device is located refers to the environment where the terminal device is located; During the remote communication process, a microphone is used to collect a sound signal of the local end, and the sound signal collected by the microphone is mixed with echo noise, and the echo noise is generated by the sound signal played by the speaker; According to the reverberation parameter and the sound signal played by the loudspeaker, echo suppression is performed on the sound signal collected by the microphone, and the sound signal after echo suppression is transmitted to the opposite communication device.

8. The method according to claim 7, characterized in that The local communication device is a student terminal or a teacher terminal in the online education system; or, The local communication device is a patient terminal or a doctor terminal in an online medical system; or, The local communication device is a user terminal in the online chat system.

9. An audio signal processing method, applicable to a terminal device, wherein the terminal device comprises a speaker and a microphone, characterized in that: The method comprises: Acquiring reverberation parameters in an environment where a terminal device is located, wherein a signal played by a speaker in the terminal device may generate echo noise in the environment; Acquire a mixed signal collected by a microphone in a terminal device, where the mixed signal includes at least a target signal and a first echo noise, where the first echo noise is generated by a first signal played by a speaker; According to the reverberation parameter and the first signal, echo suppression is performed on the mixed signal to obtain a target signal after suppressing the first echo noise; Obtaining the reverberation parameters in the environment where the terminal device is located includes: obtaining a second echo noise collected by a microphone under echo noise collection conditions, where the second echo noise is generated by a second signal played by a speaker; linearly filtering the second echo noise to obtain residual noise of the second echo noise; defining the linear filtering effect of the echo noise collected by the microphone in the environment where the terminal device is located according to the second echo noise and its residual noise, and determining the reverberation parameters of the environment where the terminal device is located according to the linear filtering effect of the echo noise.

10. The method according to claim 9, characterized in that Also includes: sending the target signal; Alternatively, the target signal is played by using the speaker.

11. The method according to claim 9, characterized in that Acquiring the second echo noise collected by the microphone under the echo noise collection condition, including: A signal collected by the microphone within a specified time after the terminal device is turned on is obtained as a second echo noise; wherein the speaker plays a power-on sound signal within a specified time after the terminal device is turned on; or, A signal collected by the microphone during a time interval when the local user is not speaking is obtained as the second echo noise, wherein the loudspeaker plays a voice signal of the opposite user during the time interval when the local user is not speaking.

12. The method according to claim 9, characterized in that Defining a linear filtering effect of the echo noise collected by the microphone in the environment where the terminal device is located according to the second echo noise and its residual noise, and determining a reverberation parameter of the environment where the terminal device is located according to the linear filtering effect of the echo noise, including: Calculating an echo round-trip loss enhancement ERLE parameter in an environment where the terminal device is located according to the power value of the second echo noise and its residual noise; According to the ERLE parameter, the correspondence between the pre-configured ERLE parameter and the reverberation parameter is queried to obtain the reverberation parameter in the environment where the terminal device is located.

13. The method according to claim 9, characterized in that According to the reverberation parameter and the first signal, echo suppression is performed on the mixed signal to obtain a target signal after suppressing the first echo noise, including: linearly filtering the mixed signal according to the first signal to obtain an intermediate signal, wherein the intermediate signal includes a target signal and residual noise of the first echo noise; The intermediate signal is nonlinearly filtered according to the reverberation parameter and the first signal to filter out residual noise of the first echo noise to obtain the target signal.

14. The method according to claim 13, characterized in that The method includes performing nonlinear filtering on the intermediate signal according to the reverberation parameter and the first signal to filter out residual noise of the first echo noise to obtain the target signal, including: estimating, according to the reverberation parameter and the first signal, late reverberation noise generated by the first signal in the environment where the terminal device is located, the late reverberation noise being nonlinear noise; The intermediate signal is nonlinearly filtered using the estimated late reverberation noise to filter out residual noise of the first echo noise to obtain the target signal.

15. The method according to claim 14, characterized in that Before performing nonlinear filtering on the intermediate signal according to the reverberation parameter and the first signal, the method further includes: The intermediate signal is de-estimated to filter out linear residual noise in the residual noise of the first echo noise.

16. The method according to any one of claims 9 to 15, characterized in that: Also includes: Optimizing playback parameters on a downlink playback link using the reverberation parameters; Processing the signal to be played by the speaker using the optimized playback parameters; as well as The processed signal is played using the speaker.

17. The method according to claim 16, characterized in that Optimizing the playback parameters on the downlink playback link by using the reverberation parameters includes at least one of the following: Optimizing the equalization parameters of the equalizer on the downlink playback link using the reverberation parameters; Utilizing the reverberation parameters to optimize the limiting parameters of the limiter in the downlink playback link; The reverberation parameter is used to perform dynamic range adjustment (DRC) on the gain parameter on the downlink playback link.

18. The method according to claim 17, characterized in that Optimizing the equalization parameters of the equalizer on the downlink playback link by using the reverberation parameters includes: When the reverberation parameter is greater than a set reverberation parameter threshold, increasing the amplitude equalization parameter in the first frequency band and decreasing the amplitude equalization parameter in the second frequency band; The lower limit frequency value of the first frequency band is greater than a set first threshold, the upper limit frequency value of the second frequency band is less than a set second threshold, and the second threshold is less than or equal to the first threshold.

19. The method according to claim 17, characterized in that Optimizing the limiting parameters of the limiter in the downlink playback link by using the reverberation parameters includes: When the reverberation parameter is greater than a set reverberation parameter threshold, the first voltage threshold in the limiter is increased, and the second voltage threshold in the limiter is decreased; wherein the first voltage threshold is greater than the second voltage threshold.

20. The method according to claim 17, characterized in that Performing DRC on the gain parameters on the downlink playback link using the reverberation parameters includes: When the reverberation parameter is greater than a set reverberation parameter threshold, increasing the gain parameter in the first frequency band and decreasing the gain parameter in the second frequency band; The lower limit frequency value of the first frequency band is greater than a set first threshold, the upper limit frequency value of the second frequency band is less than a set second threshold, and the second threshold is less than or equal to the first threshold.

21. The method according to any one of claims 9 to 15, characterized in that: The terminal equipment is located in an indoor environment.

22. A testing method, applicable to a cloud device, characterized in that: include: Sending a test signal to a terminal device in the audio and video communication system, so that the terminal device plays the test signal in the environment using its speaker; receiving echo noise collected by a microphone of the terminal device and returned by the terminal device, wherein the echo noise is generated by the test signal in the environment; Performing linear filtering on the echo noise to obtain residual noise of the echo noise, and defining a linear filtering effect of the echo noise collected by a microphone in the environment where the terminal device is located according to the echo noise and its residual noise, and determining a reverberation parameter of the environment where the terminal device is located according to the linear filtering effect of the echo noise; The reverberation parameters are sent to the terminal device so that the terminal device can optimize the signal to be sent and / or the signal to be played.

23. The method according to claim 22, characterized in that The cloud device is a server device in the audio and video communication system, or a cloud device independent of the server device in the audio and video communication system.

24. An audio signal processing method, applicable to a terminal device, wherein the terminal device comprises a speaker and a microphone, characterized in that: The method comprises: Acquiring reverberation parameters in the environment where the terminal device is located according to the method of claim 1; Optimizing playback parameters on a downlink playback link using the reverberation parameters; Processing the signal to be played by the speaker using the optimized playback parameters; and The processed signal is played using the speaker.

25. The method according to claim 24, characterized in that Optimizing the playback parameters on the downlink playback link by using the reverberation parameters includes at least one of the following: Optimizing the equalization parameters of the equalizer on the downlink playback link using the reverberation parameters; Utilizing the reverberation parameters to optimize the limiting parameters of the limiter in the downlink playback link; The reverberation parameter is used to perform dynamic range adjustment (DRC) on the gain parameter on the downlink playback link.

26. A terminal device, characterized in that: include: Speakers, microphones, memory, and processors; The memory is used to store computer programs; The processor, coupled to the memory, is configured to execute the computer program to: Acquiring reverberation parameters in an environment where the terminal device is located, wherein the signal played by the speaker may generate echo noise in the environment; and Acquire a mixed signal collected by the microphone, where the mixed signal includes at least a target signal and a first echo noise, where the first echo noise is generated by a first signal played by a speaker; According to the reverberation parameter and the first signal, echo suppression is performed on the mixed signal to obtain a target signal after suppressing the first echo noise; When acquiring the reverberation parameters in the environment where the terminal device is located, the processor is specifically used to: acquire a second echo noise collected by the microphone under echo noise collection conditions, where the second echo noise is generated by a second signal played by the speaker; linearly filter the second echo noise to obtain the residual noise of the second echo noise; define the linear filtering effect of the echo noise collected by the microphone in the environment where the terminal device is located according to the second echo noise and its residual noise, and determine the reverberation parameters of the environment where the terminal device is located according to the linear filtering effect of the echo noise.

27. The terminal device according to claim 26, characterized in that: The terminal device is an audio and video integrated machine, an audio device, a video device or an audio and video conferencing device.

28. A terminal device, characterized in that: include: speakers, memory, and processors; The memory is used to store computer programs; The processor, coupled to the memory, is configured to execute the computer program to: Acquire the reverberation parameters in the environment where the terminal device is located according to the method of claim 1; Optimizing playback parameters on a downlink playback link using the reverberation parameters; Processing the signal to be played by the speaker using the optimized playback parameters; as well as The processed signal is played using the speaker.

29. A conference device, characterized in that: include: Speakers, microphones, memory, and processors; The memory is used to store computer programs; The processor, coupled to the memory, is configured to execute the computer program to: Acquire the reverberation parameters in the conference scene where the local conference device is located according to the method of claim 1, the local conference device refers to the terminal device, and the conference scene where the local conference device is located refers to the environment where the terminal device is located; During the conference, a microphone is used to collect a sound signal of the local end, and the sound signal collected by the microphone is mixed with echo noise, and the echo noise is generated by the sound signal played by the speaker; According to the reverberation parameter and the sound signal played by the loudspeaker, echo suppression is performed on the sound signal collected by the microphone, and the sound signal after echo suppression is transmitted to the opposite end conference device.

30. A terminal device, characterized in that: include: Speakers, microphones, memory, and processors; The memory is used to store computer programs; The processor, coupled to the memory, is configured to execute the computer program to: Acquire the reverberation parameters in the scene where the local communication device is located according to the method of claim 1, wherein the local communication device refers to the terminal device, and the scene where the local communication device is located refers to the environment where the terminal device is located; During the remote communication process, a microphone is used to collect a sound signal of the local end, and the sound signal collected by the microphone is mixed with echo noise, and the echo noise is generated by the sound signal played by the speaker; According to the reverberation parameter and the sound signal played by the loudspeaker, echo suppression is performed on the sound signal collected by the microphone, and the sound signal after echo suppression is transmitted to the opposite communication device.

31. A cloud device, characterized in that: include: Memory and processor; The memory is used to store computer programs; The processor, coupled to the memory, is configured to execute the computer program to: Sending a test signal to a terminal device in the audio and video communication system, so that the terminal device plays the test signal in the environment using its speaker; receiving echo noise collected by a microphone of the terminal device and returned by the terminal device, wherein the echo noise is generated by the test signal in the environment; Performing linear filtering on the echo noise to obtain residual noise of the echo noise, and defining a linear filtering effect of the echo noise collected by the microphone in the environment where the terminal device is located according to the echo noise and its residual noise, and determining a reverberation parameter of the environment where the terminal device is located according to the linear filtering effect of the echo noise; The reverberation parameters are sent to the terminal device so that the terminal device can optimize the signal to be sent and / or the signal to be played.

32. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to implement the steps in the method according to any one of claims 1 to 25.

Images