Dynamic speech enhancement component optimization

Abstract

Systems, methods, and computer-readable storage devices are disclosed for optimizing speech enhancement components to be used in a speech communication system using non-invasive speech quality evaluation. A method includes receiving audio data from a computing device over a network, the audio data including speech; detecting a first quality of the speech of the audio data using a trained non-invasive speech quality assessment (NISQA) model, the trained NISQA model being trained to automatically detect a quality of the speech; determining whether the computing device is a low quality endpoint based on a first quality of the speech of the audio data; and transmitting at least one speech enhancement component from the computing device to at least one server device over the network when the computing device is determined to be a low quality endpoint.

Description

Technical Field

[0001] The present disclosure relates to speech enhancement by reducing echo, noise, dereverberation, etc. In particular, the present disclosure relates to speech enhancement by using a non-intrusive speech quality assessment model using a neural network that determines speech enhancement components to be used in a speech communication system. Background Art

[0002] In voice communication systems, audio signals may be affected by echo, background noise, reverberation, enhancement algorithms, network impairments, etc. Providers of voice communication systems may estimate the perceived quality of audio signals when trying to provide optimal and reliable services to their customers. For example, voice quality prediction may be useful during network design and development, and for monitoring and improving customers' Quality of Experience (QoE).

[0003] To determine QoE, one approach may include subjective listening tests to provide an accurate method for evaluating the perceived quality of a speech signal. In this approach, the estimated quality is an average of the users' determinations. For example, the average of the scores of all participants under a particular condition is called the mean opinion score (MOS) and represents the perceived speech quality after equalizing various factors. However, such an approach may be cumbersome, time consuming, and cannot be done in real time.

[0004] Intrusive methods for determining speech quality calculate the perceptually weighted distance between a clean reference and the contaminated signal to estimate the perceived speech quality. Intrusive methods are considered more accurate because they provide a higher correlation with subjective assessments. Because these measurements are intrusive, they cannot be done in real time and require a reference to a clean speech signal to estimate the MOS.

[0005] To overcome the limitations of subjective listening and intrusive estimation of speech quality, non-intrusive speech quality assessment (NISQA) models using neural networks have been implemented. Such NISQA models can be used to dynamically optimize speech enhancement components in telecommunication pipelines to improve QoE. Speech enhancement (SE) components are critical to telecommunications for reducing echo, noise, de-reverberation, etc. Many of these components can be based on acoustic digital signal processing (ADSP) algorithms, but these components can be replaced by deep learning components. However, deep neural network (DNN) models are only as good as the data used to train the deep neural network, and it is impossible to have completely representative training data. Therefore, some new SE components may bring more harm than good compared to their previous SE components. Therefore, it is necessary to dynamically select speech enhancement components that optimize the quality of user experience in real time. Summary of the invention

[0006] According to certain embodiments, systems, methods, and computer-readable media are disclosed for optimizing speech enhancement components in a speech communication system using non-intrusive speech quality assessment.

[0007] According to certain embodiments, a computer-implemented method for optimizing a speech enhancement component in a speech communication system using non-intrusive speech quality assessment is disclosed. A method includes: receiving audio data from a computing device over a network, the audio data including speech; detecting a first quality of the speech of the audio data using a trained non-intrusive speech quality assessment (NISQA) model, the trained NISQA model being trained to automatically detect the quality of speech; determining whether the computing device is a low-quality endpoint based on the first quality of the speech of the audio data; and when the computing device is determined to be a low-quality endpoint, transmitting at least one speech enhancement component from the computing device to at least one server device over the network.

[0008] According to certain embodiments, a system for optimizing a speech enhancement component in a speech communication system using non-intrusive speech quality assessment is disclosed. A system includes: a data storage device storing instructions for optimizing a speech enhancement component in a speech communication system using non-intrusive speech quality assessment; and a processor configured to execute the instructions to perform a method, the method including: receiving audio data from a computing device via a network, the audio data including speech; using a trained non-intrusive speech quality assessment (NISQA) model to detect a first quality of the speech of the audio data, the trained NISQA model being trained to automatically detect the quality of speech; determining whether the computing device is a low-quality endpoint based on the first quality of the speech of the audio data; and when the computing device is determined to be a low-quality endpoint, transmitting at least one speech enhancement component from the computing device to the system via the network.

[0009] According to certain embodiments, a computer-readable storage device storing instructions is disclosed, wherein when executed by a computer, the instructions cause the computer to perform a method for optimizing a speech enhancement component in a speech communication system using non-intrusive speech quality assessment. A method of the computer-readable storage device includes: receiving audio data from a computing device over a network, the audio data including speech; detecting a first quality of speech of the audio data using a trained non-intrusive speech quality assessment (NISQA) model, the trained NISQA model being trained to automatically detect the quality of speech; determining whether the computing device is a low-quality endpoint based on the first quality of speech of the audio data; and when determining that the computing device is a low-quality endpoint, transmitting at least one speech enhancement component from the computing device to at least one server device over the network.

[0010] Additional objects and advantages of the disclosed embodiments will be set forth in part in the following description, and in part will be apparent from the description, or may be learned by practicing the disclosed embodiments. The objects and advantages of the disclosed embodiments will be realized and obtained by means of the elements and combinations particularly pointed out in the appended claims.

[0011] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosed embodiments, as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In the course of the following detailed description, reference will be made to the accompanying drawings. The accompanying drawings illustrate different aspects of the present disclosure, and where appropriate, reference numerals illustrating the same structures, components, materials, and / or elements in different drawings are similarly labeled. It should be understood that various combinations of structures, components, and / or elements other than those specifically illustrated are contemplated and within the scope of the present disclosure.

[0013] In addition, many embodiments of the present disclosure are described and illustrated herein. The present disclosure is not limited to any single aspect or its embodiment, nor to any combination and / or arrangement of these aspects and / or embodiments. In addition, each aspect and / or its embodiment of the present disclosure may be used alone or in combination with one or more other aspects and / or its embodiments of the present disclosure. For the sake of brevity, certain arrangements and combinations are not discussed and / or illustrated separately herein.

[0014] Figure 1 An exemplary speech enhancement architecture for a speech communication system pipeline according to an embodiment of the present disclosure is depicted.

[0015] Figure 2 Another exemplary speech enhancement architecture of a speech communication system pipeline according to an embodiment of the present disclosure is depicted.

[0016] Figure 3 Yet another exemplary speech enhancement architecture of a speech communication system pipeline according to an embodiment of the present disclosure is depicted.

[0017] Figure 4 Yet another exemplary speech enhancement architecture of a speech communication system pipeline according to an embodiment of the present disclosure is depicted.

[0018] Figure 5 An exemplary cloud-based speech enhancement architecture for a speech communication system pipeline according to an embodiment of the present disclosure is depicted.

[0019] Figure 6A method for optimizing a speech enhancement component to be used in a speech communication system using non-intrusive speech quality assessment according to an embodiment of the present disclosure is depicted.

[0020] Figure 7 Another method for optimizing a speech enhancement component to be used in a speech communication system using non-intrusive speech quality assessment according to an embodiment of the present disclosure is depicted.

[0021] Figure 8 A high-level diagram of an exemplary computing device that may be used in accordance with the systems, methods, and computer-readable media disclosed herein is depicted in accordance with an embodiment of the present disclosure.

[0022] Fig. 9 A high-level diagram of an exemplary computing system that can be used in accordance with the systems, methods, and computer-readable media disclosed herein is depicted in accordance with an embodiment of the present disclosure.

[0023] Likewise, many embodiments are described and illustrated herein. The present disclosure is not limited to any single aspect or its embodiment, nor to any combination and / or arrangement of these aspects and / or embodiments. Each aspect and / or its embodiment of the present disclosure may be used alone or in combination with one or more other aspects and / or its embodiments of the present disclosure. For the sake of brevity, many of these combinations and arrangements are not discussed separately herein. DETAILED DESCRIPTION

[0024] Those skilled in the art will recognize that various embodiments and examples of the present disclosure can be practiced according to the description. All of these embodiments and examples are intended to be included within the scope of the present disclosure.

[0025] As used herein, the terms "comprises," "comprising," "have," "having," "include," "including," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements, but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. The term "exemplary" is used in the sense of "example" rather than "ideal." Additionally, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, the phrase "X employs A or B" is intended to mean any natural inclusive permutation. For example, the phrase "X employs A or B" is satisfied by any of the following instances: X employs A; X employs B; or X employs both A and B. Furthermore, the articles "a" and "an," as used in the present application and the appended claims, should generally be interpreted to mean "one or more," unless otherwise specified or clear from the context to indicate a singular form.

[0026] For the sake of brevity, conventional techniques related to other functional aspects of the system and server for performing the method and the system and server (and the various operating components of the system) may not be described in detail herein. In addition, the connecting lines shown in the various figures included herein are intended to represent exemplary functional relationships and / or physical couplings between the various elements. It should be noted that many alternative and / or additional functional relationships or physical connections may exist in the embodiments of the subject matter.

[0027] Reference will now be made in detail to exemplary embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numerals will be used throughout the drawings to refer to the same or like parts.

[0028] The present disclosure relates generally and particularly to a method of using machine learning to dynamically optimize speech enhancement components, such as the NISQA model using a neural network, to improve QoE in a voice communication system. As discussed herein, there are various aspects of speech enhancement that can be improved by using the NIQA model.

[0029] Embodiments of the present disclosure provide a machine learning method that can be used to dynamically optimize a speech enhancement component of a speech communication system. In particular, a neural network can be used as a machine learning method. More specifically, a NISQA using a neural network can be implemented. The method of an embodiment of the present disclosure can be based on training one or more NISQAs using a neural network to dynamically optimize a speech enhancement component of a speech communication system. The neural networks that can be used include, but are not limited to, deep neural networks, convolutional neural networks, recursive neural networks, and the like.

[0030] Non-limiting examples of speech enhancement components include music detection, acoustic echo cancellation, noise suppression, dereverberation, echo detection, automatic gain control, voice activity detection, jitter buffer management, packet loss concealment, and the like.

[0031] A NISQA using a neural network may be trained using a population-based QoE estimation usage dataset. An example of a NISQA using a neural network is shown in Table 1 below. Although Table 1 depicts one type of neural network-based NISQA, other types of neural network-based NISQA may be implemented within the scope of the present disclosure.

[0032] Table 1

[0033]

[0034]

[0035] Another type of NISQA using neural networks includes convolutional neural network (CNN) architectures. For example, a CNN architecture can be applied to a 2D image array and can include two operations: convolution and pooling. A convolutional layer can be responsible for mapping detected features from the receptive field in the previous layer into its cells, which can be called a feature map and is the result of a weighted sum of input features passed through a nonlinearity such as ReLU. A pooling layer can take the maximum and / or average of a set of adjacent feature maps, reducing dimensionality by merging semantically similar features.

[0036] Yet another type of NISQA using a neural network includes a multilayer perceptron (MLP). Such a deep neural network (DNN) can learn feature representations by mapping input features into a linearly separable feature space, which can be implemented by successive linear combinations of input variables followed by nonlinear activation functions. As described above, NISQA based on other types of neural networks can be implemented within the scope of the present disclosure.

[0037] Embodiments as disclosed herein dynamically optimize speech enhancement components of a speech communication system.One solution may be to use NISQA to dynamically and / or in real time optimize one or more speech enhancement components in a speech communication system pipeline.

[0038] Figure 1 An exemplary speech enhancement architecture 100 of a speech communication system pipeline according to an embodiment of the present disclosure is depicted. Specifically, Figure 1 A speech communication system pipeline with multiple speech enhancement components is depicted. Figure 1As shown, microphone 102 can capture audio data, including speech of users of the communication system, among others. The audio data captured by microphone 102 can be processed by one or more speech enhancement components of speech enhancement architecture 100. As described above, non-limiting examples of speech enhancement components include music detection, acoustic echo cancellation, noise suppression, dereverberation, echo detection, automatic gain control, voice activity detection, jitter buffer management, packet loss concealment, etc.

[0039] Figure 1 Depicted is audio data received by a music detection component 104, which can detect whether music is detected in the captured audio data. For example, if the music detection component 104 detects audio data, the music detection component 104 can notify the user that music has been detected and / or turn off the music. The audio data captured by the microphone 102 can also be received and processed by one or more other speech enhancement components (e.g., an echo cancellation component 106, a noise suppression component 108, and / or a de-reverberation component 110). One or more of the echo cancellation component 106, the noise suppression component 108, and / or the de-reverberation component 110 can be a speech enhancement component that provides microphone and speaker alignment, such as a microphone 101 and a speaker 134. The echo cancellation component 106 (also referred to as an acoustic echo cancellation component) can receive audio data captured by the microphone 102 and speaker data played by the speaker 134. The echo cancellation component 106 can be used to eliminate acoustic feedback between the speaker 134 and the microphone 102 in the voice communication system.

[0040] The noise suppression component 108 can receive the audio data captured by the microphone 102 and the speaker data played by the speaker 134. The noise suppression component 108 can process the audio data and the speaker data to isolate the voice from other sounds and music during playback. For example, when the microphone 102 is turned on, the background noise around the user (e.g., turning pages, slamming doors, barking dogs, etc.) may distract other users. The noise suppression component 108 can remove such noise around the user in the voice communication system.

[0041] The dereverberation component 110 may receive audio data captured by the microphone 102 and speaker data played by the speaker 134. The dereverberation component 110 may process the audio data and the speaker data to remove the effects of reverberation, such as reverberant sounds captured by microphones including the microphone 102.

[0042] The audio data may be speech-enhanced audio data after being processed by one or more speech enhancement components (e.g., one or more of the echo cancellation component 106, the noise suppression component 108, and / or the dereverberation component 110), and further processed by one or more other speech enhancement components. For example, the speech-enhanced audio data may be received and / or processed by one or more of the echo detector 112 and / or the automatic gain control component 114. The echo detector 112 may use the speech-enhanced audio data to detect whether there is an echo in the speech-enhanced audio data and notify the user of the echo. The automatic gain control component 114 may use the speech-enhanced audio data to amplify and / or increase the volume of the speech-enhanced audio data based on whether the voice activity detector 116 detects speech.

[0043] The voice activity detector 116 can receive the voice-enhanced audio data processed by the automatic gain control component 114 and can detect whether voice activity is detected in the voice-enhanced audio data. Based on the detection by the voice activity detector 116, the user can be notified that he or she is talking when silent, the notification can be automatically turned on or off, and / or the automatic gain control component 114 can be instructed to amplify and / or increase the volume of the voice-enhanced audio data.

[0044] The speech-enhanced audio data may then be received by encoder 118 and / or NISQA 120. Encoder 118 may be an audio codec, such as an AI-powered audio codec, for example a SATIN encoder, which is a digital signal processor with machine learning. Encoder 118 may encode (i.e., compress) the audio data for transmission over network 122. After encoding, encoder 118 may transmit the encoded speech-enhanced audio data to network 122, where other components of the voice communication system are provided. Other components of the voice communication system may then transmit audio data of the user and / or other users of the voice communication system over network 122.

[0045] The jitter buffer management component 124 may receive audio data transmitted over the network 122 and process the audio data. For example, the jitter buffer management component 124 may buffer packets of audio data to allow the decoder 126 to receive the audio data at evenly spaced intervals. Because the audio data is transmitted over the network 122, there may be variations in packet arrival times, i.e., jitter, that may occur due to network congestion, timing drift, and / or routing changes. The jitter buffer management component 124 at the receiving end of the voice communication system may delay arriving packets so that the user experiences a clear connection with minimal sound distortion.

[0046] The audio data from the jitter buffer management component 124 may then be received by a decoder 126. The decoder 126 may be an audio codec, such as an AI-powered audio codec, for example a SATIN decoder, which is a digital signal processor with machine learning. The decoder 126 may decode (i.e., decompress) the audio data received over the network 122. After decoding, the decoder 126 may provide the decoded audio data to a packet loss concealment component 128.

[0047] The packet loss concealment component 128 may receive the decoded audio data and may process the decoded audio data to conceal gaps in the audio stream caused by data transmission failures in the network 122. The processing results may be provided to one or more of the following: the network quality classifier 130, the call quality estimator component 132, and / or the speaker 134.

[0048] The network quality classifier 130 can classify the quality of the connection to the network 122 based on the information received from the jitter buffer management component 124 and / or the packet loss concealment component 128, and the network quality classifier 130 can inform the user of the quality of the connection to the network 122, such as poor, medium, excellent, etc. When the connection to the network 122 is through the public switched telephone network (PSTN), the call quality estimator component 132 can estimate the quality of the call. The speaker 134 can play the decoded audio data as speaker data. The speaker data can also be provided to one or more of the following: the echo cancellation component 106, the noise suppression component 108 and / or the de-reverberation component 110.

[0049] As described above, the voice-enhanced audio data can then be received by NISQA120. NISQA120 can be one or more of the above-mentioned NISQAs using a neural network, and the neural network can be trained to detect the quality of the voice-enhanced audio data. When the quality of the voice-enhanced audio data is detected, the result can be provided to the optimized voice enhancement component 136. The optimized voice enhancement component 136 can determine whether one or more of the voice enhancement components can be changed to another voice enhancement component to improve QoE. In an embodiment, the optimized voice enhancement component 136 can be stored on the user's device, and two or more of the various voice enhancement components discussed above can be stored. Based on the results of NISQA120, the optimized voice enhancement component 136 can dynamically and / or in real time change various voice enhancement components, such as music detection component 104, echo cancellation component 106, noise suppression component 108, de-reverberation component 110, echo detector 112, automatic gain control component 114, jitter buffer management component 124 and / or packet loss concealment component 128. For clarity in the figures, the optimized speech enhancement components 136, 236, 336, 436, 536, etc. are not shown as being connected to each speech enhancement component, but may be connected to each speech enhancement component.

[0050] For example, the optimized speech enhancement component 136 may change the noise suppression component 108 to another type of noise suppression component. Then, the new quality of the speech-enhanced audio data may be detected by NISQA 120. If the new quality of the speech-enhanced audio data is higher than the original quality of the speech-enhanced audio data, the optimized speech enhancement component 136 may keep the changed noise suppression component 108. If the new quality of the speech-enhanced audio data is not higher than the original quality of the speech-enhanced audio data, the optimized speech enhancement component 136 may change the changed noise suppression component 108 back to the original noise suppression component 108 or another type of noise suppression component.

[0051] An exemplary brute-force pseudocode for implementing the optimization is described below.

[0052]

[0053] Figure 2 Another exemplary speech enhancement architecture 200 of a speech communication system pipeline according to an embodiment of the present disclosure is depicted. Specifically, Figure 2 A speech communication system pipeline having multiple speech enhancement components is depicted. Figure 2 Similar to Figure 1The illustrated embodiment, except that the optimized speech enhancement component 236 resides on the network 122 and / or in the cloud, and the NISQA 220 sends the optimized speech enhancement component 236 over the network 122. The NISQA 220 can be one or more of the above-described NISQAs using a neural network, which can be trained to detect the quality of the speech-enhanced audio data. When the quality of the speech-enhanced audio data is detected, the results can be provided to the optimized speech enhancement component 236 over the network 122. The optimized speech enhancement component 236 can determine whether one or more of the speech enhancement components can be changed to another speech enhancement component to improve the QoE. In an embodiment, the optimized speech enhancement component 236 is sent back to the user's device, where the various speech enhancement components can be stored. Based on the results of NISQA220, the optimized speech enhancement component 236 may dynamically and / or in near real time change various speech enhancement components, such as the music detection component 104, the echo cancellation component 106, the noise suppression component 108, the de-reverberation component 110, the echo detector 112, the automatic gain control component 114, the jitter buffer management component 124 and / or the packet loss concealment component 128, depending on the connection speed to the network and / or the quality of the connection to the network 122.

[0054] Figure 3 Yet another exemplary speech enhancement architecture 300 of a speech communication system pipeline according to an embodiment of the present disclosure is depicted. Figure 3 Similar to Figure 2In the illustrated embodiment, except that NISQA 320 and optimized speech enhancement component 236 reside on network 122 and / or in the cloud. NISQA 320 can receive encoded speech-enhanced audio data and detect the quality of the encoded speech-enhanced audio data. NISQA 320 can be one or more of the above-mentioned NISQAs using a neural network, and the neural network can be trained to detect the quality of the speech-enhanced audio data. After detecting the quality of the encoded speech-enhanced audio data, the results can be provided to the optimized speech enhancement component 336 via network 122. The optimized speech enhancement component 336 can determine whether one or more of the speech enhancement components can be changed to another speech enhancement component to improve QoE. In an embodiment, the optimized speech enhancement component 336 is sent back to the user's device, where various speech enhancement components can be stored. Based on the results of NISQA 320, the optimized speech enhancement component 336 may dynamically and / or in near real time change various speech enhancement components, such as the music detection component 104, the echo cancellation component 106, the noise suppression component 108, the dereverberation component 110, the echo detector 112, the automatic gain control component 114, the jitter buffer management component 124 and / or the packet loss concealment component 128, depending on the connection speed to the network and / or the quality of the connection to the network 122.

[0055] Figure 4 Another exemplary speech enhancement architecture 400 of a speech communication system pipeline according to an embodiment of the present disclosure is depicted. Specifically, Figure 4 A speech communication system pipeline with multiple speech enhancement components is depicted. Figure 4 is shown as similar to Figure 1 The embodiment shown, however Figure 4 Can be used with Figure 2 and 3 The embodiment shown in the figure is implemented in a similar manner. Figure 4As shown, NISQA 420 can receive voice-enhanced audio data and information from the user's device (i.e., a device 440 including a microphone 402, a speaker 434, and other various components of the device 440). The information may include device information of the device (i.e., microphone 402) that captures the audio data. NISQA 420 can detect the quality of the voice of the audio data based on the received device information. For example, depending on the microphone type, the quality of the audio data may change, and NISQA may indicate an optimized voice enhancement component 436 to change one or more of the voice enhancement components based on the detected quality of the voice and the device information. Additionally and / or alternatively, when a change in device information (e.g., a change in microphone 402) is detected, depending on the new microphone type, the quality of the audio data may change, and NISQA may indicate an optimized voice enhancement component 436 to change one or more of the voice enhancement components based on the detected quality of the voice and the changed device information. In addition, instead of microphone or speaker information, NISQA 420 can receive environmental information of the device 440 that is capturing audio data. NISQA 420 may detect the quality of the speech of the audio data based on the received environmental information. NISQA may instruct the optimized speech enhancement component 436 to change one or more of the speech enhancement components based on the detected speech quality and the environmental information and / or when the environmental information changes. In addition, NISQA 420 may receive the load of at least one processor of the device 440 that is capturing the audio data. NISQA 420 may detect the quality of the speech of the audio data, which may also be based on the load of at least one processor of the device 440. NISQA may instruct the optimized speech enhancement component 436 to change one or more of the speech enhancement components based on the detected quality of the speech and the load of at least one processor of the device 440. For example, if the load is high, performance may be reduced, or if the load is low, more processor-intensive speech enhancement components may be used.

[0056] Based on the results of NISQA420, the optimized speech enhancement component 436 can dynamically and / or in real time change various speech enhancement components, such as the music detection component 104, the echo cancellation component 106, the noise suppression component 108, the de-reverberation component 110, the echo detector 112, the automatic gain control component 114, the jitter buffer management component 124 and / or the packet loss concealment component 128.

[0057] Additionally and / or alternatively, the one or more speech enhancement components that improve speech can be reported back to the server over the network along with the brand and / or model of the device with the improved speech enhancement. The server can then aggregate such reports from multiple devices of multiple users, and the one or more speech enhancement components can be used in a system with the same brand and / or model of the reporting device.

[0058] Figure 5 An exemplary cloud-based speech enhancement architecture 500 for a speech communication system pipeline according to an embodiment of the present disclosure is depicted. Specifically, Figure 5 A voice communication system pipeline with multiple voice enhancement components residing in a server / cloud device 580 is depicted. Figure 5 Shown as similar to Figure 1-4 The embodiment shown in FIG. Figure 1-4 The illustrated embodiment is implemented in a similar manner. Figure 5 Also similar to Figure 3 1 and 1 . The embodiment shown in FIG. 1 shows an embodiment in which NISQA 520 and optimized speech enhancement component 536 reside in the cloud on server / cloud device 580 via network 122 and / or. NISQA 520 can receive encoded speech-enhanced audio data and detect the quality of the encoded speech-enhanced audio data. NISQA 520 can be one or more of the above-described NISQAs using a neural network, which can be trained to detect the quality of speech-enhanced audio data.

[0059] The exemplary cloud-based voice enhancement architecture 500 can support many types of endpoints (devices 540). Some types of devices 540 may not have high-quality audio. For example, the device 540 can be a web-based client that can use web real-time communication (Web RTC). WebRTC can provide real-time communication (RTC) to web browsers and / or mobile applications via an application programming interface (API). WebRTC can allow audio and video communications to work inside a web page by allowing direct peer-to-peer communication without installing plug-ins or downloading local applications.

[0060] Web-based client devices 540 (such as web browsers and / or mobile applications using WebRTC) may have increased poor call quality (>10%) compared to other types of non-web-based client devices 540. NISQA 520 can detect poor quality calls, including impairments of one or more of noise, equipment, echo, reverberation, voice level, etc. When a poor quality transmit signal is detected from an endpoint (device 540) using NISQA 520, an appropriate cloud-based voice enhancement model can be applied to mitigate the impairment, as discussed in more detail below.

[0061] like Figure 5 As shown, the microphone 502 of the device 540 can capture audio data. The audio data can then be received by the encoder 582. The encoder 582 can obtain the audio data captured by the microphone 502 for use in the web-based device 540, and can transmit the audio data to the server / cloud device 580. In addition and / or alternatively, the encoder 582 can be an audio codec, such as an AI-powered audio codec, such as a SATIN encoder, which is a digital signal processor with machine learning. The encoder 582 can encode (i.e., compress) audio data for transmission over the network 522. When encoding, the encoder 582 can transmit the encoded audio data to the server / cloud device 580 via the network 522, providing the voice enhancement component of the voice communication system in the network 522.

[0062] Figure 5 Depicted is the audio data received by the music detection component 504, which can detect whether music is detected in the captured audio data. For example, if the music detection component 504 detects the audio data, the music detection component 504 can notify the user that music has been detected. The audio data captured by the microphone 502 can also be received and processed by one or more other speech enhancement components (e.g., echo cancellation component 506, noise suppression component 508 and / or de-reverberation component 510). One or more of the echo cancellation component 506, noise suppression component 508 and / or de-reverberation component 510 can be a speech enhancement component that provides microphone and speaker alignment, such as microphone 502 and speaker 534. The echo cancellation component 506 (also referred to as an acoustic echo cancellation component) can receive the audio data captured by the microphone 502 and the speaker data played by the speaker 534. The echo cancellation component 506 can be used to eliminate the acoustic feedback between the speaker 534 and the microphone 502 in the voice communication system.

[0063] The noise suppression component 508 can receive the audio data captured by the microphone 502 and the speaker data played by the speaker 534. The noise suppression component 508 can process the audio data and the speaker data to isolate the voice from other sounds and music during playback. For example, when the microphone 502 is turned on, the background noise around the user (e.g., turning pages, slamming doors, barking dogs, etc.) may distract other users. The noise suppression component 508 can remove such noise around the user in the voice communication system.

[0064] The dereverberation component 510 may receive audio data captured by the microphone 502 and speaker data played by the speaker 534. The dereverberation component 510 may process the audio data and the speaker data to remove the effects of reverberation, such as reverberant sounds captured by microphones including the microphone 502.

[0065] The audio data may be speech-enhanced audio data after being processed by one or more speech enhancement components (e.g., one or more of the echo cancellation component 506, the noise suppression component 508, and / or the de-reverberation component 510), and further processed by one or more other speech enhancement components. For example, the speech-enhanced audio data may be received and / or processed by one or more of the echo detector 512 and / or the automatic gain control component 514. The echo detector 512 may use the speech-enhanced audio data to detect whether there is an echo in the speech-enhanced audio data and notify the user of the echo. The automatic gain control component 514 may use the speech-enhanced audio data to amplify and / or increase the volume of the speech-enhanced audio data based on whether the voice activity detector 516 detects speech.

[0066] The voice activity detector 516 can receive the voice-enhanced audio data processed by the automatic gain control component 514 and can detect whether voice activity is detected in the voice-enhanced audio data. Based on the detection by the voice activity detector 516, the user can be notified that he or she is talking while silent, the notification can be automatically turned on or off, and / or the automatic gain control component 514 can be instructed to amplify and / or increase the volume of the voice-enhanced audio data.

[0067] The jitter buffer management component 524 can receive audio data transmitted over the network 522 and process the audio data. For example, the jitter buffer management component 524 can buffer packets of audio data to allow the decoder 526 to receive the audio data at evenly spaced intervals. Because the audio data is transmitted over the network 522, there may be variations in packet arrival times (i.e., jitter) that may occur due to network congestion, timing drift, and / or routing changes. The jitter buffer management component 524 can delay arriving packets so that the user experiences a clear connection with minimal sound distortion.

[0068] The audio data from the jitter buffer management component 524 may then be received by a decoder 526. The decoder 526 may be an audio codec, such as an AI-powered audio codec, for example a SATIN decoder, which is a digital signal processor with machine learning. The decoder 526 may decode (i.e., decompress) the audio data received over the network 522. After decoding, the decoder 526 may provide the decoded audio data to a packet loss concealment component 528.

[0069] The packet loss concealment component 528 may receive the decoded audio data and may process the decoded audio data to conceal gaps in the audio stream caused by data transmission failures in the network 522. The processing results may be provided to one or more of the network quality classifier 530, the call quality estimator component 532, and / or the device 540. The network quality classifier 530 may classify the quality of the connection to the network 522 based on information received from the jitter buffer management component 524 and / or the packet loss concealment component 528, and the network quality classifier 530 may inform the user of the quality of the connection to the network 522, such as poor, medium, excellent, etc. When the connection to the network 522 is through the public switched telephone network (PSTN), the call quality estimator component 532 may estimate the quality of the call.

[0070] After processing the audio data, the server / cloud device 580 can transmit the voice-enhanced audio data back to the device 540 via the network 522. The decoder 584 can receive the processed audio data in the web-based device 540, and provide the processed audio data to the speaker 534 for playback. Additionally and / or alternatively, the decoder 584 can be an audio codec, such as an AI-powered audio codec, such as a SATIN decoder, which is a digital signal processor with machine learning. The decoder 526 can decode (i.e., decompress) the audio data received through the network 522. When decoding, the decoder 526 can provide the decoded audio data to the speaker 534 for playback. The speaker 534 can play the decoded audio data as speaker data. The speaker data can also be provided to one or more of the echo cancellation component 506, the noise suppression component 508 and / or the de-reverberation component 510.

[0071] like Figure 5 As shown, NISQA 520 can receive audio data that is modified to produce speech-enhanced audio data. NISQA 520 can also receive information from the user's device (i.e., device 540 including microphone 502, speaker 534, and other various components of device 540). The information can include device information of the device (i.e., microphone 502) that captured the audio data.

[0072] NISQA 520 may determine whether device 540 is a low-quality endpoint. When NISQA 520 determines that a particular device 540 is a low-quality endpoint, NISQA 520 may instruct the particular device 540 to turn off audio processing on the particular device 540, and NISQA 520 may instruct server / cloud device 580 to implement and / or change one or more of the speech enhancement components. For example, when NISQA 520 detects that the particular device 540 is a web-based client and / or uses WebRTC, NISQA 520 may detect that the particular device 540 is a low-quality endpoint.

[0073] For example, if the particular device 540 is a low-quality endpoint, such as a web browser using WebRTC, the user of the voice communication system may be rated low. Additionally, if the particular device 540 is a web browser using WebRTC, the NISQA 520 may not be able to indicate how the web browser uses a speech enhancement component to process audio data. Therefore, by moving speech enhancement to the server / cloud device 580, the NISQA 520 can bypass audio processing in low-quality endpoints, such as web browsers using WebRTC.

[0074] Additionally and / or alternatively, NISQA 520 may also receive information about a specific device 540 (i.e., microphone 502, speaker 534, and other various components of the specific device 540) and determine whether the specific device 540 is a low-quality endpoint. When NISQA 520 determines that the specific device 540 is a low-quality endpoint, NISQA 520 may instruct the specific device 540 to turn off audio processing on the specific device 540, and NISQA 520 may instruct the server / cloud device 580 to implement and / or change one or more of the speech enhancement components. In one example, NISQA 520 may score and / or determine the capabilities of the device 540 based on device information, connection type (i.e., web-based and / or WebRTC connection), and / or one or more from a low-quality endpoint (LQE) database 590. The LQE database 590 may include a list of devices (i.e., devices 540) that have been pre-determined to have low quality. In addition, NISQA 520 may score device 540 and may store the score in LQE database 590. For example, NISQA 520 may generate a score on a predetermined scale, such as 1 to 5 for quality, echo impairments, background noise, bandwidth distortion, etc. NISQA 520 may then use the updated LQE database 590 to determine device capabilities, as well as additional indicators of low-quality endpoints (devices) for future voice communication sessions. When the score is below a predetermined threshold, then device 540 may be determined to be a low-quality endpoint.

[0075] NISQA 520 may detect the quality of speech of the audio data based on the received audio data, and NISQA 520 may instruct the optimized speech enhancement component 536 to change one or more of the speech enhancement components residing in the server / cloud device 580 based on the detected speech quality and device information.

[0076] Based on the results of NISQA520, the optimized speech enhancement component 536 can dynamically and / or in real time change various speech enhancement components residing in the server / cloud device 580, such as the music detection component 504, the echo cancellation component 506, the noise suppression component 508, the de-reverberation component 510, the echo detector 512, the automatic gain control component 514, the jitter buffer management component 524 and / or the packet loss concealment component 528.

[0077] In one example, when noisy speech is detected, a cloud-based noise suppressor (noise suppression component 508) may be applied by an optimized speech enhancement component 536. If an echo is detected, a cloud-based echo canceller (echo cancellation component 506) may be applied by an optimized speech enhancement component 536. NISQA 520 may be used to selectively apply these speech enhancement components on devices 540 that do not have high-quality audio (e.g., devices 540 that are web-based clients), which may minimize costs on the server / cloud device 580 that may otherwise need to perform these speech enhancement components on all calls and maximize quality.

[0078] Figure 6 A method 600 for optimizing a speech enhancement component to be used in a speech communication system using non-intrusive speech quality assessment according to an embodiment of the present disclosure is depicted. The method 600 may begin at 602, where audio data including speech may be received. The audio data is processed by at least one speech enhancement component. As mentioned above, the at least one speech enhancement component may include one or more of acoustic echo cancellation, noise suppression, dereverberation, automatic gain control, packet loss concealment, etc.

[0079] In addition to receiving the audio data, one or more of device information of a device capturing the audio data, environment information of the device capturing the audio data, and a load of at least one processor of the device capturing the audio data may be received at 604 .

[0080] In addition, before, after, and / or during receiving the audio data, device information, environmental information, and / or the load of at least one processor at 606, a non-intrusive speech quality assessment (NISQA) model trained using a neural network model, also referred to as NISQA, may be received. Upon receiving the audio data and / or the NISQA model at 608, the trained NISQA model may detect a first quality of the speech of the audio data. As mentioned in more detail above, the trained NISQA model may have been trained to automatically detect the quality of the speech by using a robust data set. In addition to receiving the audio data, the NISQA model may detect the quality of the speech using one or more of the device information, environmental information, and / or the load of at least one processor.

[0081] In certain embodiments of the present disclosure, a first quality of speech of audio data detected by a NISQA model may be transmitted to at least one server over a network at 610. At least one server may determine one or more speech enhancement components to be changed by a device at 612. Next, at least one server may transmit one or more of the at least one speech enhancement components to be changed to a device capturing audio data at 614. At 616, one or more of the at least one speech enhancement components changed based on the transmitted detected first quality of speech may be received by the device capturing audio data.

[0082] Based on the first quality of the detected speech at 618, one or more of at least one speech enhancement component can be changed based on the first quality of the detected speech. The changed one or more speech enhancement components can include one or more of acoustic echo cancellation, noise suppression, de-reverberation, automatic gain control, and packet loss concealment. Additionally and / or alternatively, a change in device information can be detected, and when a change in device information is detected, one or more of at least one speech enhancement component based on the detected speech quality can be changed.

[0083] After changing one or more of at least one speech enhancement component at 620, the trained NISQA model can be used to detect the second quality of the speech of the audio data. Then, at 622, one or more of at least one speech enhancement component can be changed based on the second quality of the detected speech. The speech enhancement component based on the change of the second quality of the detected speech and the speech enhancement component based on the change of the first quality of the speech affect the same speech enhancement component, such as the same acoustic echo cancellation, noise suppression, de-reverberation, automatic gain control and packet loss concealment. Next, determine whether the second quality of the detected speech is higher than the first quality of the detected speech. When the second quality of the detected speech is higher than the first quality of the detected speech, the one or more of the changes in at least one speech enhancement component based on the second quality of the detected speech can be maintained. On the contrary, when the second quality of the detected speech is not higher than the first quality of the detected speech, one or more of the at least one speech enhancement component based on the first quality of the detected speech can be changed from the one or more speech enhancement components based on the change of at least one speech enhancement component based on the second quality of the detected speech to the previous at least one speech enhancement component or another speech enhancement component.

[0084] Figure 7 A method 700 for optimizing a speech enhancement component to be used in a speech communication system using non-intrusive speech quality assessment according to an embodiment of the present disclosure is depicted. The method 700 may begin at 702, where audio data including speech may be received over a network from a computing device at a server / cloud device implementing a speech communication system. The audio data may or may not be processed by at least one speech enhancement component. As mentioned above, the at least one speech enhancement component may include one or more of acoustic echo cancellation, noise suppression, dereverberation, automatic gain control, packet loss concealment, etc. In addition to receiving the audio data at 704, device information of the computing device that captured the audio data may be received.

[0085] Upon receiving the audio data at 706, a non-intrusive speech quality assessment (NISQA) model (also referred to as NISQA) trained using a neural network model can detect a first quality of speech of the audio data. As mentioned in more detail above, the trained NISQA model can be trained to automatically detect the quality of the speech by using a robust data set. In addition to the received audio data, the NISQA model can use one or more of device information, environmental information, and / or a load of at least one processor to detect the quality of the speech.

[0086] At 708, a NISQA such as NISQA 520 and / or a server / cloud device such as server cloud device 580 may determine whether the computing device transmitting the audio data is a low-quality endpoint based on the first quality of the voice of the audio data. For example, determining whether the computing device is a low-quality endpoint may include detecting whether the computing device is a web-based computing device, such as a web browser using WebRTC. Alternatively and / or additionally, the NISQA and / or the server / cloud device may determine whether the computing device transmitting the audio data is a low-quality endpoint based on the first quality of the voice of the audio data being below a predetermined threshold and the received device information.

[0087] At 710, the NISQA and / or server / cloud device may determine a score for the computing device based on one or both of the first quality of the voice of the audio data and the received device information. For example, the NISQA and / or server / cloud device may generate a score of a predetermined scale, such as 1 to 5 for quality, echo impairments, background noise, bandwidth distortion, etc. When the score is below a predetermined threshold, the computing device may be determined to be a low-quality endpoint. In addition, at 712, when the score is below a predetermined threshold, the NISQA and / or server / cloud device may store the determined score of the computing device in a low-quality endpoint database (such as the LQE database 590). Then, at 714, the NISQA and / or server / cloud device may use the score stored in the low-quality endpoint database to determine whether another computing device is a low-quality endpoint based on the device information of the other computing device. For example, the low-quality endpoint database may be used to determine computing device capabilities, as well as additional indicators of low-quality endpoints (devices) for future voice communication sessions.

[0088] At 716, when it is determined that the computing device is a low-quality endpoint, at least one speech enhancement component may be transmitted from the computing device to at least one server device (such as server / cloud device 580) over a network. The at least one speech enhancement component to be transmitted from the device to the server / cloud device over a network may be determined based on the score provided by NISQA and / or information stored in the LQE database. Alternatively, when it is determined that the computing device is a low-quality endpoint, all audio processing may be transmitted to at least one server device. Then, at 718, when it is determined that the computing device is a low-quality endpoint, instructions for shutting down at least one speech enhancement component and / or all audio processing may be sent to the computing device over a network.

[0089] At 720, after transmitting at least one speech enhancement component and / or all audio processing to at least one server device, one or more of the at least one speech enhancement component can be changed based on the first quality of the detected speech. At 722, after changing one or more of the at least one speech enhancement component, the trained NIQA model can be used to detect a second quality of the speech of the audio data. Then, at 724, one or more of the at least one speech enhancement component can be changed based on the second quality of the detected speech. At 726, the audio data processed by the changed at least one speech enhancement component can be transmitted to the computing device via the network.

[0090] As described above, all speech enhancement components may reside on the device side, all speech enhancement components may reside on the server / cloud device side, or some speech enhancement components may reside on the device side and some speech enhancement components may reside on the server / cloud device side. For example, if the server / cloud device side receives narrowband audio, which can be detected by NISQA from the received audio data, a bandwidth extender can be added to make it full-band audio. Alternatively, for example, a device may have narrowband playback capabilities, which can be detected by NISQA based on device information (such as microphone data), and a speech enhancement component that optimizes speech for narrowband playback can be added.

[0091] Detecting the use of NISQA can be done by checking the user device for changes in the voice enhancement components. Additionally, looking at network packets to see if something other than audio data is being downloaded, or to determine if the quality of the voice telecommunication system suddenly improves without the user taking active steps. Additionally, if NISQA is stored on the client side, processor usage may be higher than running the voice telecommunication system alone.

[0092] Figure 8 A high-level diagram of an exemplary computing device 800 that can be used in accordance with the systems, methods, modules, and computer-readable media disclosed herein, in accordance with an embodiment of the present disclosure is depicted. For example, in accordance with an embodiment of the present disclosure, the computing device 800 can be used in a system that processes data (such as audio data) using a neural network. The computing device 800 may include at least one processor 802 that executes instructions stored in a memory 804. The instructions may be, for example, instructions for implementing the functions described as being performed by one or more of the components described above or instructions for implementing one or more of the methods described above. The processor 802 may access the memory 804 by way of a system bus 806. In addition to storing executable instructions, the memory 804 may also store data, audio, one or more neural networks, and the like.

[0093] The computing device 800 may additionally include a data store (also referred to as a database) 808, which is accessible by the processor 802 by way of the system bus 806. The data store 808 may include executable instructions, data, examples, features, etc. The computing device 800 may also include an input interface 810 that allows external devices to communicate with the computing device 800. For example, the input interface 810 may be used to receive instructions from an external computer device, from a user, etc. The computing device 800 may also include an output interface 812 that interfaces the computing device 800 with one or more external devices. For example, the computing device 800 may display text, images, etc. by way of the output interface 812.

[0094] It is contemplated that external devices that communicate with the computing device 800 via the input interface 810 and the output interface 812 may be included in an environment that provides substantially any type of user interface with which a user may interact. Examples of user interface types include graphical user interfaces, natural user interfaces, and the like. For example, a graphical user interface may accept input from a user using an input device such as a keyboard, a mouse, a remote control, and may provide output on an output device such as a display. In addition, a natural user interface may enable a user to interact with the computing device 800 in a manner that is free from the constraints imposed by input devices such as a keyboard, a mouse, a remote control, and the like. In contrast, a natural user interface may rely on speech recognition, touch and stylus recognition, gesture recognition both on and near the screen, air gestures, head and eye tracking, sound and speech, vision, touch, gestures, machine intelligence, and the like.

[0095] In addition, although shown as a single system, it should be understood that the computing device 800 can be a distributed system. Thus, for example, several devices can communicate via a network connection and can jointly perform the tasks described as being performed by the computing device 800.

[0096] Go to Fig. 9 , Fig. 9 A high-level diagram of an exemplary computing system 900 that can be used in accordance with the systems, methods, modules, and computer-readable media disclosed herein, according to an embodiment of the present disclosure is depicted. For example, computing system 900 can be or can include computing device 800. Additionally and / or alternatively, computing device 800 can be or can include computing system 900.

[0097] The computing system 900 may include multiple server computing devices, such as server computing device 902 and server computing device 904 (collectively referred to as server computing devices 902-904). Server computing device 902 may include at least one processor and a memory; at least one processor executes instructions stored in the memory. The instructions may be, for example, instructions for implementing the functions described as being performed by one or more of the above components or instructions for implementing one or more of the above methods. Similar to server computing device 902, at least a subset of server computing devices 902-904 other than server computing device 902 may each include at least one processor and a memory, respectively. In addition, at least a subset of server computing devices 902-904 may include corresponding data storage.

[0098] The processor of one or more of the server computing devices 902-904 may be or may include a processor, such as processor 802. Additionally, the memory (or memories) of one or more of the server computing devices 902-904 may be or may include a memory, such as memory 804. Additionally, the data store (or data stores) of one or more of the server computing devices 902-904 may be or may include a data store, such as data store 808.

[0099] The computing system 900 may also include various network nodes 906 that carry data between the server computing devices 902-904. In addition, the network nodes 906 may carry data from the server computing devices 902-904 to external nodes (e.g., external to the computing system 900) by way of a network 908. The network nodes 902 may also carry data from the external nodes to the server computing devices 902-904 by way of the network 908. The network 908 may be, for example, the Internet, a cellular network, etc. The network nodes 906 may include switches, routers, load balancers, etc.

[0100] The fabric controller 910 of the computing system 900 may manage the hardware resources of the server computing devices 902-904 (e.g., processors, memory, data storage, etc. of the server computing devices 902-904). The fabric controller 910 may also manage the network nodes 906. In addition, the fabric controller 910 may manage the creation, provisioning, de-provisioning, and supervision of managed runtime environments instantiated on the server computing devices 902-904.

[0101] As used herein, the terms "component" and "system" are intended to encompass computer-readable data storage configured with computer-executable instructions that, when executed by a processor, cause certain functions to be performed. Computer-executable instructions may include routines, functions, etc. It should also be understood that a component or system may be located on a single device or distributed across several devices.

[0102] The various functions described herein can be implemented with hardware, software, or any combination thereof. If implemented with software, these functions can be stored on a computer-readable medium and / or transmitted on a computer-readable medium as one or more instructions or codes. Computer-readable media may include computer-readable storage media. Computer-readable storage media may be any available storage medium accessible by a computer. As an example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that can be used to store the desired program code in the form of an instruction or data structure and can be accessed by a computer. As used herein, disks and optical disks may include compact disks ("CDs"), laser optical disks, optical disks, digital versatile disks ("DVDs"), floppy disks, and blue-ray disks ("BDs"), wherein disks generally reproduce data magnetically, and optical disks generally reproduce data optically with lasers. In addition, propagation signals are not included in the scope of computer-readable storage media. Computer-readable media may also include communication media, which includes any media that facilitates the transfer of a computer program from one place to another. For example, a connection may be a communication medium. For example, if the software is transmitted from a website, server or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line ("DSL") or wireless technologies such as infrared, radio and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave are included in the definition of communication media. Combinations of the above may also be included within the scope of computer-readable media.

[0103] Alternatively and / or additionally, the functions described herein may be performed at least in part by one or more hardware logic components. For example, but not limited to, illustrative types of hardware logic components that may be used include field programmable gate arrays ("FPGAs"), application specific integrated circuits ("ASICs"), application specific standard products ("ASSPs"), systems on chip ("SOCs"), complex programmable logic devices ("CPLDs"), etc.

[0104] The above description includes examples of one or more embodiments. Of course, for the purpose of describing the above aspects, it is not possible to describe every conceivable modification and variation of the above devices or methods, but one of ordinary skill in the art will recognize that many further modifications and permutations of various aspects are possible. Therefore, the described aspects are intended to cover all such changes, modifications and variations that fall within the scope of the appended claims.

Claims

1. A computer-implemented method for optimizing a speech enhancement component to be used in a speech communication system using non-intrusive speech quality assessment, the method comprising: receiving audio data from a computing device over a network, the audio data comprising speech; detecting a first quality of the speech of the audio data using a trained non-intrusive speech quality assessment (NISQA) model, the trained NISQA model being trained to automatically detect the quality of speech; determining whether the computing device is a low-quality endpoint based on a first quality of the speech of the audio data; as well as When the computing device is determined to be a low-quality endpoint, at least one speech enhancement component is transmitted from the computing device to at least one server device over the network.

2. The method according to claim 1, further comprising: When the computing device is determined to be a low-quality endpoint, an instruction for shutting down the at least one speech enhancement component is sent to the computing device over the network.

3. The method according to claim 1, further comprising: When the computing device is determined to be a low-quality endpoint, sending an instruction to shut down audio processing to the computing device via the network, Wherein, when the computing device is determined to be a low-quality endpoint, transmitting the at least one speech enhancement component to the at least one server device comprises: When the computing device is determined to be a low-quality endpoint, audio processing is transferred from the computing device to the at least one server device over the network.

4. The method according to claim 1, further comprising: after transmitting the at least one speech enhancement component to at least one server device, changing one or more of the at least one speech enhancement component based on the detected first quality of speech; as well as The audio data processed by the altered at least one speech enhancement component is transmitted to the computing device.

5. The method according to claim 1, further comprising: Determining whether the computing device is a low-quality endpoint based on a first quality of the speech of the audio data includes: It is detected whether the computing device is a web-based computing device.

6. The method according to claim 1, further comprising: receiving device information of the computing device that captured the audio data, Wherein, determining whether the computing device is a low-quality endpoint is also based on the received device information.

7. The method according to claim 6, further comprising: determining a score for the computing device based on one or both of a first quality of the speech of the audio data and the received device information; determining whether the determined score of the computing device is below a predetermined threshold; as well as When the score is below the predetermined threshold, the determined score of the computing device is stored in a low-quality endpoint database.

8. The method according to claim 7, further comprising: A determination is made whether the other computing device is a low-quality endpoint based on device information of the other computing device and a score stored in the low-quality endpoint database.

9. The method according to claim 1, further comprising: changing one or more of the at least one speech enhancement component based on the detected first quality of speech; detecting a second quality of the speech of the audio data using the trained NIQA model after changing the one or more of the at least one speech enhancement components; determining whether the second quality of the detected speech is higher than the first quality of the detected speech; as well as When the second quality of the detected speech is not higher than the first quality of the detected speech, changing the changed at least one speech enhancement component; as well as When the second quality of the detected speech is higher than the first quality of the detected speech, the changed one or more speech enhancement components of the at least one speech enhancement component are maintained.

10. The method according to claim 1, wherein: The at least one speech enhancement component includes one or more of: acoustic echo cancellation, noise suppression, dereverberation, automatic gain control, and packet loss concealment.

11. A system for optimizing a speech enhancement component to be used in a speech communication system using non-intrusive speech quality assessment, the system comprising: a data storage device storing instructions for optimizing a speech enhancement component to be used in a speech communication system using non-intrusive speech quality assessment; as well as A processor configured to execute the instructions to perform a method, the method comprising: receiving audio data from a computing device over a network, the audio data comprising speech; detecting a first quality of the speech of the audio data using a trained non-intrusive speech quality assessment (NISQA) model, the trained NISQA model being trained to automatically detect the quality of speech; determining whether the computing device is a low-quality endpoint based on a first quality of the speech of the audio data; and When the computing device is determined to be a low-quality endpoint, at least one speech enhancement component is transmitted from the computing device to at least one server device over the network.

12. The system according to claim 11, wherein: The processor is further configured to execute the instructions to perform the method, which includes: When the computing device is determined to be a low-quality endpoint, an instruction for shutting down the at least one speech enhancement component is sent to the computing device over the network.

13. The system of claim 11, further comprising: When the computing device is determined to be a low-quality endpoint, sending an instruction to shut down audio processing to the computing device via the network, Wherein, when the computing device is determined to be a low-quality endpoint, transmitting the at least one speech enhancement component to the at least one server device comprises: When the computing device is determined to be a low-quality endpoint, audio processing is transferred from the computing device to the at least one server device over the network.

14. A computer-readable storage device storing instructions which, when implemented by a computer, cause the computer to perform a method for optimizing a speech enhancement component to be used in a speech communication system using non-intrusive speech quality assessment, the method comprising: receiving audio data from a computing device over a network, the audio data comprising speech; detecting a first quality of the speech of the audio data using a trained non-intrusive speech quality assessment (NISQA) model, the trained NISQA model being trained to automatically detect the quality of speech; determining whether the computing device is a low-quality endpoint based on a first quality of the speech of the audio data; as well as When the computing device is determined to be a low-quality endpoint, at least one speech enhancement component is transmitted from the computing device to at least one server device over the network.

15. The computer readable storage device of claim 14, wherein: When the instructions are executed by the computer, the computer executes the method, which further includes: When the computing device is determined to be a low-quality endpoint, an instruction for shutting down the at least one speech enhancement component is sent to the computing device over the network.

Images