Dynamic speech enhancement component optimization

Abstract

Systems, methods, and computer-readable storage devices are disclosed for optimizing speech enhancement components for use in a speech communication system using non-invasive speech quality evaluation. A method comprising: receiving audio data, the audio data comprising speech and having been processed by at least one speech enhancement component; detecting a first quality of the speech of the audio data using a trained non-invasive speech quality assessment (NISQA) model, the trained NISQA model being trained to automatically detect a quality of the speech; and altering one or more of the at least one speech enhancement component based on the detected first quality of the speech.

Description

Technical Field

[0001] The present disclosure relates to speech enhancement by reducing echo, noise, dereverberation, etc. In particular, the present disclosure relates to speech enhancement using a neural network that determines speech enhancement components used in a speech communication system by using a non-intrusive speech quality assessment model. Background Art

[0002] In a voice communication system, the audio signal may be affected by echo, background noise, reverberation, enhancement algorithms, network impairments, etc. Providers of voice communication systems who are trying to provide optimal and reliable services to their customers can estimate the perceived quality of the audio signal. For example, voice quality prediction may be useful during network design and development and for monitoring and improving customer quality of experience (QoE).

[0003] To determine QoE, one approach may include subjective listening tests to provide an accurate method for evaluating the perceived quality of speech signals. In this approach, the estimated quality is the average judgment of the users. For example, the average of the scores of all participants on a particular condition is called the mean opinion score (MOS) and represents the perceived speech quality after leveling various factors. However, this approach may be cumbersome, time consuming, and cannot be done in real time.

[0004] Intrusive methods for determining speech quality can calculate the perceptually weighted distance between a clean reference and the contaminated signal to estimate the perceived speech quality. Intrusive methods are considered more accurate because they provide a higher correlation with subjective assessments. Since these measurements are intrusive, they cannot be made in real time and require a reference to a clean speech signal to estimate the MOS.

[0005] In order to overcome the limitations of subjective listening and intrusive estimation of speech quality, non-intrusive speech quality assessment (NISQA) models using neural networks have been implemented. Such NISQA models can be used to dynamically optimize speech enhancement components in telecommunication pipelines to improve QoE. Speech enhancement (SE) components are essential for telecommunications for reducing echo, noise, de-reverberation, etc. Many of these components can be based on acoustic digital signal processing (ADSP) algorithms, but these components can be replaced by deep learning components. However, deep neural network (DNN) models are only as good as the data used to train them, and it is impossible to have fully representative training data. Therefore, some new SE components may be more harmful than better than their previous SE components. Therefore, it is necessary to dynamically select speech enhancement components in real time to optimize the quality of user experience. Summary of the invention

[0006] According to certain embodiments, systems, methods, and computer-readable media are disclosed for optimizing speech enhancement components in a speech communication system using non-intrusive speech quality assessment.

[0007] According to certain embodiments, a computer-implemented method for optimizing speech enhancement components in a speech communication system using non-intrusive speech quality assessment is disclosed. A method includes: receiving audio data, the audio data including speech, and the audio data has been processed by at least one speech enhancement component; using a trained non-intrusive speech quality assessment (NISQA) model to detect a first quality of the speech of the audio data, the trained NISQA model being trained to automatically detect the quality of speech; and, based on the detected first quality of the speech, changing one or more of the at least one speech enhancement components.

[0008] According to certain embodiments, a system for optimizing speech enhancement components in a speech communication system using non-intrusive speech quality assessment is disclosed. A system includes: a data storage device storing instructions for optimizing speech enhancement components in a speech communication system using non-intrusive speech quality assessment; and a processor configured to execute the instructions to perform a method including the following operations: receiving audio data, the audio data including speech, and the audio data having been processed by at least one speech enhancement component; using a trained non-intrusive speech quality assessment (NISQA) model to detect a first quality of the speech of the audio data, the trained NISQA model being trained to automatically detect the quality of speech; and, based on the detected first quality of the speech, changing one or more of the at least one speech enhancement components.

[0009] According to certain embodiments, a computer-readable storage device storing instructions is disclosed that, when executed by a computer, causes the computer to perform a method for optimizing speech enhancement components in a speech communication system using non-intrusive speech quality assessment. A method of a computer-readable storage device includes: receiving audio data, the audio data including speech, and the audio data has been processed by at least one speech enhancement component; using a trained non-intrusive speech quality assessment (NISQA) model to detect a first quality of the speech of the audio data, the trained NISQA model being trained to automatically detect the quality of speech; and, based on the detected first quality of the speech, changing one or more of the at least one speech enhancement components.

[0010] Additional objects and advantages of the disclosed embodiments will be set forth in part in the following description and will be obvious from the description or may be learned by practicing the disclosed embodiments. The objects and advantages of the disclosed embodiments will be realized and obtained by the elements and combinations particularly pointed out in the appended claims.

[0011] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosed embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In the course of the following detailed description, reference will be made to the accompanying drawings. The accompanying drawings show different aspects of the present disclosure, and where appropriate, reference numerals of similar structures, components, materials and / or elements in different drawings are similarly labeled. It should be understood that various combinations of structures, components and / or elements other than those specifically shown are also contemplated and within the scope of the present disclosure.

[0013] In addition, many embodiments of the present disclosure are described and illustrated herein. The present disclosure is not limited to any single aspect or its embodiment, nor is it limited to any combination and / or permutation of such aspects and / or embodiments. In addition, each aspect and / or its embodiment of the present disclosure may be used alone or in combination with one or more of the other aspects and / or its embodiments of the present disclosure. For the sake of brevity, certain permutations and combinations are not discussed and / or illustrated separately herein.

[0014] Figure 1 An exemplary speech enhancement architecture for a speech communication system pipeline according to an embodiment of the present disclosure is depicted.

[0015] Figure 2 Another exemplary speech enhancement architecture for a speech communication system pipeline according to an embodiment of the present disclosure is depicted.

[0016] Figure 3 Yet another exemplary speech enhancement architecture for a speech communication system pipeline according to an embodiment of the present disclosure is depicted.

[0017] Figure 4 Yet another exemplary speech enhancement architecture for a speech communication system pipeline according to an embodiment of the present disclosure is depicted.

[0018] Figure 5 A method for optimizing a speech enhancement component for use in a speech communication system using non-intrusive speech quality assessment is depicted in accordance with an embodiment of the present disclosure.

[0019] Figure 6 A high-level diagram of an exemplary computing device that may be used in accordance with the systems, methods, and computer-readable media disclosed herein is depicted in accordance with an embodiment of the present disclosure.

[0020] Figure 7 A high-level diagram of an exemplary computing system that can be used in accordance with the systems, methods, and computer-readable media disclosed herein is depicted in accordance with an embodiment of the present disclosure.

[0021] Likewise, many embodiments are described and shown herein. The present disclosure is not limited to any single aspect or its embodiment, nor is it limited to any combination and / or permutation of such aspects and / or embodiments. Aspects and / or embodiments of the present disclosure may be used alone or in combination with one or more of other aspects and / or embodiments of the present disclosure. For the sake of brevity, many of these combinations and permutations are not discussed separately herein. DETAILED DESCRIPTION

[0022] Those skilled in the art will recognize that various embodiments and examples of the present disclosure can be practiced based on this description. All of these embodiments and examples are intended to be included within the scope of the present disclosure.

[0023] As used herein, the terms "comprises," "having," "including," or any other variation thereof, are intended to encompass a non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements, but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. The term "exemplary" is used in the sense of "example," not "ideal." Additionally, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified, as is clear from the context, the phrase "X employs A or B" is intended to mean any natural inclusive permutation. For example, the phrase "X employs A or B" is satisfied by any of the following examples: X employs A; X employs B; or X employs both A and B. Additionally, the articles "a" and "an," as used in this application and the appended claims, should generally be interpreted as meaning "one or more," unless otherwise specified or clearly specified as a singular form from the context.

[0024] For the sake of brevity, conventional techniques related to systems and servers for methods and other functional aspects of performing systems and servers (and the various operating components of the systems) may not be described in detail herein. In addition, the connecting lines shown in the various figures included herein are intended to represent exemplary functional relationships and / or physical couplings between the various elements. It should be noted that many alternative and / or additional functional relationships or physical connections may exist in the embodiments of the subject matter.

[0025] Reference will now be made in detail to exemplary embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Whenever possible, the same reference numerals will be used throughout the drawings to refer to the same or like parts.

[0026] The present disclosure generally relates to a method of using machine learning (such as the NISQA model using a neural network) to dynamically optimize speech enhancement components to improve QoE in a voice communication system. There are various aspects of speech enhancement that can be improved by using the NISQA model, as discussed herein.

[0027] Embodiments of the present disclosure provide machine learning methods that can be used to dynamically optimize the speech enhancement component of a voice communication system. In particular, a neural network can be used as a machine learning method. More specifically, a NISQA using a neural network can be implemented. The method of an embodiment of the present disclosure can be based on using a neural network to train one or more NISQAs to dynamically optimize the speech enhancement component of a voice communication system. The neural networks that can be used include, but are not limited to, deep neural networks, convolutional neural networks, recursive neural networks, and the like.

[0028] Non-limiting examples of speech enhancement components include music detection, acoustic echo cancellation, noise suppression, dereverberation, echo detection, automatic gain control, voice activity detection, jitter buffer management, packet loss concealment, and the like.

[0029] A NISQA using a neural network may be trained using a population-based QoE estimation usage dataset. An example of a NISQA using a neural network is shown in Table 1 below. Although Table 1 depicts one type of neural network-based NISQA, other types of neural network-based NISQA may be implemented within the scope of the present disclosure.

[0030] Table 1

[0031] layer Output size enter 900x120x1 Conv: 128, (3x3), 'ReLU' 900x161x128 Conv: 64, (3x3), "ReLU" 900x161x64 Conv: 64, (3x3), "ReLU" 900x161x64 Conv: 32, (3x3), "ReLU" 900x161x32 MaxPool: (2x^2), Dropout(0.3) 450x80x32 Conv: 32, (3x3), "ReLU" 450x80x32 MaxPool: (2x^2), Dropout(0.3) 225x40x32 Conv: 32, (3x3), "ReLU" 112x20x32 MaxPool: (2x^2), Dropout(0.3) 112x15x32 Conv: 64, (3x3), "ReLU" 112x20x64 GlobalMaxPool 1X64 Dense: 128, 'ReLU' 1x128 Dense: 64, 'ReLU' 1x64 Dense: 1 or 3 1x1 or 2x3

[0032] Another type of NISQA using neural networks includes convolutional neural network (CNN) architectures. For example, a CNN architecture can be applied to a 2D image array and can include two operations: convolution and pooling. A convolutional layer can be responsible for mapping detected features from a receptive region in a previous layer into its cells, which can be called a feature map and is the result of a weighted sum of input features passed through a nonlinearity such as ReLU. A pooling layer can take the maximum and / or average of a set of adjacent feature maps to reduce dimensionality by merging semantically similar features.

[0033] Yet another type of NISQA using a neural network includes a multilayer perceptron (MLP). Such a deep neural network (DNN) can learn feature representations by mapping input features into a linearly separable feature space, which can be implemented by successive linear combinations of input variables followed by nonlinear activation functions. As described above, other types of neural network-based NISQA can be implemented within the scope of the present disclosure.

[0034] As disclosed herein, embodiments dynamically optimize speech enhancement components of a speech communication system.One solution may be to use NISQA to dynamically and / or in real time optimize one or more speech enhancement components in a speech communication system pipeline.

[0035] Figure 1 An exemplary speech enhancement architecture 100 of a speech communication system pipeline according to an embodiment of the present disclosure is depicted. Specifically, Figure 1 A voice communication system pipeline having multiple voice enhancement components is depicted. Figure 1 As shown, microphone 102 may capture audio data including speech of a user of the communication system, etc. The audio data captured by microphone 102 may be processed by one or more speech enhancement components of speech enhancement architecture 100. As described above, non-limiting examples of speech enhancement components include music detection, acoustic echo cancellation, noise suppression, dereverberation, echo detection, automatic gain control, voice activity detection, jitter buffer management, packet loss concealment, etc.

[0036] Figure 1 Depicted is audio data received by a music detection component 104, which can detect whether music is detected in the captured audio data. For example, if audio data is detected by the music detection component 104, the music detection component 104 can notify the user that music has been detected and / or turn off the music. The audio data captured by the microphone 102 can also be received and processed by one or more other speech enhancement components (e.g., an echo cancellation component 106, a noise suppression component 108, and / or a de-reverberation component 110). One or more of the echo cancellation component 106, the noise suppression component 108, and / or the de-reverberation component 110 can be a speech enhancement component that provides microphone and speaker alignment (such as microphone 101 and speaker 134). The echo cancellation component 106 (also referred to as an acoustic echo cancellation component) can receive audio data captured by the microphone 102 and speaker data played by the speaker 134. The echo cancellation component 106 can be used to eliminate acoustic feedback between the speaker 134 and the microphone 102 in the voice communication system.

[0037] The noise suppression component 108 can receive the audio data captured by the microphone 102 and the speaker data played by the speaker 134. The noise suppression component 108 can process the audio data and the speaker data to isolate the voice from other sounds and music during playback. For example, when the microphone 102 is turned on, the background noise around the user (such as flipping through papers, door slamming, dog barking, etc.) may distract other users. The noise suppression component 108 can remove such noise around the user in the voice communication system.

[0038] The dereverberation component 110 may receive audio data captured by the microphone 102 and speaker data played by the speaker 134. The dereverberation component 110 may process the audio data and the speaker data to remove the effects of reverberation, such as reverberant sounds captured by microphones including the microphone 102.

[0039] The audio data may be speech-enhanced audio data after being processed by one or more speech enhancement components (e.g., one or more of the echo cancellation component 106, the noise suppression component 108, and / or the de-reverberation component 110), and further processed by one or more other speech enhancement components. For example, the speech-enhanced audio data may be received and / or processed by one or more of the echo detector 112 and / or the automatic gain control component 114. The echo detector 112 may use the speech-enhanced audio data to detect whether there is an echo in the speech-enhanced audio data, and notify the user of the echo. The automatic gain control component 114 may use the speech-enhanced audio data to amplify and / or increase the volume of the speech-enhanced audio data based on whether the voice activity detector 116 detects speech.

[0040] The voice activity detector 116 can receive the voice-enhanced audio data that has been processed by the automatic gain control component 114, and can detect whether voice activity is detected in the voice-enhanced audio data. Based on the detection by the voice activity detector 116, the user can be notified that he or she is speaking when he or she is silent, the notification can be automatically turned on or off, and / or the automatic gain control component 114 can be instructed to amplify and / or increase the volume of the voice-enhanced audio data.

[0041] The speech-enhanced audio data may then be received by encoder 118 and / or NISQA 120. Encoder 118 may be an audio codec, such as an AI-powered audio codec, for example, a SATIN encoder, which is a digital signal processor with machine learning. Encoder 118 may encode (i.e., compress) the audio data for transmission over network 122. While encoding, encoder 118 may transmit the encoded speech-enhanced audio data to network 122, where other components of the voice communication system are provided. Other components of the voice communication system voice may then transmit audio data of the user and / or other users of the voice communication system over network 122.

[0042] The jitter buffer management component 124 can receive audio data transmitted over the network 122 and process the audio data. For example, the jitter buffer management component 124 can buffer audio data packets to allow the decoder 126 to receive the audio data at evenly spaced intervals. As the audio data is transmitted over the network 122, variations in the arrival time of the data packets, i.e., jitter, may occur due to network congestion, timing drift, and / or routing changes. The jitter buffer management component 124 at the receiving end of the voice communication system can delay the arriving data packets so that the user experiences a clear connection with minimal sound distortion.

[0043] The audio data from the jitter buffer management component 124 may then be received by a decoder 126. The decoder 126 may be an audio codec, such as an AI-powered audio codec, for example, a SATIN decoder, which is a digital signal processor with machine learning. The decoder 126 may decode (i.e., decompress) the audio data received from the network 122. Upon decoding, the decoder 126 may provide the decoded audio data to a packet loss concealment component 128.

[0044] The packet loss concealment component 128 may receive the decoded audio data and may process the decoded audio data to conceal gaps in the audio stream caused by data transmission failures in the network 122. The results of the processing may be provided to one or more of the network quality classifier 130, the call quality estimator component 132, and / or the speaker 134.

[0045] The network quality classifier 130 can classify the quality of the connection to the network 122 based on the information received from the jitter buffer management component 124 and / or the packet loss concealment component 128, and the network quality classifier 130 can inform the user of the quality of the connection to the network 122, such as poor, medium, excellent, etc. When the connection to the network 122 is through the public switched telephone network (PSTN), the call quality estimator component 132 can estimate the quality of the call. The speaker 134 can play the decoded audio data as speaker data. The speaker data can also be provided to one or more of the echo cancellation component 106, the noise suppression component 108, and / or the de-reverberation component 110.

[0046] As described above, the speech enhanced audio data may then be received by NISQA 120. NISQA 120 may be one or more of the above-described NISQAs using a neural network that may be trained to detect the quality of the speech enhanced audio data. In the event that the quality of the speech enhanced audio data is detected, the results may be provided to optimized speech enhancement component(s) 136. Optimized speech enhancement component(s) 136 may determine whether one or more of the speech enhancement components may be changed to another speech enhancement component to improve QoE. In an embodiment, optimized speech enhancement component(s) 136 may be stored on the user's device and may store two or more of the various speech enhancement components discussed above. Based on the results of NISQA 120, the optimized speech enhancement component(s) 136 may dynamically and / or in real time change various speech enhancement components, such as the music detection component 104, the echo cancellation component 106, the noise suppression component 108, the dereverberation component 110, the echo detector 112, the automatic gain control component 114, the jitter buffer management component 124, and / or the packet loss concealment component 128. For clarity, in the figures, the optimized speech enhancement component(s) 136 are not shown connected to each speech enhancement component, but may be connected to each speech enhancement component.

[0047] For example, the optimized (one or more) speech enhancement components 136 may change the noise suppression component 108 to another type of noise suppression component. Then, the new quality of the speech-enhanced audio data may be detected by NISQA 120. If the new quality of the speech-enhanced audio data is higher than the original quality of the speech-enhanced audio data, the optimized (one or more) speech enhancement components 136 may keep the changed noise suppression component 108. If the new quality of the speech-enhanced audio data is not higher than the original quality of the speech-enhanced audio data, the optimized (one or more) speech enhancement components 136 may change the changed noise suppression component 108 back to the original noise suppression component 108 or another type of noise suppression component.

[0048] An exemplary brute force approach pseudocode for implementing the optimization is described below.

[0049]

[0050]

[0051] Figure 2 Another exemplary speech enhancement architecture 200 of a speech communication system pipeline according to an embodiment of the present disclosure is depicted. Specifically, Figure 2 A voice communication system circuit having multiple voice enhancement components is depicted. Figure 2 Similar to Figure 1In the illustrated embodiment, except that the optimized (one or more) speech enhancement component 236 resides in the network 122 and / or the cloud, and the NISQA 220 transmits the optimized (one or more) speech enhancement component 236 via the network 122. The NISQA 220 can be one or more of the above-mentioned NISQAs using a neural network, which can be trained to detect the quality of the speech-enhanced audio data. In the event that the quality of the speech-enhanced audio data is detected, the results can be provided to the optimized (one or more) speech enhancement component 236 via the network 122. The optimized (one or more) speech enhancement component 236 can determine whether one or more of the speech enhancement components can be changed to another speech enhancement component to improve the QoE. In an embodiment, the optimized (one or more) speech enhancement component 236 is transmitted back to the user's device, where various speech enhancement components can be stored. Based on the results of NISQA 220, the optimized(s) speech enhancement component(s) 236 may dynamically and / or in near real time change various speech enhancement components, such as the music detection component 104, the echo cancellation component 106, the noise suppression component 108, the dereverberation component 110, the echo detector 112, the automatic gain control component 114, the jitter buffer management component 124, and / or the packet loss concealment component 128, depending on the speed of the connection to the network and / or the quality of the connection to the network 122.

[0052] Figure 3 300 of a voice communication system pipeline according to an embodiment of the present disclosure. In addition to the NISQA 320 and the optimized (one or more) voice enhancement components 236 residing in the network 122 and / or the cloud, Figure 3 Similar to Figure 2Embodiment shown. NISQA320 can receive the encoded speech-enhanced audio data and detect the quality of the encoded speech-enhanced audio data. NISQA320 can be one or more of the above-mentioned NISQAs using a neural network, which can be trained to detect the quality of the speech-enhanced audio data. In the case where the quality of the encoded speech-enhanced audio data is detected, the result can be provided to the optimized (one or more) speech enhancement components 336 through the network 122. The optimized (one or more) speech enhancement components 336 can determine whether one or more of the speech enhancement components can be changed to another speech enhancement component to improve QoE. In an embodiment, the optimized (one or more) speech enhancement components 336 are transmitted back to the user's device, where various speech enhancement components can be stored. Based on the results of NISQA 320, the optimized(one or more) speech enhancement component(s) 336 may dynamically and / or in near real time change various speech enhancement components, such as the music detection component 104, the echo cancellation component 106, the noise suppression component 108, the dereverberation component 110, the echo detector 112, the automatic gain control component 114, the jitter buffer management component 124, and / or the packet loss concealment component 128, depending on the speed of the connection to the network and / or the quality of the connection to the network 122.

[0053] Figure 4 Another exemplary speech enhancement architecture 400 of a speech communication system pipeline according to an embodiment of the present disclosure is depicted. Specifically, Figure 4 A voice communication system circuit having multiple voice enhancement components is depicted. Figure 4 is shown as similar to Figure 1 The embodiment shown in Figure 4 Can be used with Figure 2 and Figure 3 The embodiment shown in is implemented in a similar manner. Figure 4As shown, NISQA 420 can receive voice-enhanced audio data and device information from the user (i.e., device 440 including microphone 402, speaker 434, and other various components of device 440). The information may include device information of the device (i.e., microphone 402) that captures the audio data. NISQA 420 can detect the quality of the voice of the audio data based on the received device information. For example, depending on the microphone type, the quality of the audio data can change, and NISQA can indicate the optimized (one or more) voice enhancement components 436 based on the quality of the detected voice and the device information to change one or more of the voice enhancement components. Additionally or alternatively, when a change in device information is detected, such as a change in microphone 402, the quality of the audio data can change according to the new microphone type, and NISQA can indicate the optimized (one or more) voice enhancement components 436 based on the quality of the detected voice and the changed device information to change one or more of the voice enhancement components. In addition, instead of microphone or speaker information, NISQA 420 can receive environmental information of the device 440 that is capturing the audio data. NISQA 420 can detect the quality of the voice of the audio data based on the received environmental information. NISQA can indicate the optimized (one or more) voice enhancement components 436 based on the detected quality of the voice and the environmental information and / or when the environmental information changes to change one or more of the voice enhancement components. In addition, NISQA 420 can receive the load of at least one processor of the device 440 that captures the audio data. NISQA 420 can also detect the quality of the voice of the audio data based on the load of at least one processor of the device 440. NISQA can indicate the optimized (one or more) voice enhancement components 436 based on the detected quality of the voice and the load of at least one processor of the device 440 to change one or more of the voice enhancement components. For example, if the load is high, the performance may be degraded, or if the load is low, more processor-intensive voice enhancement components can be used.

[0054] Based on the results of NISQA420, the optimized (one or more) speech enhancement components 436 can dynamically and / or in real time change various speech enhancement components, such as the music detection component 104, the echo cancellation component 106, the noise suppression component 108, the dereverberation component 110, the echo detector 112, the automatic gain control component 114, the jitter buffer management component 124 and / or the packet loss concealment component 128.

[0055] Additionally and / or alternatively, the one or more speech enhancement components that improve speech can be reported back to the server over the network along with the brand and / or model of the device with improved speech enhancement. The server can then aggregate such reports from multiple devices of multiple users, and the one or more speech enhancement components can be used for systems with reporting devices of the same brand and / or model.

[0056] Figure 5 A method 500 for optimizing a speech enhancement component for use in a speech communication system using non-intrusive speech quality assessment according to an embodiment of the present disclosure is depicted. The method 500 may begin at 502, where audio data including speech may be received. The audio data has been processed by at least one speech enhancement component. As described above, the at least one speech enhancement component may include one or more of: acoustic echo cancellation, noise suppression, dereverberation, automatic gain control, packet loss concealment, etc.

[0057] In addition to receiving the audio data, one or more of the following may be received at 504: device information of a device capturing the audio data; environment information of the device capturing the audio data; and a load of at least one processor of the device capturing the audio data.

[0058] In addition, before, after, and / or during receiving the audio data, device information, environmental information, and / or the load of at least one processor, a trained non-intrusive speech quality assessment (NISQA) model, also referred to as NISQA using a neural network model, may be received at 506. Upon receiving the audio data and / or the NISQA model, the trained NISQA model may detect a first quality of speech of the audio data at 508. As described in more detail above, the trained NISQA model may have been trained to automatically detect the quality of speech using a robust data set. In addition to the received audio data, the NISQA model may also detect the quality of speech using one or more of the following: device information, environmental information, and / or the load of at least one processor.

[0059] In certain embodiments of the present disclosure, a first quality of speech of the audio data detected by the NISQA model may be transmitted to at least one server over a network at 510. The at least one server may determine one or more speech enhancement components to be changed by the device at 512. The at least one server may then transmit one or more of the at least one speech enhancement components to be changed to the device capturing the audio data at 514. One or more of the at least one speech enhancement components to be changed based on the transmitted detected first quality of speech may be received by the device capturing the audio data at 516.

[0060] Based on the first quality of the detected speech, one or more of the at least one speech enhancement components may be changed based on the first quality of the detected speech at 518. The changed one or more speech enhancement components may include one or more of acoustic echo cancellation, noise suppression, de-reverberation, automatic gain control, and packet loss concealment. Additionally and / or alternatively, a change in device information may be detected, and when a change in device information is detected, one or more of the at least one speech enhancement components may be changed based on the detected speech quality.

[0061] After changing one or more of at least one speech enhancement component, the trained NISQA model can be used to detect the second quality of the speech of the audio data 520. Then, one or more of the at least one speech enhancement component can be changed at 522 based on the second quality of the detected speech. The speech enhancement component changed based on the second quality of the detected speech and the speech enhancement component changed based on the first quality of the speech act on the same speech enhancement component, such as the same acoustic echo cancellation, noise suppression, de-reverberation, automatic gain control and packet loss concealment. Next, determine whether the second quality of the detected speech is higher than the first quality of the detected speech. When the second quality of the detected speech is higher than the first quality of the detected speech, one or more of the at least one speech enhancement component changed based on the second quality of the detected speech can be maintained. On the contrary, when the second quality of the detected speech is not higher than the first quality of the detected speech, one or more of the at least one speech enhancement component based on the first quality of the detected speech can be changed from one or more of the at least one speech enhancement component changed based on the second quality of the detected speech to the previous at least one speech enhancement component or another speech enhancement component.

[0062] NISQA can be used for detection by checking the user's device for changes in voice enhancement components. Also, looking at network packets to see if something is downloaded in addition to audio data, or to determine if the quality of the voice telecommunication system suddenly improves without active steps by the user. Additionally, if NISQA is stored on the client side, processor usage may be higher than running the voice telecommunication system alone.

[0063] Figure 6A high-level diagram of an exemplary computing device 600 that can be used in accordance with the systems, methods, modules, and computer-readable media disclosed herein, according to an embodiment of the present disclosure, is depicted. For example, according to an embodiment of the present disclosure, the computing device 600 can be used in a system that processes data such as audio data using a neural network. The computing device 600 may include at least one processor 602 that executes instructions stored in a memory 604. For example, the instructions may be instructions for implementing the functions described as being performed by one or more of the components described above or instructions for implementing one or more of the methods described above. The processor 602 may access the memory 604 via a system bus 606. In addition to storing executable instructions, the memory 604 may also store data, audio, one or more neural networks, and the like.

[0064] The computing device 600 may additionally include a data store, also referred to as a database 608, which may be accessed by the processor 602 via the system bus 606. The data store 608 may include executable instructions, data, examples, features, etc. The computing device 600 may also include an input interface 610 that allows external devices to communicate with the computing device 600. For example, the input interface 610 may be used to receive instructions from an external computer device, from a user, etc. The computing device 600 may also include an output interface 612 that connects the computing device 600 to one or more external devices. For example, the computing device 600 may display text, images, etc. through the output interface 612.

[0065] It is contemplated that external devices communicating with computing device 600 via input interface 610 and output interface 612 may be included in an environment providing substantially any type of user interface with which a user may interact. Examples of user interface types include graphical user interfaces, natural user interfaces, and the like. For example, a graphical user interface may accept input from a user using (one or more) input devices such as a keyboard, a mouse, a remote controller, and may provide output on an output device such as a display. In addition, a natural user interface may enable a user to interact with computing device 600 in a manner that is not subject to constraints imposed by input devices such as a keyboard, a mouse, a remote controller, and the like. In contrast, a natural user interface may rely on speech recognition, touch and stylus recognition, gesture recognition on and near the screen, air gestures, head and eye tracking, speech and voice, vision, touch, gestures, machine intelligence, and the like.

[0066] In addition, although shown as a single system, it should be understood that the computing device 600 may be a distributed system. Thus, for example, several devices may communicate via a network connection and may jointly perform the tasks described as being performed by the computing device 600.

[0067] Steering Figure 7 , Figure 7A high-level diagram of an exemplary computing system 700 that can be used in accordance with the systems, methods, modules, and computer-readable media disclosed herein is depicted in accordance with an embodiment of the present disclosure. For example, computing system 700 can be or can include computing device 600. Additionally and / or alternatively, computing device 600 can be or can include computing system 700.

[0068] The computing system 700 may include multiple server computing devices, such as server computing device 702 and server computing device 704 (collectively referred to as server computing devices 702-704). Server computing device 702 may include at least one processor and a memory; at least one processor executes instructions stored in the memory. For example, the instructions may be instructions for implementing the functions described as being performed by the above one or more components or instructions for implementing one or more of the above methods. Similar to server computing device 702, at least one subset of server computing devices 702-704 other than server computing device 702 may respectively include at least one processor and a memory. In addition, at least one subset of server computing devices 702-704 may include corresponding data storage.

[0069] The processor(s) of one or more of the server computing devices 702-704 may be or may include a processor, such as processor 602. Additionally, the memory(s) of one or more of the server computing devices 702-704 may be or may include a memory, such as memory 604. Additionally, the data storage(s) of one or more of the server computing devices 702-704 may be or may include a data storage, such as data storage 608.

[0070] The computing system 700 may also include various network nodes 706 for transmitting data between the server computing devices 702. In addition, the network nodes 706 may transmit data from the server computing devices 702-704 to an external node (e.g., external to the computing system 700) via a network 708. The network nodes 702 may also transmit data from an external node to the server computing devices 702-704 via the network 708. For example, the network 708 may be the Internet, a cellular network, etc. The network nodes 706 may include switches, routers, load balancers, etc.

[0071] The fabric controller 710 of the computing system 700 can manage the hardware resources of the server computing devices 702-704 (e.g., the processors, memory, data storage, etc. of the server computing devices 702-704). The fabric controller 710 can also manage the network nodes 706. In addition, the fabric controller 710 can manage the creation, provisioning, de-provisioning, and supervision of the hosted execution environments instantiated on the server computing devices 702-704.

[0072] As used herein, the terms "component" and "system" are intended to encompass a computer-readable data storage that is configured with computer-executable instructions that perform certain functions when executed by a processor. Computer-executable instructions may include routines, functions, etc. It should also be understood that a component or system may be located on a single device or distributed across several devices.

[0073] The various functions described herein can be implemented in hardware, software, or any combination thereof. If implemented in software, these functions can be stored on and / or transmitted on a computer-readable medium as one or more instructions or codes. Computer-readable media may include computer-readable storage media. Computer-readable storage media may be any available storage medium that can be accessed by a computer. As an example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device, or any other medium that can be used to store the desired program code in the form of an instruction or data structure and can be accessed by a computer. As used herein, disks and optical disks may include optical disks ("CDs"), laser optical disks, optical disks, digital versatile disks ("DVDs"), floppy disks, and Blu-ray disks ("BDs"), wherein the disks typically reproduce data magnetically, and the disks typically reproduce data optically with lasers. In addition, propagation signals are not included in the scope of computer-readable storage media. Computer-readable media may also include communication media, which include any media that facilitates the transfer of a computer program from one place to another. For example, a connection may be a communication medium. For example, if the software is transmitted from a website, server or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line ("DSL") or wireless technologies such as infrared, radio and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave are included in the definition of communication media. Combinations of the above may also be included within the scope of computer-readable media.

[0074] Alternatively and / or additionally, the functions described herein may be performed at least in part by one or more hardware logic components. For example, but not limited to, illustrative types of hardware logic components that may be used include field programmable gate arrays ("FPGAs"), application specific integrated circuits ("ASICs"), application specific standard products ("ASSPs"), systems on chips ("SOCs"), complex programmable logic devices ("CPLDs"), etc.

[0075] What has been described above includes examples of one or more embodiments. Of course, it is not possible to describe every conceivable modification and variation of the above apparatus or method for the purpose of describing the aforementioned aspects, but one of ordinary skill in the art will recognize that many further modifications and variations of various aspects are possible. Therefore, the described aspects are intended to cover all such changes, modifications and variations that fall within the scope of the appended claims.

Claims

1. A computer-implemented method for optimizing a speech enhancement component for use in a speech communication system using non-intrusive speech quality assessment, the method comprising: receiving audio data, the audio data comprising speech and having been processed by at least one speech enhancement component; detecting a first quality of the speech of the audio data using a trained non-intrusive speech quality assessment (NISQA) model, the trained NISQA model being trained to automatically detect the quality of speech; as well as One or more of the at least one speech enhancement components are altered based on the detected first quality of the speech.

2. The method according to claim 1, further comprising: detecting a second quality of the speech of the audio data using the trained NISQA model after changing the one or more of the at least one speech enhancement components; as well as One or more of the at least one speech enhancement components are altered based on the detected second quality of the speech.

3. The method according to claim 2, wherein: The speech enhancement component that is changed based on the detected second quality of the speech and the speech enhancement component that is changed based on the first quality of the speech are effective on the same speech enhancement component, and The method further comprises: determining whether the second quality of the detected speech is higher than the first quality of the detected speech; When the detected second quality of the speech is higher than the detected first quality of the speech, maintaining one or more of the at least one speech enhancement components changed based on the detected second quality of the speech; and When the second quality of the detected speech is not higher than the first quality of the detected speech, one or more of the at least one speech enhancement components that were changed based on the second quality of the detected speech are changed to one or more of the at least one speech enhancement components that were changed based on the first quality of the detected speech.

4. The method according to claim 1, further comprising: Receiving the trained NISQA model; transmitting a first quality of speech of the audio data detected by the NISQA model to at least one server over a network; as well as The one or more of the at least one speech enhancement components are received over the network to be changed based on a first quality of the transmitted detected speech.

5. The method according to claim 1, wherein: The at least one speech enhancement component includes one or more of: acoustic echo cancellation, noise suppression, dereverberation, automatic gain control, and packet loss concealment.

6. The method according to claim 1, wherein: Changing the one or more of the at least one speech enhancement components based on the detected quality of the speech comprises: The one or more of the at least one speech enhancement components are transmitted to a device that captured the audio data.

7. The method according to claim 1, further comprising: receiving device information of a device that captures the audio data; Wherein, using the trained NISQA model to detect the quality of the speech of the audio data further includes: using the trained NISQA model and based on received device information to detect the quality of the speech of the audio data.

8. The method according to claim 7, further comprising: detecting a change in the device information; as well as When the change in the device information is detected, one or more of the at least one speech enhancement components are changed based on the detected quality of the speech.

9. The method according to claim 1, further comprising: receiving environment information of a device capturing the audio data; Wherein, using the trained NISQA model to detect the quality of the speech of the audio data further includes: using the trained NISQA model and based on received environmental information to detect the quality of the speech of the audio data.

10. The method according to claim 1, further comprising: receiving a load of at least one processor of a device for capturing said audio data, Wherein, using the trained NISQA model to detect the quality of the speech of the audio data further comprises: using the trained NISQA model and based on the received load of the at least one processor to detect the quality of the speech of the audio data.

11. A system for optimizing a speech enhancement component for use in a speech communication system using non-intrusive speech quality assessment, the system comprising: a data storage device storing instructions for optimizing a speech enhancement component for use in a speech communication system using a non-intrusive speech quality assessment; as well as A processor configured to execute the instructions to perform a method comprising the following operations: receiving audio data, the audio data comprising speech and having been processed by at least one speech enhancement component; detecting a first quality of the speech of the audio data using a trained non-intrusive speech quality assessment (NISQA) model, the trained NISQA model being trained to automatically detect the quality of speech; as well as One or more of the at least one speech enhancement components are altered based on the detected first quality of the speech.

12. The system according to claim 11, wherein: The processor is further configured to execute the instructions to perform the method comprising the following operations, including: detecting a second quality of the speech of the audio data using the trained NISQA model after changing the one or more of the at least one speech enhancement components; and One or more of the at least one speech enhancement components are altered based on the detected second quality of the speech.

13. The system according to claim 12, wherein: The speech enhancement component that is changed based on the detected second quality of the speech and the speech enhancement component that is changed based on the first quality of the speech are operative on the same speech enhancement component, and The processor is further configured to execute the instructions to perform the method comprising the following operations: determining whether the second quality of the detected speech is higher than the first quality of the detected speech; When the detected second quality of the speech is higher than the detected first quality of the speech, maintaining one or more of the at least one speech enhancement components that are changed based on the detected second quality of the speech; as well as When the second quality of the detected speech is not higher than the first quality of the detected speech, one or more of the at least one speech enhancement components that were changed based on the second quality of the detected speech are changed based on the first quality of the detected speech.

14. A computer readable storage device storing instructions which, when executed by a computer, cause the computer to perform a method for optimizing a speech enhancement component for use in a speech communication system using non-intrusive speech quality assessment, the method comprising: receiving audio data, the audio data comprising speech and having been processed by at least one speech enhancement component; detecting a first quality of the speech of the audio data using a trained non-intrusive speech quality assessment (NISQA) model, the trained NISQA model being trained to automatically detect the quality of speech; as well as One or more of the at least one speech enhancement components are altered based on the detected first quality of the speech.

15. The computer-readable storage medium of claim 14, wherein: When executed by the computer, the instructions cause the computer to perform the method further comprising: detecting a second quality of the speech of the audio data using the trained NISQA model after changing the one or more of the at least one speech enhancement components; as well as One or more of the at least one speech enhancement components are altered based on the detected second quality of the speech.

Images