Personalized speech enhancement without registration

Abstract

Systems, methods, and computer-readable storage devices are disclosed for personalizing speech enhancement components without registration in a speech communication system. In one embodiment, a method includes receiving audio data, the audio data including speech, and the audio data to be processed by at least one speech enhancement component; determining whether the speech of the audio data includes one of near-field speech and far-field speech or both without requiring user registration; and altering one or more of the at least one speech enhancement component based on determining that the speech of the audio data includes one of near-field speech and far-field speech or includes both near-field speech and far-field speech.

Description

Technical Field

[0001] The present disclosure relates to speech enhancement by reducing echo, noise, dereverberation, etc. In particular, the present disclosure relates to personalized speech enhancement in a voice communication system without requiring user enroll. Background Art

[0002] In a voice communication system, the audio signal may be affected by echo, background noise, reverberation, enhancement algorithms, network impairments, etc. Providers of voice communication systems who are trying to provide optimal and reliable services to their customers can estimate the perceived quality of the audio signal. For example, voice quality prediction may be useful during network design and development, and for monitoring and improving the quality of experience (QoE) of customers.

[0003] To improve the QoE of the users, voice enhancement components are critical to telecommunications, which are used to reduce echo, noise, dereverberation, etc. Many of these voice enhancement components can be based on acoustic digital signal processing (ADSP) algorithms, deep learning components, and / or personalized based on specific training of the customer. The problem with ADSP algorithms is that they are not personalized for each customer. The problem with deep learning voice enhancement components is that they are only as good as the data used to train them, and the data may not be personalized for each customer.

[0004] The benefit of personalized voice enhancement is that it is targeted to a specific customer, and such a system can remove any sound that is not the customer's voice, including speech. However, some current personalized voice enhancement components require the customer to register and / or train the voice enhancement component, which may take a lot of time, a lot of memory, and / or a lot of processing. For example, some current personalized voice enhancement components require the customer to say a few sentences to characterize their voice. A big problem with registration is that very few customers register for personalized voice enhancement themselves.

[0005] Therefore, there is a need for a personalized speech enhancement component that does not require enrollment (eg, training) and that automatically improves a customer's QoE without active participation by the customer. Summary of the invention

[0006] According to certain embodiments, systems, methods, and computer-readable media are disclosed for personalized speech enhancement without registration in a voice communication system.

[0007] According to certain embodiments, a computer-implemented method for personalizing speech enhancement components in a speech communication system without requiring registration is disclosed. A method includes: receiving audio data, the audio data including speech, and the audio data to be processed by at least one speech enhancement component; determining whether the speech of the audio data includes one or both of near-field speech and far-field speech without requiring user registration; and changing one or more of the at least one speech enhancement component based on determining whether the speech of the audio data includes one or both of near-field speech and far-field speech.

[0008] According to certain embodiments, a system for personalizing speech enhancement components in a speech communication system without registration is disclosed. A system includes: a data storage device storing instructions for personalizing speech enhancement components in a speech communication system without registration; and a processor configured to execute instructions to implement a method, the method including: receiving audio data, the audio data including speech, and the audio data to be processed by at least one speech enhancement component; determining whether the speech of the audio data includes one of near-field speech and far-field speech or both near-field speech and far-field speech without requiring user registration; and changing one or more speech enhancement components of the at least one speech enhancement component based on determining whether the speech of the audio data includes one of near-field speech and far-field speech or both near-field speech and far-field speech.

[0009] According to certain embodiments, a computer-readable storage device storing instructions is disclosed that, when executed by a computer, causes the computer to implement a method for personalizing speech enhancement components in a speech communication system without registration. A method of a computer-readable storage device includes: receiving audio data, the audio data including speech, and the audio data to be processed by at least one speech enhancement component; determining whether the speech of the audio data includes one of near-field speech and far-field speech or both near-field speech and far-field speech without requiring user registration; and changing one or more speech enhancement components of the at least one speech enhancement component based on determining whether the speech of the audio data includes one of near-field speech and far-field speech or both near-field speech and far-field speech.

[0010] Additional objects and advantages of the disclosed embodiments will be set forth in part in the following description, and in part will be obvious from the description, or can be learned by practicing the disclosed embodiments. The objects and advantages of the disclosed embodiments will be realized and obtained by means of the elements and combinations particularly pointed out in the appended claims.

[0011] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosed embodiments, as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In the course of the following detailed description, reference will be made to the accompanying drawings. The accompanying drawings illustrate different aspects of the present disclosure, and where appropriate, reference numerals showing the same structures, components, materials and / or elements in different drawings are similarly labeled. It should be understood that various combinations of structures, components and / or elements other than those specifically shown are contemplated and within the scope of the present disclosure.

[0013] In addition, many embodiments of the present disclosure are described and illustrated herein. The present disclosure is not limited to any single aspect or its embodiment, nor to any combination and / or arrangement of these aspects and / or embodiments. In addition, each of the aspects and / or embodiments of the present disclosure may be adopted alone or in combination with one or more of the other aspects and / or embodiments of the present disclosure. For the sake of brevity, specific arrangements and combinations are not discussed and / or illustrated separately herein.

[0014] Figure 1 An exemplary speech enhancement architecture for a speech communication system pipeline according to an embodiment of the present disclosure is depicted.

[0015] Figure 2 Methods for training a neural network to detect near-field speech and / or far-field speech and / or for training a personalized speech enhancement component using a neural network are described according to embodiments of the present disclosure.

[0016] Figure 3 A method 300 for personalizing a speech enhancement component in a speech communication system without registration is depicted according to an embodiment of the present disclosure.

[0017] Figure 4 A high-level diagram of an exemplary computing device that may be used in accordance with the systems, methods, and computer-readable media disclosed herein is depicted in accordance with an embodiment of the present disclosure.

[0018] Figure 5 A high-level diagram of an exemplary computing system that can be used in accordance with the systems, methods, and computer-readable media disclosed herein is depicted in accordance with an embodiment of the present disclosure.

[0019] Once again, many embodiments are described and illustrated herein. The present disclosure is not limited to any single aspect or embodiment thereof, nor to any combination and / or arrangement of these aspects and / or embodiments. Each of the aspects and / or embodiments of the present disclosure may be employed alone or in combination with one or more of the other aspects and / or embodiments of the present disclosure. For the sake of brevity, many of these combinations and arrangements are not discussed separately herein. DETAILED DESCRIPTION

[0020] Those skilled in the art will recognize that various implementations and embodiments of the present disclosure can be practiced according to the description, and all such implementations and embodiments are intended to be included within the scope of the present disclosure.

[0021] As used herein, the terms "comprises," "comprising," "have," "having," "include," "including," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. The term "exemplary" is used in the sense of "example" rather than "ideal." Additionally, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, the phrase "X employs A or B" is intended to mean any of the natural inclusive permutations. For example, the phrase "X employs A or B" is satisfied by any of the following instances: X employs A; X employs B; or X employs both A and B. Furthermore, the articles "a" and "an" used in the present application and the appended claims should generally be interpreted as meaning "one or more," unless otherwise specified or clear from the context to be directed to a singular form.

[0022] For the sake of brevity, conventional techniques related to systems and servers for performing methods and other functional aspects of systems and servers (and the various operating components of the systems) may not be described in detail herein. In addition, the connecting lines shown in the various figures included herein are intended to represent exemplary functional relationships and / or physical couplings between the various elements. It should be noted that many alternative and / or additional functional relationships or physical connections may exist in the embodiments of the subject matter.

[0023] Reference will now be made in detail to exemplary embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numerals will be used throughout the drawings to refer to the same or like parts.

[0024] Among other features, the present disclosure generally relates to a method for improving QoE in a voice communication system by personalizing an enhancement component without registration using machine learning.As discussed herein, there are various aspects of speech enhancement that can be improved by using a personalized speech enhancement component.

[0025] Embodiments of the present disclosure provide a machine learning method that can be used to personalize a speech enhancement component of a speech communication system. In particular, a neural network can be used as a machine learning method. The method of an embodiment of the present disclosure can be based on training one or more neural networks to detect a specific type of speech, and then based on detecting the specific type of speech, the speech enhancement component of the speech communication system is changed to a personalized speech enhancement component. The neural networks that can be used include, but are not limited to, deep neural networks, convolutional neural networks, recursive neural networks, etc.

[0026] Non-limiting examples of speech enhancement components that may be personalized using a trained neural network include one or more of acoustic echo cancellation, noise suppression, dereverberation, automatic gain control, etc. For example, a personalized noise suppression component may remove all noise and speech except the user's own voice, and a personalized automatic gain control may be activated when the user's own voice is detected.

[0027] A data set can be used to train a neural network. For example, such a data set may include: a data set of audio data containing noise, a data set of audio data containing clean speech, and a data set of audio data containing room response. These data sets can be combined to create noisy speech, and then the neural network can be trained to remove everything except human speech. In addition, the data set may include audio data of one or both of near-field speech and / or far-field speech. Far-field speech is the speech that a user speaks to a receiving device (i.e., a microphone) from a long distance (e.g., greater than or equal to 0.5m). Near-field speech is the speech that a user speaks to a receiving device (i.e., a microphone) from a close distance (e.g., less than 0.5m). More specifically, near-field speech can be speech captured by a personal endpoint (device), and far-field speech can be speech that is not captured by a personal endpoint. In a particular embodiment, a data set that uses only near-field speech as clean speech and adds far-field speech as an interference term (noise) can be used to train one or more neural networks.

[0028] In an embodiment of the present disclosure, when it is determined that the user is wearing headphones, using a mobile phone, using earbuds, and / or using any personal endpoint (device) that captures near-field sound, or when one or more neural networks trained to detect near-field speech detect near-field speech, one or more speech enhancement components may be changed to corresponding personalized speech enhancement components. Therefore, the user is not required to register, that is, to undergo a personalized training process using pre-trained data. Because the user's device is a personal endpoint or because the user is determined to be near a personal endpoint, there is no impairment of suppressing far-field speech.

[0029] In addition, the neural network can use various speech enhancement components (e.g., acoustic echo cancellation, noise suppression, dereverberation, automatic gain control, etc.) and one or more of the above-mentioned data sets to create personalized speech enhancement components using the model creation, model validation, and model utilization techniques described below. For example, the neural network can use noise suppression techniques and be trained to remove noise and far-field speech.

[0030] Those skilled in the art will appreciate that a neural network can be constructed with respect to a model and can include the following phases: model creation (neural network training), model validation (neural network testing), and model utilization (neural network evaluation), although these phases may not be mutually exclusive. According to an embodiment of the present disclosure, a neural network can be implemented through training, testing, and evaluation phases. The input samples of the above-mentioned audio data can be used for neural network training and testing together with corresponding ground truth labels. For a baseline neural network, the model can have an input layer including a predetermined number of neurons, at least one intermediate (hidden) layer each including another predetermined number of neurons, and an output layer having another predetermined number of neurons.

[0031] At least one server may execute the machine learning component of the audio processing system described herein. As will be appreciated by those skilled in the art, machine learning may be performed with respect to a model and may include at least three phases: model creation, model validation, and model utilization, although these phases may not be mutually exclusive. As discussed in more detail below, model creation, validation, and utilization may be an ongoing process of machine learning.

[0032] For machine learning, the model creation phase can involve extracting features from a training data set. The machine learning component can monitor ongoing audio data to extract features. As will be appreciated by those skilled in the art, these extracted features and / or other data can be derived based on patterns based on machine learning techniques for large amounts of data collected over time. Based on observations of this monitoring, the machine learning component can create a model (i.e., a set of rules or heuristics) for extracting features from audio data. A baseline neural network can be trained to, for example, minimize classification error and / or minimize the squared error between ground truth and predicted labels.

[0033] During the second stage of machine learning, the accuracy of the created model can be verified. During this stage, the machine learning component can monitor a test data set, extract features from the test data set, and compare those extracted features with the predicted labels made by the model. By continuing to track and compare this information and over a period of time, the machine learning component can determine whether the model accurately recognizes near-field speech and far-field speech. This verification is typically expressed in terms of accuracy: that is, the percentage of time that the model predicts the correct label (e.g., near-field speech and far-field speech). Information about the success or failure of the model's predictions can be fed back to the model creation stage to improve the model, thereby improving the accuracy of the model.

[0034] During the inference phase, additional data from the test dataset can be applied to the trained baseline neural network to generate predicted labels. The predicted labels can then be compared with the ground truth labels to calculate performance metrics including mean squared error.

[0035] The third stage of machine learning can be based on a model that is verified to have an accuracy of a predetermined threshold. For example, a model determined to have at least 90% accuracy can be suitable for the utilization stage. According to an embodiment of the present disclosure, during this third utilization stage, a machine learning component can extract features from audio data, wherein the model suggests classification of near-field speech or far-field speech of the audio data. When classifying the voice type in the audio data, the model outputs the classification and the classification can be stored as a data segment. Of course, information based on confirmation or rejection of various stored data segments can be returned to the previous two stages (verification and creation) as data for refining the model in order to improve the accuracy of the model. 100% accuracy may not be required, and a user interface can be shown to the user, which shows that a personalized microphone capture mode is enabled. If the personalized voice enhancement component is not used, the user can easily see that it is in a personalized microphone capture mode and change it to a non-personalized mode.

[0036] As described above, the neural network can use various speech enhancement components (e.g., acoustic echo cancellation, noise suppression, dereverberation, automatic gain control, etc.) and one or more of the above-mentioned data sets to create personalized speech enhancement components using the above-mentioned model creation, model validation, and model utilization techniques. For example, the neural network can use noise suppression techniques and be trained to remove noise and far-field speech and produce a personalized noise suppressor.

[0037] In combination with the above, embodiments of the present disclosure provide personalized voice enhancement components for voice communication systems that can be used without requiring registration. One solution may be to dynamically select one or more voice enhancement components when one or both of a personal endpoint and / or near-field voice is detected in received audio data, the one or more voice enhancement components being created using a neural network trained to recognize and remove far-field voice.

[0038] Figure 1 An exemplary speech enhancement architecture 100 of a speech communication system pipeline according to an embodiment of the present disclosure is depicted. Specifically, Figure 1 A speech communication system pipeline with multiple speech enhancement components is depicted. Figure 1 As shown, the microphone 102 of the device 140 can capture audio data, including the speech of a user of the communication system, etc. The audio data captured by the microphone 102 can be processed by one or more speech enhancement components of the speech enhancement architecture 100. Non-limiting examples of speech enhancement components include music detection, acoustic echo cancellation, noise suppression, dereverberation, echo detection, automatic gain control, voice activity detection, jitter buffer management, packet loss concealment, etc.

[0039] Figure 1 Depicted is audio data received by a music detection component 104, which can detect whether music is detected in the captured audio data. For example, if the music detection component 104 detects audio data, the music detection component 104 can notify the user that music has been detected and / or turn off the music. The audio data captured by the microphone 102 can also be received and processed by one or more other speech enhancement components (e.g., an echo cancellation component 106, a noise suppression component 108, and / or a de-reverberation component 110). One or more of the echo cancellation component 106, the noise suppression component 108, and / or the de-reverberation component 110 can be a speech enhancement component that provides alignment of a microphone and a speaker (e.g., a microphone 101 and a speaker 134 of a device 140). The echo cancellation component 106 (also referred to as an acoustic echo cancellation component) can receive audio data captured by the microphone 102 and speaker data played by the speaker 134. The echo cancellation component 106 can be used to eliminate acoustic feedback between the speaker 134 and the microphone 102 in a voice communication system.

[0040] The noise suppression component 108 can receive audio data captured by the microphone 102 and speaker data played by the speaker 134. The noise suppression component 108 can process the audio data and the speaker data to isolate the voice from other sounds and music during playback. For example, when the microphone 102 is turned on, background noise around the user (e.g., shuffling papers, slamming doors, barking dogs, etc.) may distract other users. The noise suppression component 108 can remove such noise around the user in the voice communication system.

[0041] The dereverberation component 110 may receive audio data captured by the microphone 102 and speaker data played by the speaker 134. The dereverberation component 110 may process the audio data and the speaker data to remove the effects of reverberation (eg, reverberant sound captured by microphones including the microphone 102).

[0042] The audio data may be speech-enhanced audio data after being processed by one or more speech enhancement components (e.g., one or more of the echo cancellation component 106, the noise suppression component 108, and / or the de-reverberation component 110), and further processed by one or more other speech enhancement components. For example, the speech-enhanced audio data may be received and / or processed by one or more of the echo detector 112 and / or the automatic gain control component 114. The echo detector 112 may use the speech-enhanced audio data to detect whether an echo is present in the speech-enhanced audio data, and notify the user of the echo. The automatic gain control component 114 may use the speech-enhanced audio data to amplify and / or increase the volume of the speech-enhanced audio data based on whether the voice activity detector 116 detects speech.

[0043] The voice activity detector 116 can receive the voice-enhanced audio data that has been processed by the automatic gain control component 114, and can detect whether voice activity is detected in the voice-enhanced audio data. Based on the detection by the voice activity detector 116, the user can be notified that he or she is talking while silent, the notification can be automatically turned on or off, and / or the automatic gain control component 114 can be instructed to amplify and / or increase the volume of the voice-enhanced audio data.

[0044] The speech-enhanced audio data may then be received by encoder 118. Encoder 118 may be an audio codec, such as an AI-enabled audio codec, such as a SATIN encoder, which is a digital signal processor with machine learning. Encoder 118 may encode (i.e., compress) the audio data for transmission over network 122. While encoding, encoder 118 may transmit the encoded speech-enhanced audio data to network 122, where other components of the voice communication system are provided. Other components of the voice communication system may then transmit audio data of the user and / or other users of the voice communication system over network 122.

[0045] The jitter buffer management component 124 can receive audio data transmitted over the network 122 and process the audio data. For example, the jitter buffer management component 124 can buffer packets of audio data to allow the decoder 126 to receive the audio data at evenly spaced intervals. Because the audio data is transmitted over the network 122, there may be variations in packet arrival times, i.e., jitter, which may occur due to network congestion, timing drift, and / or routing changes. The jitter buffer management component 124 at the receiving end of the voice communication system can delay arriving packets so that the user experiences a clear connection with minimal sound distortion.

[0046] The audio data from the jitter buffer management component 124 may then be received by a decoder 126. The decoder 126 may be an audio codec, such as an AI-enabled audio codec, such as a SATIN decoder, which is a digital signal processor with machine learning. The decoder 126 may decode (i.e., decompress) the audio data received via the network 122. Upon decoding, the decoder 126 may provide the decoded audio data to a packet loss concealment component 128.

[0047] The packet loss concealment component 128 may receive the decoded audio data and may process the decoded audio data to conceal gaps in the audio stream caused by data transmission failures in the network 122. The results of the processing may be provided to one or more of the network quality classifier 130, the call quality estimator component 132, and / or the speaker 134.

[0048] The network quality classifier 130 can classify the quality of the connection to the network 122 based on the information received from the jitter buffer management component 124 and / or the packet loss concealment component 128, and the network quality classifier 130 can inform the user of the quality of the connection to the network 122, for example, poor, medium, excellent, etc. The call quality estimator component 132 can estimate the quality of the call when the connection to the network 122 is through the public switched telephone network (PSTN). The speaker 134 can play the decoded audio data as speaker data. The speaker data can also be provided to one or more of the echo cancellation component 106, the noise suppression component 108, and / or the de-reverberation component 110. The device 140 can include one or both of the microphone 102 and / or the speaker 134, for example, the device 140 can be a combined microphone and speaker, for example, a headset, a mobile phone, a teleconferencing device, a smart speaker, etc., and / or the microphone and speaker of the device 140 can be independent and different, etc.

[0049] The audio data of speech enhancement may be received by the personalized device detection component 120 and / or the audio data including speech from the microphone 102 of the device 140 and / or the speaker data of the speaker 134 of the device 140 may also be received by the personalized device detection component 120. In addition, the personalized device detection component 120 may be directly or indirectly connected to one or more speech enhancement components, such as the echo cancellation component 106, the noise suppression component 108, the de-reverberation component 110, the automatic gain control component 114, etc. The personalized device detection component 120 may additionally / optionally receive information from one or more speech enhancement components to determine that one or more speech enhancement components are being used. Additionally, the personalized device detection component 120 transmits an indication to the one or more speech enhancement components and / or transmits a specific one or more speech enhancement components to be used, which indicates that the corresponding one or more speech enhancement components are changed to different one or more speech enhancement components.

[0050] As described above, the personalized device detection component 120 can receive information about the device 140, and the information about the microphone 102 can be used to determine whether the device 140 is capturing near-field speech or far-field speech. For example, if the information about the device 140 indicates that a personal device is being used, the personalized device detection component 120 can determine that the audio data being captured by the personalized device (i.e., the device 140) is near-field speech. As described above, personal devices include personal audio devices, such as headphones, mobile phones, earbuds, etc. In other words, a personal audio device is a device intended to be used alone only at the near end. In contrast, a non-personal audio device may include a speakerphone, which is intended to be used by a group of individuals. The audio data from the personal device may have a high signal-to-noise ratio and low reverberation. Therefore, the personalized device detection component 120 can determine whether the audio data is near-field speech or far-field speech.

[0051] Additionally and / or alternatively, the personalized device detection component 120 may be one or more neural networks in a neural network trained to determine whether the audio data and / or voice-enhanced audio data is near-field speech or far-field speech. One or more of the trained neural networks of the personalized device detection component 120 may determine whether the audio data is near-field speech or far-field speech. Neither the determination of whether the audio data is near-field speech or far-field speech through information about the device 140 and / or through the use of one or both of the trained neural networks requires user registration.

[0052] Additionally and / or alternatively, the personalized device detection component 120 can use a reverberation time 60 (RT60) metric to determine whether the audio data includes near-field speech. The RT60 metric is defined as a measure of the time it takes for the sound pressure level to drop by 60 dB after speech in the audio data ceases. In addition to the RT60 metric, a signal-to-noise ratio and / or a speech-to-reverberation modulation energy ratio (SRMR) greater than 40 dB for near-field devices can also be used.

[0053] Therefore, the personalized device detection component 120 can determine whether the voice of the audio data includes one of the near-field voice and the far-field voice or both the near-field voice and the far-field voice without requiring the user to register. For example, depending on the use case, if it is determined that the personal device is in use, the far-field voice can be removed. For non-personal devices, the far-field voice is not removed. When it is determined that the voice of the audio data includes one of the near-field voice and the far-field voice or includes both the near-field voice and the far-field voice, the personalized device detection component 120 can change one or more voice enhancement components in at least one voice enhancement component based on the determination. For example, the personalized device detection component 120 can change one or more voice enhancement components in the voice enhancement component to one or more voice enhancement components that have been trained using near-field voice and far-field voice, wherein the near-field voice is used as a clean voice and the far-field voice is used as an interference item. In particular, when the voice of the audio data includes only near-field voice, or includes both near-field voice and far-field voice, each of the one or more voice enhancement components can be changed to a corresponding personalized voice enhancement component. For example, each of an acoustic echo cancellation component, a noise suppression component, a de-reverberation component, and an automatic gain control can be changed to a corresponding personalized acoustic echo cancellation component, a personalized noise suppression component, a personalized de-reverberation component, and a personalized automatic gain control. Additionally and / or alternatively, each of the changed corresponding personalized speech enhancement components can be a corresponding neural network model that has been trained with far-field speech. The personalized speech enhancement component using the trained neural network model is a personalized noise suppression component that uses a data set with only near-field speech as clean speech and adds a data set with only far-field speech as interference to train a personalized noise suppression component neural network to perform noise suppression on far-field speech.

[0054] When the speech of the audio data is determined to include only far-field speech and / or when a non-personal device is detected, one or more speech enhancement components can remain the same and / or be changed by the personalized device detection component 120 to a corresponding speech enhancement component that does not remove far-field speech.

[0055] Based on the results of the personalized device detection component 120, one or more speech enhancement components can dynamically and / or in real time change various personalized speech enhancement components, such as the echo cancellation component 106, the noise suppression component 108, the de-reverberation component 110, the automatic gain control component 114, the jitter buffer management component 124 and / or the packet loss concealment component 128.

[0056] Additionally and / or alternatively, one or more speech enhancement components that improve speech can be reported back to the server over the network by the personalized device detection component 120 along with the brand and / or model of the device with improved speech enhancement. In turn, the server can aggregate such reports from multiple devices of multiple users, and the one or more speech enhancement components can be used in a system with the same brand and / or model as the reporting device. Alternatively, the personalized device detection component 120 resides on the network 122 and / or in the cloud and communicates with one or more of the speech enhancement components of the speech enhancement architecture 100 of the voice communication system pipeline over the network.

[0057] Figure 2 A method 200 for training a neural network to detect near-field speech and / or far-field speech and / or for training a personalized speech enhancement component using a neural network according to an embodiment of the present disclosure is depicted. Method 200 may begin at 202, where a neural network model may be constructed and / or received according to an instruction set. The neural network model may include a plurality of neurons. The neural network model may be configured to output a classification of audio data as near-field speech and / or far-field speech, and / or to output features of a corresponding personalized speech enhancement component based on whether the speech includes near-field speech and / or far-field speech. The plurality of neurons may be arranged in a plurality of layers including at least one hidden layer, and may be connected by connections. Each connection includes a weight. The neural network model may include, for example, a convolutional neural network model.

[0058] Then, at 204, a training data set may be received. The training data set may include audio data. The audio data may include only near-field speech as clean speech and add far-field speech as interference (noise). Near-field speech may be speech captured by a personal endpoint (device), and far-field speech may be speech not captured by a personal endpoint. Therefore, for personalized speech enhancement, a far-field data set and a near-field data set may be used, the far-field data set being the sound to be removed for noise suppression and / or the sound to be ignored for automatic gain control. However, embodiments of the present disclosure are not necessarily limited to audio data, and may include, for example, video data with audio data.

[0059] At 206, a neural network model may be trained using a training data set. Then, at 208, the trained neural network model may be output. The trained neural network model may be used to output a predicted label for audio data (e.g., near-field speech and / or far-field speech), and / or the trained neural network model may be a trained personalized speech enhancement component using a neural network. The trained deep neural network model may include a plurality of neurons arranged in a plurality of layers including at least one hidden layer, and may be connected by connections. Each connection may include a weight. In a particular embodiment of the present disclosure, the neural network may include one of one hidden layer, two hidden layers, three hidden layers, and four hidden layers.

[0060] At 210, a test data set may be received. Alternatively and / or additionally, a test data set may be created. Furthermore, embodiments of the present disclosure are not necessarily limited to audio data. For example, the test data set may include one or more video data containing audio content.

[0061] Then, at 212, the trained neural network can then be tested using the test data set for evaluation. Furthermore, once evaluated to exceed a predetermined threshold, the trained neural network can be utilized. Additionally, in certain embodiments of the present disclosure, the steps of method 200 can be repeated to generate multiple trained neural networks. The multiple trained neural networks can then be compared to each other and / or to other neural networks. Alternatively, 210 and 212 can be omitted. Then, the trained and tested neural network model can be output at 214.

[0062] Figure 3 A method 300 for personalizing a speech enhancement component in a speech communication system without registration according to an embodiment of the present disclosure is depicted. The method 300 may begin at 302, where audio data including speech may be received. The audio data has been processed by at least one speech enhancement component. As described above, the at least one speech enhancement component may include one or more of acoustic echo cancellation, noise suppression, dereverberation, automatic gain control, etc. In addition to receiving the audio data, one or more of device information of a device that captured the audio data may be received at 304.

[0063] Additionally, before, after, and / or during receiving audio data, device information, a trained neural network may be received at 306, the neural network being trained to detect whether the speech of the audio data is near-field speech or far-field speech. For example, each personalized speech enhancement component is a neural network model that has been trained using far-field speech. In particular, for example, a personalized noise suppression component uses a data set with only near-field speech as clean speech and adds a data set with only far-field speech as interference to train a personalized noise suppression component neural network to perform noise suppression on far-field speech. Additionally, after and / or during receiving a trained neural network, one or more personalized speech enhancement components using a trained neural network model may be received. In one embodiment, one or more personalized speech enhancement components may be received at step 310 below.

[0064] Then, at 308, without any user registration and / or requiring user participation, it can be determined whether the voice of the received audio data includes one of near-field voice and far-field voice or both. In an embodiment, the determination can be made by one or both of the following: determining whether the audio data is captured using a personalized device based on the received device information, and determining whether the audio data includes near-field voice using a trained neural network that may have been previously received. Alternatively and / or in addition, as opposed to using a trained neural network, it can be determined whether the audio data includes near-field voice by using a reverberation time 60 (RT60) metric. The RT60 metric is defined as a measurement of the time it takes for the sound pressure level to drop by 60 dB after the voice of the audio data stops.

[0065] Next, at 310, one or more speech enhancement components in at least one speech enhancement component can be changed based on whether the voice of the audio data includes one of the near-field voice and the far-field voice or includes both the near-field voice and the far-field voice. In particular, one or more speech enhancement components in at least one speech enhancement component can be changed to one or more speech enhancement components that have been trained with near-field voice as a clean voice and far-field voice as an interference item. Additionally and / or alternatively, when the voice of the audio data includes only near-field voice or includes near-field voice and far-field voice, each speech enhancement component in one or more speech enhancement components in at least one speech enhancement component can be changed to a corresponding personalized speech enhancement component. When the voice of the audio data includes only far-field voice, each speech enhancement component in one or more speech enhancement components in at least one speech enhancement component can remain the same or can be changed to a corresponding speech enhancement component that does not remove the far-field voice. The changed one or more speech enhancement components in at least one speech enhancement component include one or more of the following: acoustic echo cancellation, noise suppression, de-reverberation, automatic gain control, etc.

[0066] Detecting the use of a personalized voice enhancement component can be accomplished by checking the user device for changes to the voice enhancement component without user involvement. Additionally, network packets can be viewed to check if something other than audio data is being downloaded, or to determine if the quality of a voice telecommunications system has suddenly improved, without active steps by the user.

[0067] Figure 4 A high-level diagram of an exemplary computing device 400 that can be used in accordance with the systems, methods, modules, and computer-readable media disclosed herein, according to an embodiment of the present disclosure, is depicted. For example, according to an embodiment of the present disclosure, the computing device 400 can be used in a system that processes data (e.g., audio data) using a neural network. The computing device 400 may include at least one processor 402 that executes instructions stored in a memory 404. The instructions may be, for example, instructions for implementing the functions described as being performed by one or more of the components described above or instructions for implementing one or more of the methods described above. The processor 402 may access the memory 404 via a system bus 406. In addition to storing executable instructions, the memory 404 may also store data, audio, one or more neural networks, and the like.

[0068] The computing device 400 may additionally include a data store 408, also referred to as a database, accessible by the processor 402 via the system bus 406. The data store 408 may include executable instructions, data, examples, features, etc. The computing device 400 may also include an input interface 410 that allows external devices to communicate with the computing device 400. For example, the input interface 410 may be used to receive instructions from an external computer device, from a user, etc. The computing device 400 may also include an output interface 412 that interfaces the computing device 400 with one or more external devices. For example, the computing device 400 may display text, images, etc. through the output interface 412.

[0069] It is contemplated that external devices communicating with computing device 400 via input interface 410 and output interface 412 may be included in an environment providing substantially any type of user interface with which a user may interact. Examples of user interface types include graphical user interfaces, natural user interfaces, etc. For example, a graphical user interface may accept input from a user using an input device such as a keyboard, mouse, remote controller, etc., and output may be provided on an output device such as a display. In addition, a natural user interface may enable a user to interact with computing device 400 without being constrained by an input device such as a keyboard, mouse, remote controller, etc. Instead, a natural user interface may rely on speech recognition, touch and stylus recognition, gesture recognition on and near the screen, air gestures, head and eye tracking, voice and speech, vision, touch, gestures, machine intelligence, etc.

[0070] Additionally, although shown as a single system, it should be understood that computing device 400 may be a distributed system. Thus, for example, several devices may communicate via a network connection and may jointly perform the tasks described as being performed by computing device 400.

[0071] Go to Figure 5 , Figure 5 A high-level diagram of an exemplary computing system 500 that can be used in accordance with the systems, methods, modules, and computer-readable media disclosed herein is depicted in accordance with an embodiment of the present disclosure. For example, computing system 500 can be or include computing device 400. Additionally and / or alternatively, computing device 400 can be or include computing system 500.

[0072] The computing system 500 may include multiple server computing devices, for example, server computing device 502 and server computing device 504 (collectively referred to as server computing devices 502-504). Server computing device 502 may include at least one processor and a memory; at least one processor executes instructions stored in the memory. The instructions may be, for example, instructions for implementing the functions described as being performed by one or more of the above components or instructions for implementing one or more of the above methods. Similar to server computing device 502, at least a subset of server computing devices 502-504 other than server computing device 502 may each include at least one processor and a memory, respectively. In addition, at least a subset of server computing devices 502-504 may include corresponding data storage.

[0073] The processor of one or more of server computing devices 502-504 may be or may include a processor, such as processor 402. Additionally, the memory (or memories) of one or more of server computing devices 502-504 may be or may include a memory, such as memory 404. Additionally, the data store (or data stores) of one or more of server computing devices 502-504 may be or may include a data store, such as data store 408.

[0074] The computing system 500 may also include various network nodes 506 for transmitting data between the server computing devices 502-504. In addition, the network nodes 506 may transmit data from the server computing devices 502-504 to external nodes (e.g., nodes external to the computing system 500) via a network 508. The network nodes 502 may also transmit data from the external nodes to the server computing devices 502-504 via the network 508. The network 508 may be, for example, the Internet, a cellular network, etc. The network nodes 506 may include switches, routers, load balancers, etc.

[0075] The fabric controller 510 of the computing system 500 can manage the hardware resources of the server computing devices 502-504 (e.g., the processors, memory, data storage, etc. of the server computing devices 502-504). The fabric controller 510 can also manage the network nodes 506. In addition, the fabric controller 510 can manage the creation, provisioning, de-provisioning, and supervision of managed runtime environments instantiated on the server computing devices 502-504.

[0076] As used herein, the terms "component" and "system" are intended to encompass computer-readable data storage configured with computer-executable instructions that, when executed by a processor, cause specific functions to be performed. Computer-executable instructions may include routines, functions, etc. It should also be understood that a component or system may be located on a single device or distributed across several devices.

[0077] The various functions described herein can be implemented in hardware, software, or any combination thereof. If implemented in software, these functions can be stored on and / or transmitted on a computer-readable medium as one or more instructions or codes. Computer-readable media may include computer-readable storage media. Computer-readable storage media may be any available storage medium that can be accessed by a computer. By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired program codes in the form of instructions or data structures and can be accessed by a computer. As used herein, disks and optical disks may include compact disks ("CDs"), laser disks, optical disks, digital versatile disks ("DVDs"), floppy disks, and Blu-ray disks ("BDs"), wherein disks typically reproduce data magnetically, and optical disks typically reproduce data optically with lasers. In addition, propagated signals are not included in the scope of computer-readable storage media. Computer-readable media may also include communication media, including any media that facilitates the transfer of computer programs from one place to another. For example, a connection may be a communication medium. For example, if the software is transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line ("DSL"), or wireless technologies such as infrared, radio and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio and microwave are included in the definition of communications media. Combinations of the above may also be included within the scope of computer-readable media.

[0078] Alternatively and / or additionally, the functions described herein may be performed at least in part by one or more hardware logic components. For example, but not limited to, illustrative types of hardware logic components that may be used include field programmable gate arrays ("FPGAs"), application specific integrated circuits ("ASICs"), application specific standard products ("ASSPs"), systems on chip ("SOCs"), complex programmable logic devices ("CPLDs"), etc.

[0079] What has been described above includes examples of one or more embodiments. Of course, it is not possible to describe every conceivable modification and alteration of the above apparatus or method for the purpose of describing the aforementioned aspects, but one of ordinary skill in the art will recognize that many further modifications and permutations of the various aspects are possible. Therefore, the described aspects are intended to cover all such changes, modifications and variations that fall within the scope of the appended claims.

Claims

1. A computer-implemented method for personalizing a speech enhancement component in a speech communication system without requiring registration, the method comprising: receiving audio data, the audio data comprising speech and the audio data to be processed by at least one speech enhancement component; determining, without requiring user registration, whether the speech of the audio data includes one or both of near-field speech and far-field speech; and Based on determining that the speech of the audio data includes one of near-field speech and far-field speech or both near-field speech and far-field speech, one or more speech enhancement components of the at least one speech enhancement component are changed.

2. The method according to claim 1, wherein: Changing the one or more speech enhancement components in the at least one speech enhancement component comprises: One or more of the at least one speech enhancement component is changed to one or more speech enhancement components that have been trained using near-field speech and far-field speech, wherein the near-field speech serves as clean speech and the far-field speech serves as interference.

3. The method according to claim 1, wherein: Changing one or more speech enhancement components of the at least one speech enhancement component comprises: When the speech of the audio data includes i) only near-field speech or ii) near-field speech and far-field speech, changing each of the one or more speech enhancement components in the at least one speech enhancement component to a corresponding personalized speech enhancement component; and When the speech of the audio data includes only far-field speech, each of the one or more speech enhancement components in the at least one speech enhancement component is changed to a corresponding speech enhancement component that does not remove the far-field speech.

4. The method according to claim 2, wherein: Each of the corresponding personalized speech enhancement components is a neural network model that has been trained using far-field speech.

5. The method according to claim 4, wherein: The personalized speech enhancement component using the trained neural network model is a personalized noise suppression component, which uses a data set with only near-field speech as clean speech and adds a data set with only far-field speech as interference to train the personalized noise suppression component neural network to perform noise suppression on far-field speech.

6. The method according to claim 1, wherein: Determining whether the speech of the audio data includes one of near-field speech and far-field speech or includes both near-field speech and far-field speech includes one or both of the following: i) determining whether the audio data was captured using a personalized device, and ii) using the trained neural network to determine whether the audio data includes near-field speech.

7. The method according to claim 6, further comprising: receiving the trained neural network, the neural network being trained to detect whether speech of the audio data is near-field speech or far-field speech, Wherein, determining whether the speech of the audio data includes one of near-field speech and far-field speech or includes both near-field speech and far-field speech includes: A determination is made using the trained neural network whether the audio data includes near-field speech.

8. The method according to claim 6, further comprising: receiving device information of a device that captures the audio data, Wherein, determining whether the speech of the audio data includes one of near-field speech and far-field speech or includes both near-field speech and far-field speech includes: A determination is made based on the received device information whether the audio data is captured using a personalized device.

9. The method according to claim 1, wherein: Determining whether the speech of the audio data includes one of near-field speech and far-field speech or both near-field speech and far-field speech includes: Whether the audio data includes near-field speech is determined using one or more of: i) a reverberation time 60 (RT60) metric, the RT60 metric being defined as a measure of the time it takes for the sound pressure level to decrease by 60 dB after speech in the audio data ceases, ii) a signal-to-noise ratio greater than 40 dB, and iii) a speech to reverberation modulation energy ratio (SRMR).

10. A system for personalizing a speech enhancement component in a speech communication system without registration, the system comprising: a data storage device storing instructions for personalizing a speech enhancement component in a speech communication system without requiring registration; as well as A processor configured to execute the instructions to implement a method, the method comprising: receiving audio data, the audio data comprising speech and the audio data to be processed by at least one speech enhancement component; determining, without requiring user registration, whether the speech of the audio data includes one or both of near-field speech and far-field speech; and Based on determining that the speech of the audio data includes one of near-field speech and far-field speech or both near-field speech and far-field speech, one or more speech enhancement components of the at least one speech enhancement component are changed.

11. The system according to claim 10, wherein: Changing the one or more speech enhancement components in the at least one speech enhancement component comprises: One or more of the at least one speech enhancement component is changed to one or more speech enhancement components that have been trained using near-field speech and far-field speech, wherein the near-field speech serves as clean speech and the far-field speech serves as interference.

12. The system according to claim 10, wherein: Changing one or more speech enhancement components of the at least one speech enhancement component comprises: When the speech of the audio data includes i) only near-field speech or ii) near-field speech and far-field speech, changing each of the one or more speech enhancement components in the at least one speech enhancement component to a corresponding personalized speech enhancement component; and When the speech of the audio data includes only far-field speech, each of the one or more speech enhancement components in the at least one speech enhancement component is changed to a corresponding speech enhancement component that does not remove the far-field speech.

13. The system according to claim 11, wherein: Each of the corresponding personalized speech enhancement components is a neural network model that has been trained using far-field speech.

14. A computer-readable storage device storing instructions that, when executed by a computer, cause the computer to implement a method for personalizing a speech enhancement component in a speech communication system without registration, the method comprising: receiving audio data, the audio data comprising speech and the audio data to be processed by at least one speech enhancement component; determining, without requiring user registration, whether the speech of the audio data includes one or both of near-field speech and far-field speech; and Based on determining that the speech of the audio data includes one of near-field speech and far-field speech or both near-field speech and far-field speech, one or more speech enhancement components of the at least one speech enhancement component are changed.

15. The computer readable storage device of claim 14, wherein: Changing the one or more speech enhancement components in the at least one speech enhancement component comprises: One or more of the at least one speech enhancement component is changed to one or more speech enhancement components that have been trained using near-field speech and far-field speech, wherein the near-field speech serves as clean speech and the far-field speech serves as interference.

Images