Speech recognition with language processing system combining machine learning and deep learning

Abstract

The application relates to the technical field of language processing, in particular to a language processing system for speech recognition combining machine learning and deep learning, which comprises an audio acquisition processing unit for acquiring an audio signal; a body timbre learning unit for continuously learning a normal timbre distribution of a user and constructing a mainstream timbre learning model based on a variational autoencoder combining a time sequence dynamic modeling technology, for constructing a main and auxiliary stream dynamic gating mechanism through a generative adversarial network combining a KL divergence detection technology, and for judging whether the current timbre deviates from the normal timbre distribution; a noise learning and storage unit for continuously learning wind noise features and KTV noise features; and a true-color speech recognition unit for screening user speech signals for language processing based on the mainstream timbre learning model and a noise feature library. The language processing system for speech recognition combining machine learning and deep learning locks and extracts the body user speech in a high-noise environment based on the mainstream timbre learning model and the noise feature library.

Description

Technical Field

[0001] This invention relates to the field of language processing technology, and more specifically, to a language processing system for speech recognition that combines machine learning and deep learning. Background Technology

[0002] The speech recognition language processing system combining machine learning and deep learning aims to accurately identify and continuously adapt to the user's individual timbre and improve the robustness of speech recognition in complex noise environments. Through long-term timbre modeling of the mainstream timbre learning model and the secondary timbre compensation mechanism, it controls recognition errors caused by short-term timbre changes due to user illness and fatigue or environmental noise interference, and realizes dynamic adaptation of the user's personalized timbre. This allows the system to accurately lock the original user's speech signal even in situations with multiple voice overlap, wind noise during cycling, noisy KTV environments, and changes in the user's timbre.

[0003] Existing speech recognition language processing systems often struggle to handle short-term changes in a user's voice, such as changes caused by a cold or illness. Furthermore, existing Deep Speaker Embeddings technology cannot perform online voice adaptation when the user is in a noisy environment or when their voice changes, leading to the system misidentifying the user's voice input as a different speaker and rejecting the user's voice input. Therefore, we design a speech recognition language processing system that combines machine learning and deep learning. Summary of the Invention

[0004] The purpose of this invention is to provide a speech processing system for speech recognition that combines machine learning and deep learning, in order to solve the problem mentioned in the background art that the existing Deep SpeakerEmbeddings technology cannot perform online timbre adaptation when in a noisy environment or when the user's timbre changes, which leads to the system misjudging the user's speech input as a different speaker and rejecting the user's speech input.

[0005] To achieve the above objectives, the present invention aims to provide a speech processing system for speech recognition that combines machine learning and deep learning, comprising:

[0006] An audio acquisition and processing unit is used to acquire audio signals, perform preprocessing, and extract audio features.

[0007] It also includes an ontological timbre learning unit, which continuously learns the normal timbre distribution of the user and constructs a mainstream timbre learning model based on variational autoencoder combined with temporal dynamic modeling technology. It constructs a main and secondary stream dynamic gating mechanism by combining generative adversarial network with KL divergence detection technology to dynamically determine whether the current timbre deviates from the normal timbre distribution.

[0008] It also includes a noise learning and storage unit, which is used to continuously learn wind noise characteristics and KTV noise characteristics and store them in a noise feature library;

[0009] It also includes an authentic speech recognition unit, which is based on mainstream timbre learning models and noise feature libraries. It is used to filter user speech signals in noisy environments and convert them into text language using end-to-end speech recognition technology.

[0010] As a further improvement to this technical solution, the audio acquisition and processing unit includes an audio acquisition module and an audio processing module;

[0011] The audio acquisition module uses sensors to collect user audio signals and ambient audio signals.

[0012] The audio processing module is used to preprocess user audio signals and ambient audio signals, and extract audio features from user audio signals and ambient audio signals.

[0013] As a further improvement to this technical solution, the main body timbre learning unit includes a mainstream timbre learning module and a secondary timbre monitoring module;

[0014] The mainstream timbre learning module is based on variational autoencoder combined with temporal dynamic modeling technology to construct a mainstream timbre learning model, which is used to continuously learn the normal timbre distribution of users.

[0015] The secondary stream timbre monitoring module constructs a dynamic gating mechanism for the main and secondary streams by combining generative adversarial networks with KL divergence detection technology. It dynamically determines whether the user's real-time timbre deviates from the normal timbre distribution and performs real-time compensation for abnormal timbre when the user's timbre is abnormal.

[0016] As a further improvement to this technical solution, the mainstream timbre learning module is based on variational autoencoder combined with temporal dynamic modeling technology to construct a mainstream timbre learning model. The specific method and steps are as follows:

[0017] S2.1.1 Extract user audio signals and user audio features from the audio acquisition and processing unit;

[0018] The user audio signal includes: time-domain user audio signal and frequency-domain user audio signal;

[0019] The user audio features include: user MFCC timbre features, user LPC cepstral coefficients, and user timbre temporal features;

[0020] S2.1.2 Construct user timbre feature vectors based on user audio features, and reconstruct user timbre distribution and optimize variational autoencoders through variational autoencoders;

[0021] S2.1.3 Based on the user's timbre distribution, use temporal dynamic modeling technology to track timbre changes and analyze the current timbre state;

[0022] S2.1.4. Based on the user's timbre distribution and the current timbre state, construct a mainstream timbre learning model.

[0023] As a further improvement to this technical solution, in S2.1.2, a user timbre feature vector is constructed based on user audio features, and the user timbre distribution is reconstructed and the variational autoencoder is optimized through variational autoencoder:

[0024] Input the user's vocal feature vector into the encoder:

[0025] x = [M] u ,L u ,T u ];

[0026]

[0027] Where x is the user's timbre feature vector; z is the user's timbre latent variable; M u For user MFCC tone characteristics; L u For user LPC cepstral coefficients; T u For user timbre timing characteristics; μ u The average timbre; Let be the timbre variance; q(z|x) be the variational posterior distribution. It follows a Gaussian distribution;

[0028] User vocal characteristics reconstructed by the decoder:

[0029]

[0030] Where p(x|z) is the reconstructed distribution of timbre features; f(z) is the mapping function of the decoder; Variance of timbre characteristics;

[0031] Optimize the variational autoencoder by calculating the reconstruction error:

[0032]

[0033] in, Let p(z) be the total loss function of the variational autoencoder; p(z) be the prior distribution; D KL (q(z|x)||p(z)) is the KL divergence between the variational posterior distribution q(z|x) and the prior distribution p(z); This represents the reconstruction error.

[0034] As a further improvement to this technical solution, in S2.1.3, based on the user's timbre distribution, temporal dynamic modeling technology is used to track timbre changes and analyze the current timbre state. The specific method is as follows:

[0035] Track tone changes using Kalman filters:

[0036] X t =AX t-1 +w t ;

[0037] Where t is the time step; X t X is the timbre state vector at time step t; A is the timbre transition matrix; X t-1 w is the timbre state vector at time step t-1; t The Gaussian noise term represents the uncertainty of the timbre state changing over time t, i.e., the model error term;

[0038] Analyze the current tone state using Gated LSTM:

[0039] h t =σ(W h X t +U h h t-1 +b h );

[0040] Among them, h t The timbre state at the current time step t; σ(·) is the Sigmoid gated activation function; W h Input a weight matrix for the timbre; U h h is the state weight matrix of the previous time step; t-1 For time step t-1, the timbre state is as follows; b h For the bias term of the Gated LSTM;

[0041] In S2.1.4, a mainstream timbre learning model is constructed based on the user's timbre distribution and the current timbre state. The specific method is as follows:

[0042] p(x normal |z t )=αp(x normal |z t-1 )+(1-α)p(x t |z t );

[0043] α=σ(W α h t +b α );

[0044] Where p(x) normal |zt p(x) represents the normal user timbre distribution at time step t; α is the dynamic smoothing factor; p(x) normal |z t-1 p(x) represents the normal user timbre distribution at time step t-1; t |z t ) represents the reconstructed distribution of timbre features at time step t; W α b is the weight matrix of the Gated LSTM; α This is a bias term.

[0045] As a further improvement to this technical solution, the secondary stream timbre monitoring module constructs a dynamic gating mechanism for the main and secondary streams by combining generative adversarial networks with KL divergence detection technology. This mechanism dynamically determines whether the user's real-time timbre deviates from the normal timbre distribution and performs real-time compensation for abnormal timbre when the user's timbre is abnormal. The specific steps are as follows:

[0046] Dynamic gating mechanism for main and secondary streams:

[0047] S2.2.1 Calculate the KL divergence of user timbre based on the user timbre features reconstructed by the decoder and the mainstream timbre learning model:

[0048]

[0049] Among them, D KL (p(x|z)||p(x normal |z)) represents the KL divergence of the user's timbre;

[0050] S2.2.2 Set a user timbre dispersion threshold ∈ to determine whether the user timbre is abnormal:

[0051] If D KL (p(x|z)||p(x normal If |z))≤∈, then the user's timbre is normal, and the mainstream timbre learning model will continue to be used;

[0052] If D KL (p(x|z)||p(x normal If |z))>∈, then timbre compensation is triggered, and S2.2.3 is executed to perform timbre compensation using a generative adversarial network;

[0053] S2.2.3. Use a generator from a generative adversarial network to generate a modified timbre, and use a discriminator to evaluate whether the modified timbre is reasonable. If it is reasonable, input the modified timbre into the mainstream timbre learning model.

[0054] As a further improvement to this technical solution, the noise learning and storage unit includes a noise feature learning module and a noise feature storage module;

[0055] The noise feature learning module is used to collect environmental noise features in real time and learn wind noise features and KTV background noise features.

[0056] The noise feature storage module is used to store and manage noise feature data, forming a noise feature library.

[0057] As a further improvement to this technical solution, the authentic speech recognition unit is based on mainstream timbre learning models and noise feature libraries. It is used to filter user speech signals in noisy environments and convert them into text using end-to-end speech recognition technology. The specific steps are as follows:

[0058] S4.1 Acquire a new audio signal and determine whether the user's audio signal in the new audio signal matches the user's own timbre. The specific method is as follows:

[0059] New audio signals are acquired from the audio acquisition and processing unit (1), the current timbre shift is calculated using KL divergence detection technology, and a conformity comparison is performed with the user's own timbre in the mainstream timbre learning model:

[0060] If the current timbre offset is greater than the user's timbre divergence threshold, then the user's timbre does not match and the new audio signal is rejected.

[0061] If the current timbre offset is less than or equal to the user's timbre dispersion threshold, then it matches the user's own timbre. The new audio signal is then input into the original timbre learning unit (2) for analysis and execution of S4.2.

[0062] S4.2 Based on the noise feature library, Wiener filtering technology is used to separate the noise signals present in the noise feature library from the new audio signal, retaining only the pure user audio signal;

[0063] S4.3 Based on the clean user audio signal, end-to-end speech recognition technology is used to convert the clean user audio signal into text language.

[0064] As a further improvement to this technical solution, in S4.1, the user audio signal is input to the body timbre learning unit, which is used to trigger the main and secondary stream dynamic gating mechanism to perform timbre compensation when a large deviation of the user timbre from the existing distribution is detected. It is also used to continuously and dynamically learn the user's normal timbre each time the user's voice is recognized.

[0065] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0066] 1. In this speech recognition language processing system that combines machine learning and deep learning, a mainstream timbre learning model is constructed by combining variational autoencoder with temporal dynamic modeling technology. This model continuously learns and updates the user's long-term timbre characteristics, establishes a personalized timbre distribution, and enables the system to adapt to the user's natural timbre changes over a long period of time.

[0067] 2. In this speech recognition language processing system that combines machine learning and deep learning, by constructing a noise feature library, the system can extract the original user's speech in environments with overlapping speech from multiple users or high noise levels, based on the noise feature library and mainstream timbre learning models. Attached Figure Description

[0068] Figure 1 This is an overall flowchart of the present invention;

[0069] The meanings of the labels in the diagram are as follows:

[0070] 1. Audio acquisition and processing unit; 11. Audio acquisition module; 12. Audio processing module; 2. Original timbre learning unit; 21. Mainstream timbre learning module; 22. Secondary timbre monitoring module; 3. Noise learning and storage unit; 31. Noise feature learning module; 32. Noise feature storage module; 4. Original speech recognition unit. Detailed Implementation

[0071] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0072] Please see Figure 1 As shown, a speech processing system for speech recognition that combines machine learning and deep learning is provided, including:

[0073] Audio acquisition and processing unit 1, which is used to acquire audio signals, perform preprocessing, and extract audio features;

[0074] In this embodiment, the audio acquisition and processing unit 1 includes an audio acquisition module 11 and an audio processing module 12;

[0075] The audio acquisition module 11 uses sensors to collect user audio signals and ambient audio signals.

[0076] The audio processing module 12 is used to preprocess the user audio signal and the ambient audio signal, and extract the audio features of the user audio signal and the ambient audio signal.

[0077] In this embodiment, a six-microphone ring array is used to collect audio signals from multiple directions; adaptive beamforming technology is used to locate and enhance the target user's speech, and the Generalized SidelobeCanceller algorithm is used for directional speech enhancement; the Time Difference of Arrival method is used to calculate the direction of the sound source, and the SRP-PHAT method is combined to improve the localization accuracy of multi-source speech and distinguish between near-field audio of user audio and far-field audio of environmental noise.

[0078] The user audio signal and the ambient audio signal are preprocessed using the following methods:

[0079] Noise removal is performed on user audio signals and ambient audio signals based on spectral subtraction and SEGANGAN-based Speech Enhancement. For user audio signals, Mel-frequency cepstral coefficients are extracted as audio features, and LPC cepstral coefficients are extracted to track the short-time dynamic characteristics of speech, aiding in subsequent timbre modeling. For ambient audio signals, MFCC, spectral entropy, and zero-crossing rate are used to identify noise types: wind noise typically has low-frequency characteristics; KTV noise has strong mid-to-high frequency components.

[0080] It also includes a timbre learning unit 2, which continuously learns the normal timbre distribution of the user and constructs a mainstream timbre learning model based on variational autoencoder combined with temporal dynamic modeling technology. It constructs a main and secondary stream dynamic gating mechanism by combining generative adversarial network with KL divergence detection technology to dynamically determine whether the current timbre deviates from the normal timbre distribution.

[0081] In this embodiment, the main body timbre learning unit 2 includes a mainstream timbre learning module 21 and a secondary stream timbre monitoring module 22;

[0082] The mainstream timbre learning module 21 is based on variational autoencoder combined with temporal dynamic modeling technology to construct a mainstream timbre learning model, which is used to continuously learn the normal timbre distribution of users.

[0083] In this embodiment, the variational autoencoder is a probabilistic generative model used to learn the latent distribution of data. It can map the timbre features of the input speech to a low-dimensional latent variable space and generate a personalized distribution of the user's timbre. Temporal dynamic modeling technology is used to track changes in the user's timbre over time, ensuring that the system can adapt to both short-term and long-term timbre changes. In traditional speech recognition systems, especially those using fixed Deep Speaker Embeddings, changes in user timbre, whether long-term or short-term, often lead to incorrect matching and prevent effective updates to the user's timbre model. This system tracks user timbre changes based on a variational autoencoder combined with temporal dynamic modeling technology, ensuring the stability and accuracy of speech recognition, and can dynamically learn user timbre online in real time. Specifically, even in the case of short-term timbre changes caused by illness such as a cold or sore throat, the variational autoencoder can still recognize the user because it learns the overall distribution of timbre, rather than a fixed embedding.

[0084] The secondary stream timbre monitoring module 22 constructs a dynamic gating mechanism for the main and secondary streams by combining generative adversarial networks with KL divergence detection technology. It dynamically judges whether the user's real-time timbre deviates from the normal timbre distribution and performs real-time compensation for abnormal timbre when the user's timbre is abnormal.

[0085] In this embodiment, Generative Adversarial Network (GAN) is a deep learning model based on game theory, consisting of a generator and a discriminator. The generator learns to generate realistic data samples from the input data, that is, to map the current abnormal timbre back to the normal timbre. The discriminator is responsible for distinguishing between the generated samples and the real samples to ensure that the generated timbre matches the user's normal timbre. The two train against each other, enabling the generator to generate more realistic samples. Overall, in this module, the GAN is used to compensate for timbre abnormalities, so that the user can still maintain normal speech recognition when sick, experiencing equipment changes, or when there is environmental noise interference.

[0086] KL divergence detection technology is a measure of the difference between two probability distributions, representing the information loss between a distribution and a reference distribution. It is used to determine whether the current user's timbre deviates significantly from the normal timbre distribution.

[0087] The main and secondary stream dynamic gating mechanism is used to coordinate the main stream timbre learning module and the secondary stream timbre monitoring module. When the KL divergence detection does not exceed the threshold and the timbre is normal, only the main stream timbre learning model is used for identification. When the KL divergence exceeds the threshold and the timbre is abnormal, the generative adversarial network timbre compensation is triggered to make the timbre return to the normal distribution.

[0088] By combining generative adversarial networks with KL divergence detection technology, a dynamic gating mechanism for main and secondary streams is constructed. This mechanism dynamically determines whether a user's real-time timbre deviates from the normal timbre distribution and compensates for abnormal timbre in real time. For example, a user's normal timbre is a clear, mid-frequency male voice. When the user has a cold, their voice becomes hoarse. The system calculates the current timbre distribution, and KL divergence detection finds that the timbre deviation is too large, triggering the timbre compensation mechanism. This prevents the system from rejecting the user due to short-term timbre changes, ensuring that the system can still correctly recognize the user's voice even when they have a cold. For example, in environments where multiple people are talking simultaneously, such as conferences, teleconferences, or karaoke, the system needs to distinguish between the original user and other users. KL divergence detection can determine whether the current voice belongs to the original user. If the KL divergence is abnormally large, it means that the timbre does not conform to the original user's long-term timbre distribution, and the input is rejected.

[0089] In this embodiment, the mainstream timbre learning module 21 constructs a mainstream timbre learning model based on variational autoencoder combined with temporal dynamic modeling technology. The specific method steps are as follows:

[0090] S2.1.1 Extract user audio signals and user audio features from audio acquisition and processing unit 1;

[0091] The user audio signal includes: time-domain user audio signal and frequency-domain user audio signal;

[0092] The user audio features include: user MFCC timbre features, user LPC cepstral coefficients, and user timbre temporal features;

[0093] S2.1.2 Construct user timbre feature vectors based on user audio features, and reconstruct user timbre distribution and optimize variational autoencoders through variational autoencoders;

[0094] S2.1.3 Based on the user's timbre distribution, use temporal dynamic modeling technology to track timbre changes and analyze the current timbre state;

[0095] S2.1.4. Based on the user's timbre distribution and the current timbre state, construct a mainstream timbre learning model.

[0096] In this embodiment S2.1.1, the user audio signal includes: a time-domain user audio signal and a frequency-domain user audio signal. This is the system's most basic sound input, used for subsequent extraction of user audio features. The frequency-domain user audio signal is derived from the time-domain user audio signal using short-time Fourier transform technology. The user audio features include: user MFCC timbre features, user LPC cepstral coefficients, and user timbre temporal features. Among these, the user MFCC timbre features are Mel-frequency cepstral coefficients, used for speech recognition and timbre analysis, reflecting the main auditory characteristics of the audio. In this system, the user MFCC timbre features are mainly used to characterize the user's timbre spectrum distribution; the user LPC cepstral coefficients are... PC cepstral coefficients are used to analyze the excitation source and vocal tract transfer function of speech, helping to track the short-term dynamic characteristics of speech. User LPC cepstral coefficients serve as auxiliary features, forming a more complete timbre representation together with MFCC timbre features. User timbre temporal features are used to represent the temporal variation pattern of user timbre. These features are obtained by modeling the statistical characteristics of audio frames (including changes in energy, pitch, zero-crossing rate, etc. over time). User timbre temporal features assist in VAE and subsequent temporal dynamic modeling. The user MFCC timbre features, user LPC cepstral coefficients, and user timbre temporal features are combined to form a user timbre feature vector, which is then input into the variational autoencoder. In this embodiment S2.1.2, a user timbre feature vector is constructed based on user audio features, and the user timbre distribution is reconstructed and optimized using a variational autoencoder.

[0097] Input the user's vocal feature vector into the encoder:

[0098] x = [M] u ,L u ,T u ];

[0099]

[0100] Where x is the user's timbre feature vector; z is the user's timbre latent variable; M u For user MFCC tone characteristics; L u For user LPC cepstral coefficients; T u For user timbre timing characteristics; μ u The average timbre; Let be the timbre variance; q(z|x) be the variational posterior distribution. It follows a Gaussian distribution;

[0101] User vocal characteristics reconstructed by the decoder:

[0102]

[0103] Where p(x|z) is the reconstructed distribution of timbre features; f(z) is the mapping function of the decoder; Variance of timbre characteristics;

[0104] Optimize the variational autoencoder by calculating the reconstruction error:

[0105]

[0106] in, Let p(z) be the total loss function of the variational autoencoder; p(z) be the prior distribution; D KL (q(z|x)||p(z)) is the KL divergence between the variational posterior distribution q(z|x) and the prior distribution p(z); This represents the reconstruction error.

[0107] In this embodiment, in S2.1.3, based on the user's timbre distribution, temporal dynamic modeling technology is used to track timbre changes and analyze the current timbre state. The specific method is as follows:

[0108] Track tone changes using Kalman filters:

[0109] X = AX t-1 +w t ;

[0110] Where t is the time step; X t Let X be the timbre state vector at time step t; A is the timbre transition matrix, representing the change relationship of the user's timbre from the previous time step to the current time step; X t-1 w is the timbre state vector at time step t-1; t The Gaussian noise term represents the uncertainty of the timbre state changing over time t, i.e., the model error term;

[0111] Analyze the current tone state using Gated LSTM:

[0112] h t =σ(W h Xt + U h h t-1 +b h );

[0113] Among them, h t The timbre state is at the current time step t; σ(·) is the Sigmoid-gated activation function, used to control the state update of the LSTM computation; W h Input a weight matrix for the timbre; U h h is the state weight matrix of the previous time step; t-1 For time step t-1, the timbre state is as follows; b h For the bias term of the Gated LSTM;

[0114] In this embodiment, in S2.1.4, a mainstream timbre learning model is constructed based on the user's timbre distribution and the current timbre state. The specific method is as follows:

[0115] p(x normal |z t )=αp(x normal |z t-1 )+(1-α)p(x t |z t );

[0116] α=σ(W α h t +b α );

[0117] Where p(x) normal |z t p(x) represents the normal user timbre distribution at time step t; α is the dynamic smoothing factor; p(x) normal |z t-1 p(x) represents the normal user timbre distribution at time step t-1; t |z t ) represents the reconstructed distribution of timbre features at time step t; W α b is the weight matrix of the Gated LSTM; α This is a bias term.

[0118] In this embodiment, the current timbre state h calculated by Gated LSTM is... t As a dynamic adjustment factor, it provides smooth control when updating the long-term timbre distribution of the mainstream timbre learning model; the α of the mainstream timbre learning model is not fixed, but is determined by the current timbre state h. t Dynamic weighting parameters for control: When the system detects small changes in timbre, such as normal fluctuations in the user's voice, the current timbre state h is adjusted. t Small changes keep α at a large value, and mainstream timbre models mainly follow past timbre distributions; when the system detects large timbre changes, such as long-term timbre change trends, the current timbre state h... t The changes are significant, causing α to decrease and making the mainstream timbre learning model more reliant on the current timbre distribution for iterative updates.

[0119] In this embodiment, the secondary stream timbre monitoring module 22 constructs a dynamic gating mechanism for the main and secondary streams by combining generative adversarial networks with KL divergence detection technology. This mechanism dynamically determines whether the user's real-time timbre deviates from the normal timbre distribution and performs real-time compensation for abnormal timbre when the user's timbre is abnormal. The specific steps are as follows:

[0120] S2.2.1 Calculate the KL divergence of user timbre based on the user timbre features reconstructed by the decoder and the mainstream timbre learning model:

[0121]

[0122] Among them, D KL (p(x|z)||p(x normal |z)) represents the KL divergence of the user's timbre;

[0123] S2.2.2 Set a user timbre dispersion threshold ∈ to determine whether the user timbre is abnormal:

[0124] If D KL (p(x|z)||p(x normal If |z))≤∈, then the user's timbre is normal, and the mainstream timbre learning model will continue to be used;

[0125] If D KL (p(x|z)||p(x normal If |z))>∈, then timbre compensation is triggered, and S2.2.3 is executed to perform timbre compensation using a generative adversarial network;

[0126] S2.2.3. Use a generator from a generative adversarial network to generate a modified timbre, and use a discriminator to evaluate whether the modified timbre is reasonable. If it is reasonable, input the modified timbre into the mainstream timbre learning model.

[0127] In this embodiment, a dynamic gating mechanism for the main and secondary streams is used to determine in real time whether the user's timbre deviates from the normal timbre distribution, and to perform real-time compensation when the timbre is abnormal.

[0128] The system dynamically monitors the user's timbre deviation to prevent speech recognition failure or misrecognition due to short-term timbre changes caused by illness; when the timbre deviates from the normal range, a timbre compensation mechanism is triggered to ensure that the speech recognition system can adapt to the user's personalized timbre and is not affected by short-term timbre changes; and it ensures that the mainstream timbre learning model can still operate stably under long-term timbre changes.

[0129] KL divergence detection can measure the information deviation between two probability distributions and is suitable for statistical modeling of timbre features. Generative adversarial networks are used to learn timbre mapping relationships for timbre compensation, which can generate features that are closer to normal timbre when timbre is abnormal.

[0130] It also includes a noise learning and storage unit 3, which is used to continuously learn wind noise characteristics and KTV noise characteristics and store them in a noise feature library;

[0131] In this embodiment, the noise learning and storage unit 3 includes a noise feature learning module 31 and a noise feature storage module 32;

[0132] The noise feature learning module 31 is used to collect environmental noise features in real time and learn wind noise features and KTV background noise features.

[0133] The noise feature storage module 32 is used to store and manage noise feature data to form a noise feature library.

[0134] It also includes a natural voice recognition unit 4, which is based on a mainstream timbre learning model and a noise feature library, and is used to filter user voice signals in noisy environments and use end-to-end voice recognition technology to convert them into text language.

[0135] In this embodiment, the natural voice recognition unit 4, based on mainstream timbre learning models and noise feature libraries, is used to filter user voice signals in noisy environments and convert them into text using end-to-end voice recognition technology. The specific steps are as follows:

[0136] S4.1 Acquire a new audio signal and determine whether the user's audio signal in the new audio signal matches the user's own timbre. The specific method is as follows:

[0137] New audio signals are acquired from audio acquisition and processing unit 1, the current timbre shift is calculated using KL divergence detection technology, and a comparison is made with the user's own timbre in the mainstream timbre learning model:

[0138] If the current timbre offset is greater than the user's timbre divergence threshold, then the user's timbre does not match and the new audio signal is rejected.

[0139] If the current timbre offset is less than or equal to the user's timbre dispersion threshold, then it matches the user's own timbre. The new audio signal is then input into the original timbre learning unit 2 for analysis and execution of S4.2.

[0140] In this embodiment, KL divergence detection technology is a detection method based on Kullback-Leibler divergence to measure the difference between two probability distributions. It is used to determine whether the current data distribution deviates significantly from the reference distribution. In this system, KL divergence detection is used to monitor the distance between the user's current new timbre distribution and the user's normal timbre distribution.

[0141] The user timbre dispersion threshold ∈ is the user timbre dispersion threshold ∈ set in S2.2.2, and is also used to determine whether a new audio signal matches the user's own timbre;

[0142] S4.2 Based on the noise feature library, Wiener filtering technology is used to separate the noise signals present in the noise feature library from the new audio signal, retaining only the pure user audio signal;

[0143] In this embodiment S4.2, Wiener filtering technology is an optimal linear filtering technology used to estimate the optimal value of the target signal under noise interference. It is widely used in speech enhancement, signal denoising and other fields. In this system, it is used to separate known types of noise from new audio signals based on the noise feature library, retaining only the pure user audio signal; and to selectively remove existing noise types based on the noise feature library.

[0144] S4.3 Based on the clean user audio signal, end-to-end speech recognition technology is used to convert the clean user audio signal into text language.

[0145] In this embodiment S4.3, end-to-end speech recognition technology is a technology that directly maps speech waveforms or spectral features to text output without relying on traditional staged modeling. Common end-to-end speech recognition technologies include: DeepSpeech based on CTC, Seq2Seq model based on attention mechanism, and end-to-end ASR based on Transformer.

[0146] In S4.1, the user's audio signal is input to the body timbre learning unit 2, which is used to trigger the main and secondary stream dynamic gating mechanism to perform timbre compensation when a large deviation of the user's timbre from the existing distribution is detected. It is also used to continuously and dynamically learn the user's normal timbre each time the user's voice is recognized.

[0147] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A speech processing system for speech recognition that combines machine learning and deep learning, characterized in that, include: Audio acquisition and processing unit (1), the audio acquisition and processing unit (1) is used to acquire audio signals, perform preprocessing and extract audio features; The ontological timbre learning unit (2) continuously learns the normal timbre distribution of users and constructs a mainstream timbre learning model based on variational autoencoder combined with temporal dynamic modeling technology. It constructs a main and secondary stream dynamic gating mechanism by combining generative adversarial network with KL divergence detection technology to dynamically determine whether the current timbre deviates from the normal timbre distribution. Noise learning storage unit (3), the noise learning storage unit (3) is used to continuously learn wind noise characteristics and KTV noise characteristics, and store them in the noise characteristic library; The natural voice recognition unit (4) is based on the mainstream timbre learning model and noise feature library, and is used to filter user voice signals in noisy environments and use end-to-end voice recognition technology to convert them into text language. The main body timbre learning unit (2) includes a main timbre learning module (21) and a secondary timbre monitoring module (22). Among them, the mainstream timbre learning module (21) is based on variational autoencoder combined with temporal dynamic modeling technology to construct a mainstream timbre learning model, which is used to continuously learn the normal timbre distribution of users; The mainstream timbre learning module (21) constructs a mainstream timbre learning model based on variational autoencoder combined with temporal dynamic modeling technology. The specific steps are as follows: S2.1.1 Extract user audio signals and user audio features from the audio acquisition and processing unit (1); The user audio signal includes: time-domain user audio signal and frequency-domain user audio signal; The user audio features include: user MFCC timbre features, user LPC cepstral coefficients, and user timbre temporal features; S2.1.2 Construct user timbre feature vectors based on user audio features, and reconstruct user timbre distribution and optimize variational autoencoders through variational autoencoders; S2.1.3 Based on the user's timbre distribution, use temporal dynamic modeling technology to track timbre changes and analyze the current timbre state; S2.1.

4. Based on the user's timbre distribution and the current timbre state, construct a mainstream timbre learning model; In step S2.1.2, a user timbre feature vector is constructed based on user audio features, and the user timbre distribution is reconstructed and optimized using a variational autoencoder: Input the user's vocal feature vector into the encoder: ； ； in, For user timbre feature vectors; For potential variables of user timbre; For user MFCC timbre characteristics; For user LPC cepstral coefficients; For user timbre timing characteristics; The average timbre; For timbre variance; It is a variational posterior distribution; It follows a Gaussian distribution; User vocal characteristics reconstructed by the decoder: ； in, The reconstructed distribution of timbre features; For the decoder's mapping function; Variance of timbre characteristics; Optimize the variational autoencoder by calculating the reconstruction error: ； in, The total loss function of the variational autoencoder; It is the prior distribution; For variational posterior distribution and prior distribution KL divergence; This represents the reconstruction error; In S2.1.3, based on the user's timbre distribution, temporal dynamic modeling technology is used to track timbre changes and analyze the current timbre state. The specific method is as follows: Track tone changes using Kalman filters: ;in, For time steps; For time step The timbre state vector; This is the timbre transition matrix; For time step The timbre state vector; The Gaussian noise term represents the timbre state over time. The uncertainty of change, i.e., the model error term; using Gated LSTM to analyze the current timbre state: ; in, For the current time step Tone characteristics; It is a Sigmoid-gated activation function; Input a weight matrix for the timbre; This is the state weight matrix of the previous time step; For time step Tone characteristics; For the bias term of the Gated LSTM; In S2.1.4, a mainstream timbre learning model is constructed based on the user's timbre distribution and the current timbre state. The specific method is as follows: ； ； in, For time step Normal user tone distribution; It is a dynamic smoothing factor; For time step Normal user tone distribution; For time step The reconstructed distribution of timbre features; The weight matrix of the Gated LSTM; This is a bias term.

2. The speech processing system for speech recognition combining machine learning and deep learning according to claim 1, characterized in that: The audio acquisition and processing unit (1) includes an audio acquisition module (11) and an audio processing module (12). The audio acquisition module (11) uses sensors to acquire user audio signals and environmental audio signals; The audio processing module (12) is used to preprocess the user audio signal and the ambient audio signal, and extract the audio features of the user audio signal and the ambient audio signal.

3. The speech processing system for speech recognition combining machine learning and deep learning according to claim 2, characterized in that: The secondary stream timbre monitoring module (22) constructs a dynamic gating mechanism for the main and secondary streams by combining generative adversarial networks with KL divergence detection technology, dynamically judges whether the user's real-time timbre deviates from the normal timbre distribution, and performs real-time compensation for abnormal timbre when the user's timbre is abnormal.

4. The speech processing system for speech recognition combining machine learning and deep learning according to claim 3, characterized in that: The secondary stream timbre monitoring module (22) constructs a dynamic gating mechanism for the main and secondary streams by combining generative adversarial networks with KL divergence detection technology. It dynamically determines whether the user's real-time timbre deviates from the normal timbre distribution and performs real-time compensation for abnormal timbre when the user's timbre is abnormal. The specific steps are as follows: S2.2.1 Calculate the KL divergence of user timbre based on the user timbre features reconstructed by the decoder and the mainstream timbre learning model: ； in, Set the user timbre KL divergence; S2.2.2, Set the user timbre divergence threshold. Determine if the user's voice is abnormal: If If the user's timbre is normal, continue using the mainstream timbre learning model; if If this is triggered, timbre compensation will be performed, and S2.2.3 will be executed to use a generative adversarial network for timbre compensation. S2.2.

3. Use a generator from a generative adversarial network to generate a modified timbre, and use a discriminator to evaluate whether the modified timbre is reasonable. If it is reasonable, input the modified timbre into the mainstream timbre learning model.

5. The speech processing system for speech recognition combining machine learning and deep learning according to claim 4, characterized in that: The noise learning storage unit (3) includes a noise feature learning module (31) and a noise feature storage module (32). The noise feature learning module (31) is used to collect environmental noise features in real time and learn wind noise features and KTV background noise features. The noise feature storage module (32) is used to store and manage noise feature data to form a noise feature library.

6. The speech processing system for speech recognition combining machine learning and deep learning according to claim 5, characterized in that: The native speech recognition unit (4) is based on a mainstream timbre learning model and a noise feature library. It is used to filter user speech signals in noisy environments and convert them into text language using end-to-end speech recognition technology. The specific steps are as follows: S4.1 Acquire a new audio signal and determine whether the user's audio signal in the new audio signal matches the user's own timbre. The specific method is as follows: New audio signals are acquired from the audio acquisition and processing unit (1), the current timbre shift is calculated using KL divergence detection technology, and a conformity comparison is performed with the user's own timbre in the mainstream timbre learning model: If the current timbre offset is greater than the user's timbre dispersion threshold If the input of a new audio signal does not match the user's own voice, then the user will refuse to input the new audio signal. If the current timbre offset is less than or equal to the user's timbre dispersion threshold If the new audio signal matches the user's own timbre, it will be input into the timbre learning unit (2) for analysis and execution of S4.

2. S4.2 Based on the noise feature library, Wiener filtering technology is used to separate the noise signals present in the noise feature library from the new audio signal, retaining only the pure user audio signal; S4.3 Based on the clean user audio signal, end-to-end speech recognition technology is used to convert the clean user audio signal into text language.

7. The speech processing system for speech recognition combining machine learning and deep learning according to claim 6, characterized in that: In S4.1, the user's audio signal is input to the body timbre learning unit (2), which is used to trigger the main and secondary stream dynamic gating mechanism to perform timbre compensation when the user's timbre deviates significantly from the existing distribution. It is also used to continuously and dynamically learn the user's normal timbre when the user's voice is recognized each time.

Images