Speech recognition method and device based on environmental noise enhancement and medium

Abstract

The invention discloses a speech recognition method and device based on environmental noise enhancement and a medium, and the method comprises the steps: carrying out the framing processing of a to-be-recognized target speech signal, and obtaining at least one frame of speech signal; determining respective first power spectrums of the at least one frame of voice signal, and further determining respective Mel spectrums of the at least one frame of voice signal; determining respective noise feature vectors of the at least one frame of voice signal based on a noise perception model; and target model parameters of the speech enhancement model are determined based on the parameter generation model, so that the model parameters of the speech enhancement model are adaptively generated according to the to-be-recognized target speech signal. Then, inputting the respective first power spectrum of the at least one frame of voice signal into a voice enhancement model, and determining the respective second power spectrum of the at least one frame of voice signal based on the voice enhancement model; and finally, determining a target text sequence corresponding to the target voice signal based on the voice recognition model. Therefore, the accuracy of the determined target text sequence is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech recognition, and in particular to a speech recognition method, device and medium based on environmental noise enhancement. BACKGROUND

[0002] Currently, the scheme of combining speech enhancement as a preprocessing module with an end-to-end speech recognition system such as FunASR, etc. has become the mainstream technical path to improve recognition performance in noisy environments. Figure 1 For a speech recognition process in the related art, as shown in Figure 1 , a noisy speech input is obtained using a general speech enhancement module in the preprocessing stage, and a final recognition text is obtained through a FunASR recognition engine in the recognition stage.

[0003] The related art generally uses a general, non-adaptive deep neural network speech enhancement model. The enhancement module uses large-scale synthetic noisy speech data for training in the training stage, and learns a static mapping function from noisy speech to "pure" speech. In the inference stage, regardless of the specific environment (such as a factory or a car) from which the input audio comes, the enhancement module uses the same set of fixed model parameters for processing.

[0004] The technical problem existing in the related art is that the parameters of the pre-trained general enhancement model are static and cannot perceive and adapt to the real-time changing noise characteristics in the input audio. When encountering rapidly changing non-stationary noise that is not fully covered in the training set, the enhancement effect will decrease sharply, and even new distortions will be introduced. That is, the related art model is rigid and lacks adaptability, and the accuracy of speech recognition is poor. SUMMARY

[0005] The present application provides a speech recognition method, device and medium based on environmental noise enhancement to solve the problem of a related art model being rigid and lacking adaptability, and poor accuracy of speech recognition.

[0006] In a first aspect, the present application provides a speech recognition method based on environmental noise enhancement, comprising: obtaining a target speech signal to be recognized, performing frame processing on the target speech signal to obtain at least one frame of speech signal; determining a first power spectrum of each of the at least one frame of speech signal; determining a mel spectrum of each of the at least one frame of speech signal according to the first power spectrum of each of the at least one frame of speech signal; inputting the mel spectrum of each of the at least one frame of speech signal into a noise perception model, and determining a noise feature vector of each of the at least one frame of speech signal based on the noise perception model; inputting the noise feature vector of each of the at least one frame of speech signal into a parameter generation model, determining a target model parameter based on the parameter generation model, and updating a model parameter of a speech enhancement model using the target model parameter. inputting the first power spectrum of each of the at least one frame of speech signal into the speech enhancement model, determining a second power spectrum of each of the at least one frame of speech signal based on the speech enhancement model; inputting the second power spectrum of each of the at least one frame of speech signal into a speech recognition model, and determining a target text sequence corresponding to the target speech signal based on the speech recognition model.

[0007] The above technical solution has the following advantages or beneficial effects: The present application considers that the parameters of the pre-trained general enhancement model in the related art are static and cannot perceive and adapt to the real-time changes in the noise characteristics of the input audio, the model is rigid and lacks adaptability, and the accuracy of speech recognition is poor. The technical solution of adaptively generating a model parameter of a speech enhancement model according to a target speech signal to be recognized is proposed, thereby solving the problem of model rigidity in the related art and improving the adaptability and accuracy of speech recognition. Specifically, the target speech signal to be recognized is frame-processed to obtain at least one frame of speech signal; the first power spectrum of each of the at least one frame of speech signal is determined, and the mel spectrum of each of the at least one frame of speech signal is further determined; the noise feature vector of each of the at least one frame of speech signal is determined based on the noise perception model; and the target model parameter of the speech enhancement model is further determined based on the parameter generation model, thereby adaptively generating the model parameter of the speech enhancement model according to the target speech signal to be recognized. Then, the first power spectrum of each of the at least one frame of speech signal is input into the speech enhancement model, the second power spectrum of each of the at least one frame of speech signal is determined based on the speech enhancement model; and finally, the target text sequence corresponding to the target speech signal is determined based on the speech recognition model. Thus, the accuracy of the determined target text sequence is improved.

[0008] In a second aspect, the present application provides a speech recognition device based on environmental noise enhancement, the device comprising: determining module, configured to acquire a target speech signal to be recognized, perform frame processing on the target speech signal to obtain at least one frame of speech signal, determine a first power spectrum of each of the at least one frame of speech signal, determine a mel spectrum of each of the at least one frame of speech signal according to the first power spectrum of each of the at least one frame of speech signal, input the mel spectrum of each of the at least one frame of speech signal into a noise perception model, determine a noise feature vector of each of the at least one frame of speech signal based on the noise perception model, input the noise feature vector of each of the at least one frame of speech signal into a parameter generation model, determine a target model parameter based on the parameter generation model, and update a model parameter of a speech enhancement model by using the target model parameter; recognizing module, configured to input the first power spectrum of each of the at least one frame of speech signal into the speech enhancement model, determine a second power spectrum of each of the at least one frame of speech signal based on the speech enhancement model, and input the second power spectrum of each of the at least one frame of speech signal into a speech recognition model to determine a target text sequence corresponding to the target speech signal based on the speech recognition model.

[0009] In a third aspect, the present application provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus; The memory is configured to store a computer program. The processor is configured to execute the program stored on the memory to implement the method.

[0010] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method.

[0011] In a fifth aspect, the present application provides a computer program product, comprising an executable program, and the executable program is executed by a processor to implement the method. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0013] Figure 1 a speech recognition process in the related art is shown; Figure 2 a speech recognition process based on environmental noise enhancement provided by the present application is shown; Figure 3 A schematic diagram for calculating the real part and the imaginary part of the rotation factor provided in the present application; Figure 4 A schematic diagram for calculating the real part product and the imaginary part product provided in the present application; Figure 5 A schematic diagram for mapping linear frequency to mel frequency provided in the present application; Figure 6 A schematic diagram for the process of generating a two-dimensional feature matrix from feature mapping provided in the present application; Figure 7 A schematic diagram for the process of determining the enhanced second power spectrum provided in the present application; Figure 8 A schematic diagram for the training process of the noise perception model and the parameter generation model provided in the present application; Figure 9 A schematic diagram for the process of updating the parameters of the noise perception model and the parameter generation model provided in the present application; Figure 10 A schematic diagram for the overall flowchart of speech recognition based on ambient noise enhancement provided in the present application; Figure 11 A schematic diagram for the structure of the speech recognition device based on ambient noise enhancement provided in the present application; Figure 12 A schematic diagram for the structure of the electronic device provided in the present application. DETAILED DESCRIPTION

[0014] In order to make the purpose and implementation of the present application more clear, the following will combine the drawings in the exemplary embodiments of the present application to clearly and completely describe the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only a part of the embodiments of the present application, but not all the embodiments.

[0015] It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the following described embodiments, and is not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and general meanings.

[0016] The terms "first", "second", "third" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit the specific order or sequence, unless otherwise specified. It should be understood that the terms used in this way can be interchanged under appropriate circumstances.

[0017] The terms "comprises", "comprising", "includes", "including", "has", "having" and their conjugates, mean "including but not limited to", and are used to indicate that the list of elements or the list of components that it follows are not exhaustive and that other elements or components can be included or be utilized.

[0018] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software codes that can perform the function related to the element.

[0019] It should be noted that the above-mentioned embodiments are only used to illustrate the technical solutions of the present application, but not limit the technical solutions of the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still make modifications to the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some or all of the technical features; and the modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0020] For the convenience of explanation, the above description has been made in combination with specific embodiments. However, the above exemplary discussion is not intended to exhaust or limit the embodiments to the specific forms disclosed above. Various modifications and variations can be derived according to the above teachings. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different modified embodiments suitable for specific use considerations.

[0021] Figure 1 For the voice recognition process in the related art, as shown in Figure 1 , noisy speech input, in the preprocessing stage, a general speech enhancement module is used to obtain enhanced speech, in the recognition stage, through the FunASR recognition engine, the final recognition text is obtained.

[0022] As shown in Figure 1 , in the related art, a general, non-adaptive deep neural network speech enhancement model (such as a model based on convolutional neural network CNN or long short-term memory network LSTM) is usually used. The enhancement module uses large-scale synthetic noisy speech data in the training stage to learn a static mapping function from noisy speech to "pure" speech. In the inference stage, no matter what specific environment (such as a factory, a car) the input audio comes from, the enhancement module uses the same set of fixed model parameters for processing.

[0023] The core technology is to estimate a gain mask M(t, f) : M(t, f) = G( ); wherein is the value of the noisy speech spectrum or mel-spectrum at time frame t and frequency bin f, the enhanced spectrum is obtained by the following equation: = M(t, f) · ; Then is sent to the acoustic feature extractor of FunASR, and finally recognized by the Paraformer model and the like.

[0024] In the multi-person social activity scenarios such as meetings, speeches, conferences, and the like, the noise environment is complex and non-stationary. The noise sources mainly come from human activities themselves, including: Sudden impulse noise: such as applause, cheers, coughing, and moving of tables and chairs.

[0025] Non-target human voice interference: such as whispering of the audience, and “babble noise” (mixed speech) generated by collective discussion.

[0026] Double-talk phenomenon: the speech of the main speaker overlaps with the speech of the questioner or interlocutor.

[0027] Background music and broadcast: background music and broadcast sound interspersed in the conference activities.

[0028] The general speech enhancement model of the related art has more obvious defects when facing these noises: 1) Difficulty in distinguishing target speech and interference human voice: since the noise itself is also speech, the spectral characteristics are similar, and the static model is prone to false suppression (weakening the target speaker's speech) or false retention (retaining interference chatting voice).

[0029] 2) Slow response to sudden noise: the model parameters are fixed, and cannot respond quickly and accurately to the sudden burst of applause, often resulting in a small segment of speech after the applause being suppressed.

[0030] 3) Destroying the naturalness of speech: excessive processing to suppress noise makes the retained speech sound dull and mechanical, with decreased intelligibility and naturalness, although the noise is reduced, but the user experience is actually worse.

[0031] The fundamental technical defects of the technical solutions of the related art are: insufficient response to non-stationary noise, leading to decreased generalization performance in real complex scenarios, and high word error rate (Word Error Rate, WER) of speech recognition. The specific reasons are as follows: Model rigidity and lack of adaptability: The parameters of pre-trained general-purpose enhancement models are static and cannot perceive and adapt to the real-time changes in noise characteristics in the input audio. When encountering rapidly changing non-stationary noise (such as sudden metal impact sounds) that is not adequately covered in the training set, the enhancement effect will decrease sharply, and even introduce new distortions.

[0032] Training and inference scene mismatch: Despite the large scale of training data, it is impossible to exhaust all possible noise types and signal-to-noise ratio combinations. When the inference environment is inconsistent with the training data distribution, the performance degrades severely.

[0033] Distortion propagation: Inappropriate enhancement distorts the spectral structure of the original speech, especially the phase information and formant structure of the speech, and this distortion is propagated to the FunASR model in the back end, which interferes with the model's judgment based on a large amount of clean or stationary noise data, and may even increase recognition errors.

[0034] The direct consequence of the above defects is that the accuracy and reliability of the speech recognition system cannot meet the commercial requirements in high-value conference, speech, annual meeting, production workshop, vehicle-mounted, etc. scenarios, limiting its large-scale deployment, causing waste of development cost and decline of user experience.

[0035] In order to solve the above technical problems existing in the related art, the present application provides a speech recognition method based on environmental noise enhancement, which relates to the speech recognition technology field of the Mel-Frequency Cepstral Coefficients (MCC) project of the intelligent creation platform.

[0036] First, the key terms involved in the present application are explained as follows: FunASR: Chinese full name "Fundamental End-to-end Automatic Speech Recognition", English full name "Fundamental End-to-end Automatic Speech Recognition". A unified model framework that integrates speech endpoint detection, speech recognition, punctuation, and other functions. FunASR is based on deep learning technology and integrates multiple advanced models (such as Paraformer, FSMN-VAD, and CT-Transformer), supporting speech recognition (ASR), speech activity detection (VAD), punctuation recovery (PUNC), and other full-process functions. Its main features include: High recognition accuracy: The model is trained on a large-scale industrial corpus to ensure accurate and reliable recognition results.

[0037] Flexible customization: Supports user-defined hot words (keywords) to optimize the recognition effect of specific terms or proper nouns.

[0038] High Performance: Supports real-time processing and offline batch transcription, suitable for various scenarios such as meeting recording and interview transcription.

[0039] Paraformer: A core autoregressive end-to-end speech recognition model in the FunASR framework, achieving efficient inference through single-pass decoding and supporting timestamp prediction and hotword customization.

[0040] Non-stationary noise: Noise whose statistical properties (such as mean, variance, and spectrum) change over time. Unlike stationary noise, its energy distribution and waveform structure are time-varying, commonly found in sudden traffic noise, crowd conversations, or mechanical start-stop sounds. Due to its dynamic characteristics, it is difficult to eliminate using traditional filters and requires adaptive algorithms (such as spectral subtraction or deep learning) for real-time suppression.

[0041] Speech Enhancement (SE): A technique for extracting or enhancing the target speech signal from a noisy speech signal and suppressing noise interference.

[0042] Wiener filter: A classic speech enhancement algorithm based on the minimum mean square error criterion, which estimates pure speech by suppressing noise in the frequency domain.

[0043] STFT: Short-Time Fourier Transform, a transformation method that converts time-domain signals into a joint frequency and time domain, commonly used for time-frequency analysis of speech signals.

[0044] SNR: Signal-to-Noise Ratio, the ratio of the power of the target signal to the power of the noise, used to measure the degree of noise interference in the signal, with units of decibels (dB).

[0045] VAD: Voice Activity Detection, a technology used to detect whether there is valid speech in a speech signal, which can distinguish between speech segments and non-speech segments

[0046] Deep learning: An artificial intelligence technology based on deep neural networks.

[0047] Spectrum: The energy distribution of a speech signal in the frequency domain.

[0048] Mel-spectrogram: A frequency spectrum feature based on the auditory characteristics of the human ear (Mel scale), commonly used as the front-end feature in deep learning speech recognition systems.

[0049] Timestamp predictor: Integrated into the Paraformer model, it can achieve accurate timestamp alignment without the need for additional hybrid models.

[0050] Hotword Customization: Allow users to define specific keywords (hotwords) to improve the recall and accuracy of key terms through hotword embedder and multi-head attention mechanism.

[0051] Voice Activity Detection (VAD): Based on FSMN structure, effectively detect the start and end of speech, filter invalid sound, and improve recognition efficiency.

[0052] Text Post-processing: Use CT-Transformer model to add punctuation and remove speech incoherence in real time, improve the readability of transcribed text.

[0053] CNN: Convolutional Neural Network, a kind of feedforward deep neural network, through "convolution kernel" sliding on input feature map, extracting local spatial-spectral correlation structure; weight sharing greatly reduces parameter quantity. In speech enhancement, CNN is good at capturing short-term, local time-frequency patterns (such as harmonics, formants), often used for local feature modeling of spectral graph or mel spectrum.

[0054] LSTM: Long Short-Term Memory, a kind of recurrent neural network with gating mechanism, uses input gate, forget gate and output gate to dynamically control information flow, which can effectively model long-term temporal dependence. In speech enhancement task, LSTM is suitable for processing dynamic features that change continuously between frames, capturing long-term context relationship of speech, and improving the suppression ability of non-stationary noise.

[0055] Speech Recognition Error Rate: WER, Word Error Rate, is a core indicator to measure the difference between automatic speech recognition (ASR) system output and reference text, defined as: WER = ; Where S (Substitutions): the number of words replaced by error; D (Deletions): the number of words missed by recognition; I (Insertions): the number of words recognized more; N: total number of words in the reference text.

[0056] Features: The lower the value, the higher the recognition accuracy (ideal is 0%).

[0057] Obtained by dynamic programming (such as Levenshtein algorithm) to calculate the minimum edit distance.

[0058] Applicable to languages ​​such as English that are word-based; replaces common Chinese character error rates (CER).

[0059] Spectral Leakence: In Discrete Fourier Transform (DFT / FFT), signal truncation (finite-length window) causes discontinuities in the spectrum at the window boundaries. Energy "leaks" from the true frequency to adjacent frequency components, causing spectral peaks to broaden, amplitudes to decrease, and sidelobes to rise, thereby reducing frequency resolution and amplitude estimation accuracy.

[0060] Window function: In digital signal processing, a window function is a weighting function w(n) applied to a finitely long signal by truncating it into segments. It reduces spectral leakage caused by truncation by gradually attenuating the samples at both ends. Commonly used windows include rectangular windows, Hamming windows, and Hanning windows.

[0061] FFT points: i.e. , refers to the number of discrete sampling points taken when performing one Fast Fourier Transform. It must be an integer power of 2 (such as 512, 1024, etc.), and determines the resolution and computational cost of frequency domain analysis. It is a fundamental parameter for processing speech signals such as time-frequency conversion (e.g., STFT), noise estimation, and spectrum enhancement. The choice of [specific parameter] directly affects the accuracy of noise spectrum estimation and the real-time performance of the algorithm.

[0062] U-Net structure: A symmetrical convolutional network shaped like the letter "U".

[0063] Left half: Layer-by-layer downsampling (convolution + pooling) extracts high-level features.

[0064] Bottom: The most abstract representation of features.

[0065] Right half: Layer-by-layer upsampling (transposed convolution / interpolation) restores spatial resolution.

[0066] Skip connections: stitch downsampled features at the same scale to upsampled layers, fusing details and semantics to improve the accuracy of edge and fine-grained reconstruction.

[0067] Figure 2 A schematic diagram of the speech recognition process based on environmental noise enhancement provided in this application, the process including the following steps: S101: Obtain a target speech signal to be recognized, perform frame processing on the target speech signal to obtain at least one frame of speech signal, determine a first power spectrum of each of the at least one frame of speech signal, determine a mel spectrum of each of the at least one frame of speech signal according to the first power spectrum of each of the at least one frame of speech signal, input the mel spectrum of each of the at least one frame of speech signal into a noise perception model, determine a noise feature vector of each of the at least one frame of speech signal based on the noise perception model, input the noise feature vector of each of the at least one frame of speech signal into a parameter generation model, determine a target model parameter based on the parameter generation model, and update a model parameter of a speech enhancement model using the target model parameter; S102: Input the first power spectrum of each of the at least one frame of speech signal into the speech enhancement model, determine a second power spectrum of each of the at least one frame of speech signal based on the speech enhancement model, input the second power spectrum of each of the at least one frame of speech signal into a speech recognition model, and determine a target text sequence corresponding to the target speech signal based on the speech recognition model.

[0068] The speech recognition method based on environmental noise enhancement provided in the application is applied to an electronic device, which can be a personal computer (PC), a smart terminal, a computer, a server, or the like.

[0069] In the application, the speech signal to be recognized obtained by the electronic device is referred to as a target speech signal, and the target speech signal is an original noisy speech signal.

[0070] Input: original noisy speech signal ; wherein n = 0, 1, 2, 3,..., N-1 is a discrete time index representing the nth sampling point. N is the total number of sampling points of the speech signal; represents the amplitude value of the speech signal at the sampling point n, which is a discrete, real-valued time-domain waveform. The signal is usually collected by a microphone and digitized at a certain sampling rate (e.g., 16 kHz). This means that the continuous time signal is sampled once every = second.

[0071] The following is an example: A speech signal with a duration of 1 second and a sampling rate of 16 kHz, wherein N = 16000, n ranges from 0 to 15999, represents the amplitude at the 100th sampling point.

[0072] After the electronic device obtains the target voice signal, the target voice signal is first subjected to frame processing to obtain at least one frame of voice signal. Optionally, relevant frame parameters can be set in advance, and the target voice signal is subjected to frame processing based on the frame parameters. The frame parameters include frame length, frame shift, total frame number and the like. The frame length refers to the number of sampling points contained in one frame of voice signal; the frame shift refers to the number of sampling points between the starting points of adjacent two frames of voice signal; and the total frame number refers to the total number of frames of voice signal obtained through frame processing.

[0073] After the electronic device determines at least one frame of voice signal, the first power spectrum of the frame of voice signal is determined for the at least one frame of voice signal. Optionally, the frame of voice signal is first subjected to Fourier transform to obtain the frequency domain signal value of the frame of voice signal. The frequency domain signal value is a complex number including real part and imaginary part; and the first power spectrum of the frame of voice signal is obtained by summing the square of the real part and the square of the imaginary part of the frequency domain signal value of the frame of voice signal.

[0074] Further, after the electronic device determines the first power spectrum of the frame of voice signal for the at least one frame of voice signal, the mel spectrum of the frame of voice signal is determined according to the first power spectrum of the frame of voice signal. Optionally, the product of the first power spectrum of the frame of voice signal and the transfer function value of the mel filter set in advance in the electronic device is determined as the mel spectrum of the frame of voice signal.

[0075] The present application receives an original target voice signal with noise , which is subjected to frame processing and then subjected to short-time Fourier transform (STFT) to obtain a complex spectrum , and the power spectrum and the mel spectrum thereof are calculated. The purpose is to convert the original time-domain voice waveform signal into a frequency-domain representation form which can not only preserve key voice information but also facilitate neural network processing.

[0076] The noise perception model and the parameter generation model pre-trained in the electronic device are deployed. The noise perception model is used to extract features from the mel spectrum of each of the at least one frame of voice signal to obtain a noise feature vector of each of the at least one frame of voice signal; and the parameter generation model is used to extract features from the noise feature vector of each of the at least one frame of voice signal to adaptively obtain target model parameters of the voice enhancement model. Optionally, the noise perception model and the parameter generation model can be trained by the electronic device, and saved after training; or the noise perception model and the parameter generation model can be trained by other electronic devices, and then deployed to the electronic device for voice recognition.

[0077] The speech enhancement model includes default model parameters. After determining the target model parameters of the speech enhancement model based on the parameter generation model, the target model parameters are used to update the speech enhancement model's parameters. After the update, the speech enhancement model includes the target model parameters. Then, the first power spectrum of at least one frame of speech signal is input into the speech enhancement model containing the target model parameters. Based on the speech enhancement model, the first power spectrum of each of the at least one frame of speech signal is enhanced to determine the enhanced second power spectrum of each of the at least one frame of speech signal. Finally, the second power spectrum of each of the at least one frame of speech signal is input into the speech recognition model, and the target text sequence corresponding to the target speech signal is determined based on the speech recognition model. This completes the speech recognition process. During the training of the speech recognition model, the power spectrum of each frame of speech signal is used as input, and the corresponding text sequence is used as output.

[0078] In one optional implementation, during speech recognition model training, the Mel spectrum of each frame of speech signal can be used as input, and the corresponding text sequence can be used as output. In this case, the first power spectrum of each of at least one frame of speech signal is input into the speech enhancement model. After determining the second power spectrum of each of the at least one frame of speech signal based on the speech enhancement model, the second Mel spectrum of each of the at least one frame of speech signal can be determined based on the second power spectrum of each of the at least one frame of speech signal. Then, the second Mel spectrum of each of the at least one frame of speech signal is input into the speech recognition model, and the target text sequence corresponding to the target speech signal is determined based on the speech recognition model.

[0079] Optionally, the noise-aware model can be a lightweight network model consisting of three CNN layers, which uses the Mel spectrum. Input a noise perception model, output a noise feature vector : = N( ; ) ; in, It is the Mel spectrum. represents the weight parameters of the noise-aware model (such as neural network weights, statistical model coefficients), and N is the implicit function of the noise-aware model.

[0080] The parameter generation model includes a fully connected layer, which... Input parameters generate model H, which in turn generates the convolutional kernel weights of the last layer of the speech enhancement model G. : = .

[0081] where, is the hidden variable weight of the parameter generation model H, H is a generative hidden mapping function, which can be a deep generative network, and the result is the reconstruction weight of the speech enhancement model.

[0082] It should be noted that z(l,m) is the output parameter of the noise perception network: it is the noise feature vector processed by the lightweight noise perception model on the input mel spectrum M(l,m), representing the noise characteristics (such as noise spectrum, non-stationarity) of the current audio segment. z(l,m) is dynamically changing, depending on the input content.

[0083] where W H is not a control parameter, but a fixed weight of the parameter generation model H: W H is the weight parameter of the H model itself (such as the weight of the fully connected layer), which is learned in the training stage and remains unchanged in the inference stage. It is used to map z(l,m) to W G .

[0084] W G is a dynamically generated control parameter: W G is the weight of the speech enhancement model G, which is generated by H according to z(l,m) and W H . W G is a "control parameter" that makes the behavior of the speech enhancement model G adaptive to the noise environment. Therefore, the core of the control parameter is W G , while z(l,m) is an intermediate variable, and W H is the inherent parameter of the parameter generation model. The whole mechanism realizes "content-based parameter adaptation".

[0085] When the noise perception model detects a sudden impulse noise (such as impact sound, machine roar), through the noise perception model output high non-stationarity index, drive H to generate which focuses on transient suppression. The generated will emphasize the rapid suppression of transient interference, avoiding the deletion of speech segments. The weight tends to produce a high-pass filter or transient suppressor, with high time resolution (fast response, acting within milliseconds), which can quickly attenuate noise energy, but preserves the spectral structure of the speech body. For example, the weight may strengthen the suppression of high-frequency burst components, reducing spectral leakage. This is the strong suppression weight.

[0086] When detecting stationary or low-frequency noise (such as multi-person conversation interference), the low-frequency energy distribution information in The weights will emphasize the precise separation of noise and speech, especially for low frequency bands. The weights tend to produce low-pass or band-stop filters with high frequency resolution, which can finely separate the spectrally overlapping parts. For example, the weights can adjust the convolution kernel to enhance the noise estimation in the low-frequency region, avoiding the distortion of speech formants. This is the fine separation weight.

[0087] In summary, the lightweight noise perception model (small parameter quantity) and the real-time operation of the parameter generation model H ensure the overall response speed of the system. The generation process is lightweight and efficient, and the parameter generation model H is only a fully connected layer, ensuring that the adjustment is completed within 10 milliseconds. The overall weights change dynamically, and the system can quickly respond to noise changes, achieving "targeted treatment".

[0088] Through this step, content-based parameter adaptation is achieved. For different input audio (such as sudden applause, smooth machine roar, and continuous human voice interference), the noise perception model will output different , thereby generating different . This means that the behavior of the speech enhancement model G is dynamically changing: for applause, it will produce a filter that is good at suppressing instantaneous pulses; for low-frequency roar, it will produce a filter that is good at suppressing low frequencies. For multi-person conversation interference, it will produce a low-pass or band-stop filter with high frequency resolution, which can finely separate the spectrally overlapping parts. This "targeted treatment" mechanism is the key to overcoming the shortcomings of traditional static enhancement models and effectively handling non-stationary noise.

[0089] The present application considers the technical problem that the parameters of the pre-trained general enhancement model in the related art are static and cannot perceive and adapt to the real-time changes in the noise characteristics of the input audio, the model is rigid and lacks adaptability, and the accuracy of speech recognition is poor, and proposes a technical solution for adaptively generating model parameters of a speech enhancement model according to a target speech signal to be recognized, thereby solving the problem of model rigidity in the related art and improving the adaptability and accuracy of speech recognition. Specifically, the target speech signal to be recognized is frame-processed to obtain at least one frame of speech signal; the first power spectrum of each of the at least one frame of speech signal is determined, and then the Mel spectrum of each of the at least one frame of speech signal is determined; the noise feature vector of each of the at least one frame of speech signal is determined based on a noise perception model; and then the target model parameters of the speech enhancement model are determined based on a parameter generation model, thereby adaptively generating the model parameters of the speech enhancement model according to the target speech signal to be recognized. Then, the first power spectrum of each of the at least one frame of speech signal is input into the speech enhancement model, and the second power spectrum of each of the at least one frame of speech signal is determined based on the speech enhancement model; finally, the target text sequence corresponding to the target speech signal is determined based on a speech recognition model. Thus, the accuracy of the determined target text sequence is improved.

[0090] In the present application, after obtaining the target speech signal to be recognized, the target speech signal is subjected to frame processing to obtain at least one frame of speech signal, and before that, the method further comprises: performing pre-emphasis processing on the target speech signal; wherein the process of pre-emphasis processing comprises: for at least one sampling point in the target speech signal, determining the amplitude of the sampling point after pre-emphasis processing according to the sampling amplitude of the sampling point, the sampling amplitude of the previous sampling point of the sampling point and a preset pre-emphasis coefficient.

[0091] The present application considers that the energy of the high frequency part in the speech signal is usually lower than that of the low frequency part, and directly performing Mel frequency spectrum feature extraction on the speech signal will cause the balance of the obtained Mel frequency spectrum to be poor, thereby affecting the accuracy of subsequent speech recognition. Based on this consideration, the present application proposes, after obtaining the target speech signal to be recognized, performing pre-emphasis processing on the target speech signal before performing frame processing on the target speech signal to obtain at least one frame of speech signal.

[0092] Pre-emphasis processing is a high-pass filtering process, the purpose of which is to enhance high frequency components, balance the Mel frequency spectrum, make the Mel frequency spectrum of the speech signal flat, facilitate subsequent Mel frequency spectrum analysis, and thereby improve the accuracy of speech recognition. Optionally, the process of pre-emphasis processing can be represented as: = - α· ; wherein, is the sampling amplitude of sampling point n, is the sampling amplitude of sampling point n-1, and a is a pre-emphasis coefficient, the value range of which is usually [0.95, 0.99]. The closer a is to 1, the more obvious the high frequency enhancement is. Optionally, a = 0.97, is the sampling amplitude of sampling point n after pre-emphasis processing.

[0093] It should be noted that for n = 0, is usually considered to be 0 or is processed by symmetric extension.

[0094] For example: Let a = 0.97, = 0.5, = 0.2; then = 0.5-0.97 0.2 = 0.306.

[0095] The present application considers that the speech signal is non-stationary, but its characteristics (such as spectrum) can be considered to be relatively stable (quasi-stationary) in a short time segment (usually 10-30 ms). The purpose of framing is to cut the long-time signal into many short, possibly overlapping segments, so as to analyze each segment independently. Based on the above considerations, the present application proposes to perform framing processing on the target speech signal to obtain at least one frame of speech signal, including: According to the total number of sampling points of the target speech signal, a preset frame shift parameter and a preset frame length parameter, a total number of frames is determined; according to the total number of frames, the preset frame shift parameter and the preset frame length parameter, the target speech signal is subjected to framing processing to obtain at least one frame of speech signal.

[0096] Optionally, the first value of the mth sampling point in the frame: = ; wherein, = 0,1,2,3,..., -1 is the index of the frame; m =0,1,2,3,....M-1 is the index of the sampling point within a frame; M is the frame length, i.e. the number of sampling points contained in a frame. For example, it is set to 20-40 ms, for example, when =16kHz, M= ·0.025 = 400 sampling points (25 ms); R is the frame shift, i.e. the number of sampling points between the starting points of adjacent two frames, which is usually set to 10 ms (the overlapping part is 15 ms). For example: when =16kHz, R = ·0.01 =160 sampling points; L is the total number of frames, L = + 1; R is usually less than M, which means that the frames are partially overlapped.

[0097] The value of the mth sampling point in the first frame is calculated, which corresponds to the absolute position n in the original signal (n).

[0098] For example, it is assumed that the parameters of a pre-processed speech signal are as follows: The total number of sampling points N=2000 (i.e. the total length of the signal is 2000 sampling points); The frame length M = 400 (each frame contains 400 sampling points, corresponding to a frame length of 25 ms if the sampling rate is 16 kHz); The frame shift R = 160 (the adjacent frame interval is 160 sampling points, corresponding to a frame shift of 10 ms).

[0099] Substitute into the formula to calculate the total number of frames L: L = (2,000 - 400) / 160 + 1 = 11; After the speech signal is framed according to the above parameters, a total of 11 valid speech frames can be obtained. Among them: The 1st frame ( = 0): The starting sampling point is 0, and the ending sampling point is 399; The 2nd frame ( = 1): The starting sampling point is 160, and the ending sampling point is 559; ... The 11th frame ( = 10): The starting sampling point is 10×160 = 1,600, and the ending sampling point is 1,600 + 400 - 1 = 1,999.

[0100] In addition, it should be noted that if the total number of sampling points N of the speech signal is small, or the frame length M is large, the situation of (N - M)<0 may occur, that is, the total signal length is less than one frame. At this time (N - M) / R is negative , for example, N = 300, M = 400, then (N - M) / R = -100 / 160 = -0.625, -0.625 = -1), substituting into the formula gives L = -1 + 1 = 0, and this situation has an unreasonable problem. To avoid the above problems, in practical applications, if N < M, the total number of frames L is forced to be set to 1, that is, only 1 frame is taken, including all N sampling points, and the insufficient part can be filled with zeros or directly truncated.

[0101] This application takes into account the spectral leakage effect introduced by truncating the signal after the speech signal is framed, thereby reducing the frequency resolution and amplitude estimation accuracy, and affecting the accuracy of subsequent speech recognition. A technical solution for determining the power spectrum of the frame speech signal by combining windowing processing and short-time Fourier transform is proposed. Specifically, determining the respective first power spectra of the at least one frame of speech signals includes: The at least one frame of speech signals is windowed according to a window function; the at least one frame of speech signals after the windowing is processed is subjected to short-time Fourier transform to obtain respective frequency domain signal values of the at least one frame of speech signals; and respective first power spectrums of the at least one frame of speech signals are determined according to the respective frequency domain signal values of the at least one frame of speech signals.

[0102] In the present application, in determining the respective first power spectrums of the at least one frame of speech signals, the at least one frame of speech signals is first windowed according to a window function. Each frame of speech signals is multiplied by a window function so as to be smoothly attenuated to zero at both ends of the frame. Thus, the spectral leakage effect is improved.

[0103] The windowing process can be represented as: = ; Wherein, is a time domain signal value (Hz) of an mth sampling point of an nth frame before windowing, l is a time domain signal value (Hz) of the mth sampling point of the nth frame after windowing, is a window function, and optionally, the window function is a Hamming window, and a calculation formula of the Hamming window is: l = 0.54 - 0.46·cos( ); After the windowing, each frame of data is close to 0 at the boundary and maintains the original signal characteristics inside. An example is shown as follows: For an m = 200th sampling point of an nth frame, a window length M = 400, and a window value of the point is:

[0104] = 0.54 - 0.46·cos( )≈0.54-0.46 (-1) = 1.0. It is shown that the center gain of the window is maximum and almost no attenuation.

[0105] = = .

[0106] ​​​​After the at least one frame of speech signal is windowed according to the window function; the at least one frame of speech signal after the windowing processing is subjected to short-time Fourier transform, aiming to convert the time domain signal value of the frame speech signal into a frequency domain signal value. That is, the above frame time domain signal is converted into a frequency domain to obtain its spectral representation . This reveals the distribution of the speech signal on different frequency components, for example, which frequency components are contained in this small piece of sound, and how strong each is.

[0107] Specifically, the short-time Fourier transform of the at least one frame of speech signal after the windowing processing to obtain the respective frequency domain signal value of the at least one frame of speech signal comprises: According to the formula = · , the respective frequency domain signal value of the at least one frame of speech signal is determined; Wherein, l represents the frame index, k represents the frequency index, X(l, k) represents the frequency domain signal value of the frame speech signal, m represents the sampling point, =0, 1, 2, 3,..., M-1, M represents a preset frame length parameter, , j is an imaginary unit, N FFT is a preset FFT point number.

[0108] = -1, N FFT , which can be an integer power of 2 greater than or equal to M, for example, N FFT = .

[0109] is a rotation factor, according to Euler's formula: = cos + j·sin When = , then = cos( ) + j·sin( ). is a complex number, containing the amplitude and phase information of the first frame, the kth frequency point.

[0110] The following is an example: Assume that the frame length M=8, the FFT point number =8 (example value, take a smaller value for easy calculation), the current frame index =0 (the first frame), the windowed signal The values ​​are: [0.2, 0.5, 1.0, 0.8, 0.3, 0.1, 0.4, 0.6]; Frequency index k=2, intra-frame position m ranges from 0 to 7, then the rotation factor... = . Figure 3 A schematic diagram illustrating the calculation of the real and imaginary parts of the rotation factor provided in this application.

[0111] Final calculation : = · ]. Figure 4 This is a schematic diagram illustrating the calculation of the real part product and the imaginary part product provided in this application.

[0112] Summing the real parts: 0.2 + 0 - 1.0 + 0 + 0.3 + 0 - 0.4 + 0 = -0.9; Summing the imaginary parts: 0 - 0.5 + 0 + 0.8 + 0 - 0.1 + 0 + 0.6 = 0.8; Final = -0.9 + 0.8j.

[0113] In this application, after the electronic device obtains the frequency domain signal values ​​of at least one frame of speech signal, it determines the first power spectrum of each of the at least one frame of speech signal based on these values. The power spectrum is the basis for many subsequent speech enhancement algorithms (such as Wiener filtering). Optionally, the energy of the signal at each frequency point is calculated; this is a real number for ease of subsequent processing. It is the squared magnitude of the STFT result. Then, it represents the... The signal power spectrum of the frame and the k-th frequency point for: = = + .

[0114] Representing complex numbers The real part; Representing complex numbers The imaginary part; For example: = a+bj then = .

[0115] In this application, determining the Mel spectrum of each of the at least one frame of speech signal based on the first power spectrum of each of the at least one frame of speech signal includes: According to the mapping relationship between the Mel scale and the linear physical frequency, the linear physical frequency is converted to the Mel scale; a Mel filter is set on the Mel scale, and a transfer function value of the Mel filter is determined; According to the first power spectrum of each of the at least one frame of voice signals and the transfer function value, a Mel spectrum of each of the at least one frame of voice signals is determined.

[0116] The application first simulates the nonlinear hearing characteristics of the human ear, the human ear is more sensitive to low-frequency differences and not sensitive to high-frequency differences, and the linear frequency scale is mapped to the Mel scale. This representation is more in line with human auditory perception and shows excellent performance in deep learning speech processing.

[0117] The Mel scale is defined as mel(f) = 2595· ; Where f represents the linear physical frequency, unit Hz; mel(f) is the converted Mel frequency, unit mel, representing the "pitch" perceived by the human ear; Log10 represents the logarithmic function with base 10; 2595 and 700 are empirical constants that can be determined by psychoacoustic experiments, aiming to make a standard pitch (such as 1000 Hz) correspond to an integer Mel value (about 1000 mel). Figure 5 The mapping relationship between the linear frequency and the Mel frequency provided by the application is shown in the schematic diagram.

[0118] Create a Mel filter bank: design a set of (such as 40) triangular overlapping filters on the Mel scale. The transfer function value of the mth Mel filter acting on the frequency point k is denoted as (k) (a frequency point corresponds to 40 Mel scale values of the group) m = 0, 1, 2,..., 39.

[0119] Apply the filter bank: apply the power spectrum Through the Mel filter bank, the linear spectrum energy is integrated into each Mel band. For the power spectrum of each frame The total energy M of the filter channel is: = · .

[0120] At this point, the original time-domain signal is converted into a two-dimensional feature matrix.

[0121] Figure 6 The process diagram for generating a two-dimensional feature matrix by the feature mapping provided by the application includes: The input is a raw noisy speech signal; pre-emphasis processing; frame processing; windowing processing; short-time Fourier transform; calculate power spectrum and mel spectrum, get two-dimensional feature matrix.

[0122] In this application, the first power spectrum of each of the at least one frame of speech signal is input into the speech enhancement model, and the second power spectrum of each of the at least one frame of speech signal is determined based on the speech enhancement model, which comprises: The first power spectrum of each of the at least one frame of speech signal is input into the speech enhancement model, and the second power spectrum of each of the at least one frame of speech signal is determined based on the speech enhancement model; according to the first power spectrum and the ratio mask of each of the at least one frame of speech signal, the second power spectrum of each of the at least one frame of speech signal is determined.

[0123] Figure 7 The process diagram for determining the enhanced second power spectrum provided in this application comprises: The first power spectrum of the noisy speech is input, and the U-Net encoder is used for downsampling / feature extraction; after the U-Net bottleneck layer, it is connected to the U-Net decoder for upsampling / feature fusion; the dynamic weight parameter is input, and the dynamic weight parameter is used for final convolution; the ideal ratio mask IRM is output; the ideal ratio mask IRM and the first power spectrum are multiplied, and the enhanced second power spectrum is output.

[0124] The forward propagation process of the dynamically generated weight used in this application is the enhancement process. The speech enhancement model G adopts the U-Net structure, and the input is the power spectrum of the noisy speech , wherein the power spectrum Pnosiy is another representation of P(l,k), i.e. the first power spectrum of the noisy speech. It is used as the input of the adaptive speech enhancement module to estimate the ideal ratio mask M(t,f) and the enhanced second power spectrum; the output is the ideal ratio mask (IRM) is: = ; Where G is a speech synthesis function, which can be a neural vocoder or a spectrum generator. The estimated second power spectrum of the pure speech is: = · .

[0125] The U-Net encoder downsampling (feature extraction) is explained as follows.

[0126] Purpose: Extract and compress input features layer by layer, capture high-level abstract features of noise and speech in the spectrogram (such as global patterns of noise and formant structures of speech).

[0127] Specific process: Input: Power spectrum of noisy speech As a two-dimensional time-frequency graph (height = number of time frames t, width = number of frequency points f) input the network.

[0128] Operation: The encoder is composed of multiple cascaded "downsampling blocks". Each downsampling block usually performs the following operations: 1. Convolution operation: use multiple convolution kernels to convolve the input spectrogram, extract local time-frequency features (for example, a certain convolution kernel may be responsible for detecting the vertical stripe pattern of burst noise). 2. Activation function: introduce a nonlinear transformation (such as ReLU), enhance the expression ability of the model. 3. Pooling operation: downsample (such as max pooling), reduce the time-frequency resolution of the feature map. Pooling will reduce the size of the feature map, but will increase its feature channel number (depth), so as to gradually expand the receptive field, so that the network can pay attention to more global context information, not just local details.

[0129] Result: After multiple downsampling, the feature map becomes very small in space (time-frequency dimension) size, but the channel number is very deep, forming a highly abstract feature representation, preparing for the bottleneck layer processing.

[0130] The U-Net bottleneck layer is explained as follows.

[0131] Purpose: At the lowest resolution, integrate and convert the most abstract and core features extracted by the encoder. Here is the "hub" of information processing.

[0132] Specific process: Input: Feature map generated in the previous step.

[0133] Operation: Usually composed of a series of consecutive convolution layers (such as two 3x3 convolutions), without pooling or upsampling. These convolution layers further fuse and process these highly compressed features, learning the most essential mapping relationship between noise and speech.

[0134] Result: Output a deeply processed feature tensor, which will serve as the starting point for the decoder upsampling.

[0135] The U-Net decoder upsampling (feature fusion) process is explained as follows.

[0136] Purpose: Gradually restore the spatial resolution (time-frequency size) of the feature map, and decode the abstract features of the bottleneck layer back to the same time-frequency mask (IRM) as the original input size.

[0137] Specific process: Input: Feature tensor output by the bottleneck layer.

[0138] Operation: The decoder is composed of multiple cascaded "upsampling blocks". Each upsampling block usually performs the following operations: 1. Upsampling: enlarge the size of the feature map in the time-frequency dimension by one time through methods such as transposed convolution (deconvolution) or interpolation (such as bilinear interpolation). 2. Skip connection: This is the core innovation of U-Net. The feature map produced by the intermediate layer of the encoder at the same down-sampling scale is spliced with the feature map after the current up-sampling of the decoder in the channel dimension. For example, the feature map before the second down-sampling of the encoder is spliced onto the feature map after the second up-sampling of the decoder. The role of the skip connection: the encoder feature map contains more low-level details (such as the precise edges and harmonic structure of the speech signal), while the decoder feature map contains high-level semantic information. By fusing the two through the skip connection, the details lost in the down-sampling process can be made up, and the final generated At the same time of suppressing noise, the intelligibility and naturalness of speech can be more finely preserved, and distortion can be reduced. Convolution operation: the spliced feature map is convolved again to smooth the artifacts that may be brought by up-sampling and fuse information at different levels.

[0139] Result: After multiple up-sampling and feature fusion, an output is finally outputted which is completely the same in size as the input Ideal ratio mask in time frame t and frequency point f Each value in the mask is between 0 and 1, representing the preservation ratio of the speech component in the corresponding time-frequency cell.

[0140] Figure 8 The training process of the noise perception model and the parameter generation model provided in the present application includes: S201: frame processing is performed on a sample speech signal to obtain at least one frame of sample speech signal; the first sample power spectrum of each of the at least one frame of sample speech signal is determined; the sample Mel spectrum of each of the at least one frame of sample speech signal is determined according to the first sample power spectrum of each of the at least one frame of sample speech signal; the sample Mel spectrum of each of the at least one frame of sample speech signal is input into a noise perception model to be trained, and the sample noise feature vector of each of the at least one frame of sample speech signal is determined based on the noise perception model to be trained; the sample noise feature vector of each of the at least one frame of sample speech signal is input into a parameter generation model to be trained, and the sample model parameter is determined based on the parameter generation model to be trained, and the model parameter of the speech enhancement model is updated using the sample model parameter; S202: input the first sample power spectrum of each of the at least one frame of sample voice signals into the voice enhancement model, determine the second sample power spectrum of each of the at least one frame of sample voice signals based on the voice enhancement model; input the second sample power spectrum of each of the at least one frame of sample voice signals into a voice recognition model, and determine the sample text sequence corresponding to the sample voice signal based on the voice recognition model; S203: determine a first loss value according to the second sample power spectrum of each of the at least one frame of sample voice signals and the label sample power spectrum of each of the at least one frame of sample voice signals; determine a second loss value according to the sample text sequence corresponding to the sample voice signal and the label text sequence corresponding to the sample voice signal; determine a fusion loss value according to the first loss value and the second loss value; and train the noise perception model and the parameter generation model according to the fusion loss value.

[0141] In an optional implementation, the noise perception model and the parameter generation model can be independently trained.

[0142] The training process of the noise perception model can be: frame processing is performed on the sample voice signal to obtain at least one frame of sample voice signals; the first sample power spectrum of each of the at least one frame of sample voice signals is determined; the sample Mel spectrum of each of the at least one frame of sample voice signals is determined according to the first sample power spectrum of each of the at least one frame of sample voice signals; the sample Mel spectrum of each of the at least one frame of sample voice signals is input into the noise perception model to be trained, and the sample noise feature vector of each of the at least one frame of sample voice signals is determined based on the noise perception model to be trained; a loss value is determined according to the sample noise feature vector of each of the at least one frame of sample voice signals and the label noise feature vector of each of the at least one frame of sample voice signals pre-labeled, and the noise perception model is trained through iterative training when the loss value meets the requirement or the number of iterations reaches a set number of threshold values.

[0143] The training process of the parameter generation model can be: The sample noise feature vector of each of the at least one frame of sample voice signals is input into the parameter generation model to be trained, and the sample model parameter is determined based on the parameter generation model to be trained; a loss value is determined according to the sample model parameter and the label model parameter pre-labeled, and the parameter generation model is trained through iterative training when the loss value meets the requirement or the number of iterations reaches a set number of threshold values.

[0144] Preferably, in order to improve the accuracy of the training of the noise perception model and the parameter generation model, the two can be jointly trained. Specifically, the sample mel spectrum of at least one frame of sample speech signal is input into the noise perception model to be trained, and the sample noise feature vector of the at least one frame of sample speech signal is determined based on the noise perception model to be trained; the sample noise feature vector of at least one frame of sample speech signal is input into the parameter generation model to be trained, and the sample model parameter is determined based on the parameter generation model to be trained, and the sample model parameter is used to update the model parameter of the speech enhancement model; the first sample power spectrum of at least one frame of sample speech signal is input into the speech enhancement model, and the second sample power spectrum of at least one frame of sample speech signal is determined based on the speech enhancement model; the second sample power spectrum of at least one frame of sample speech signal is input into the speech recognition model, and the sample text sequence corresponding to the sample speech signal is determined based on the speech recognition model; the first loss value is determined according to the second sample power spectrum of at least one frame of sample speech signal and the label sample power spectrum of at least one frame of sample speech signal; the second loss value is determined according to the sample text sequence corresponding to the sample speech signal and the label text sequence corresponding to the sample speech signal; the fusion loss value is determined according to the first loss value and the second loss value; and the noise perception model and the parameter generation model are trained according to the fusion loss value. Through iterative training, when the fusion loss value meets the requirements or the number of iterations reaches a set number of threshold, the training of the noise perception model and the parameter generation model is completed.

[0145] The fusion loss value can be determined according to the first loss value and the second loss value. The sum of the first loss value and the second loss value can be determined as the fusion loss value. Alternatively, different weights can be configured for the first loss value and the second loss value, and the first loss value, the second loss value and the respective weights are weighted and summed to obtain the fusion loss value.

[0146] Figure 9 The process of updating the parameters of the noise perception model and the parameter generation model provided in the present application includes: Input noisy speech; output enhanced features and enhanced power spectrum from the adaptive speech enhancement model; obtain predicted text from the FunSAR recognition engine with frozen parameters; determine recognition loss based on the predicted text; determine enhancement loss based on the enhanced power spectrum; determine joint loss according to the recognition loss and the enhancement loss; update the parameters of the noise perception model and the parameter generation model according to the gradient backpropagation of the joint loss.

[0147] In the present application, the text sequence or the mel spectrum converted therefrom is input into the frozen, pre-trained FunASR model (Paraformer) for recognition to obtain a text sequence FunASR model is a parameter-fixed speech recognition model.

[0148] Total loss function is composed of two parts: 1. Recognition loss , the cross-entropy loss between the predicted text output by the FunASR model and the true text; 2. Enhancement loss , the mean square error (MSE) loss between the estimated clean speech spectrum and the true clean speech spectrum, as a regularization term to prevent the enhanced speech from being too distorted.

[0149] = ( ) + λ· ( ); where λ is a hyperparameter that balances the two losses, and can be set to a small value such as 0.1.

[0150] The training method provided in the present application has the following advantages: 1. Consistency of goals: It breaks the optimization goal split between the enhancement module (targeting MSE) and the recognition module (targeting WER) in the traditional pipeline. The front-end enhancer no longer pursues "clean sound", but learns how to enhance to "make the back-end recognizer hear the most accurate".

[0151] 2. Implicit perceptual optimization: Even without explicit speech separation or denoising labels, the system can implicitly learn how to suppress noise and preserve speech information through the guidance of the recognition loss. For example, it will learn to preserve the information of the final consonant, which is crucial for Chinese recognition, even if this information may not be significant in the waveform.

[0152] 3. System robustness: The enhancement system trained in this way, in cooperation with the specific recognizer (FunASR) in the back-end, achieves a high degree of tacit understanding, and can perform better generalization than traditional methods when facing noise that has not been seen during training, thereby significantly reducing the word error rate (WER) in complex environments.

[0153] The present application updates only the parameters of the noise perception model N and the parameter generation model H ( , ) through the gradient descent algorithm. The parameters of the FunASR model are frozen at this time, and the architecture of the speech enhancement model G is fixed, but its weights ​​​​generated by H, so the behavior of G is changing, but its parameters themselves are not directly updated by gradient. This design greatly reduces the training complexity and overfitting risk. The advantages of the above design provided by the application are as follows: 1. Target-oriented enhancement: the gradient from the recognition task directly guides how the noise-aware network and the parameter generator adjust. Instead of simply learning "noise reduction", the network learns "how to reduce noise to benefit recognition the most".

[0154] 2. System synergy: through the ingenious gradient flow design, the front-end lightweight network and the back-end large recognizer are organically integrated into a trainable framework, while avoiding the destruction of the performance of the pre-trained model.

[0155] 3. Efficient adaptability: ultimately, by minimizing , the system learns to generate the most suitable enhancement network weights for different noise scenarios , realizing the "on-demand customization" of the front-end enhancer, thereby significantly improving the overall robustness and accuracy of speech recognition in complex non-stationary noise environments.

[0156] The speech recognition method provided by the application belongs to the FunASR speech recognition system, supports the intelligent creation platform and the MCC project, and aims to support the efficient and stable operation of the speech recognition system based on FunASR in complex acoustic environments. The application adopts a dynamic network parameter adjustment and joint optimization strategy based on noise perception, realizes a self-adaptive speech enhancement and recognition method that conforms to acoustic laws, can enhance speech intelligibility and intelligibility, and improve user auditory perception experience, effectively solves the problems of poor generalization ability of traditional speech enhancement models, serious speech distortion and significant decrease in recognition accuracy in non-stationary noise environments. At the same time, by introducing a lightweight noise perception and parameterized generation mechanism, the application solves the problem of high system computing load and response delay caused by the need to process a large number of diversified noise types in high-concurrency and multi-scene applications, ensuring the robustness and real-time performance of the speech recognition service in high-noise environments.

[0157] The core innovation of the application lies in proposing a speech enhancement front-end that is adaptive to input audio, and deeply optimizing it with the FunASR recognition engine. The specific improvements are as follows: 1. Construct a lightweight noise perception model: in addition to the main enhancement network, a small neural network (such as a shallow CNN) is constructed in parallel, which is used to analyze the noise characteristics (such as noise spectrum, non-stationarity) of the input audio in real time, and outputs a set of control parameters θ, θ is the noise feature vector z(l, m) output by the noise perception model and the output W of the parameter generation model H G ​Specifically: a lightweight noise-aware model (e.g., a three-layer CNN) analyzes the Mel-spectrogram M(l,m) of the input audio in real-time and outputs a noise feature vector z(l,m), which can be regarded as an instantiation of θ.

[0158] Then, the parameter generation model H uses z(l,m) and its weight W H The weight W of the speech enhancement model G is dynamically generated G (i.e., W G = H(z(l,m); W H )). Therefore, the parameter θ corresponds to the overall concept of z(l,m) and W G , which together achieve adaptive adjustment of the parameters.

[0159] 2. Dynamically generate speech enhancement model parameters: use the control parameter θ output by the noise-aware model to dynamically generate or modulate the weight of the speech enhancement model G through a hypernetwork or linear transformation layer, i.e. = H(θ), where H is the parameter generation model. This enables the speech enhancement model to adaptively adjust its behavior according to the current noise environment.

[0160] 3. Joint optimization strategy with FunASR: Instead of using "clean speech" as the training target for the enhancement module, the accuracy of the final recognition result is used as the optimization target. Through gradient propagation, the loss function (e.g., CTC / Attention Loss) of the FunASR recognizer is backpropagated to the front-end enhancement network, guiding it to learn how to enhance in order to best benefit the back-end recognition, rather than simply pursuing waveform similarity.

[0161] Figure 10 The overall flowchart of the speech recognition based on environmental noise enhancement provided in this application includes: Start processing the input noisy audio; feature extraction: calculate the Mel-spectrum / power spectrum; noise awareness: lightweight noise-aware model analyzes noise features; environmental judgment and parameter generation: generate dynamic weights based on noise features; industrial impulse noise scenario, generate "strong suppression" weights; multi-person conversation interference scenario, generate "fine separation" weights; scenarios such as sudden applause, generate "fast response" weights; perform adaptive speech enhancement; input FunSAR engine for recognition; in the training stage, calculate the joint loss, backpropagate only to update the noise-aware model parameters and the parameters of the parameter generation model, end this round of training; in the application stage, output the final recognition text, end the inference.

[0162] This application achieves a breakthrough improvement in complex acoustic environments through noise-aware dynamic adjustment and enhancement-recognition joint optimization, with the following specific effects: 1. Dynamic environmental adaptability is significantly enhanced.

[0163] Noise perception real-time response: A lightweight noise perception model (such as a three-layer CNN) analyzes the spectral characteristics (such as non-stationarity and noise spectrum distribution) of the input audio in real time, generating control parameters θ to drive the main enhancement network to dynamically adjust.

[0164] Effect: The response speed to sudden impulse noise (applause, impact sound) is improved to 10 milliseconds, avoiding the voice mis-suppression caused by the parameter solidification of traditional static models (such as voice loss after applause).

[0165] 2. Speech recognition accuracy is significantly improved.

[0166] Joint optimization reduces distortion transmission: The joint training target dominated by the recognition loss (ASR Loss) makes the enhancement network learn "identification-friendly" speech features rather than simply waveform similarity.

[0167] Effect: In a factory noise test set with a signal-to-noise ratio of 0-10dB, the word error rate (WER) is reduced to 5.2% (the WER of the traditional scheme is ≥15%).

[0168] Mechanism: By suppressing irrelevant spectral details (such as phase disturbance) and preserving key formants and vowel structures, the robustness of the FunASR backend is improved.

[0169] 3. Optimization of computational efficiency and real-time performance.

[0170] Lightweight parameter generation: The parameter generation model H (fully connected layer) dynamically outputs the enhancement network weights , avoiding the deployment of redundant model branches.

[0171] Data support: The parameter quantity of the noise perception model accounts for a very low proportion of the main enhancement network.

[0172] 4. Improved speech quality and user experience.

[0173] Adaptive mask optimization for listening: The enhancement network based on U-Net outputs an ideal ratio mask M(t,f), which combines the Mel spectrum mapping and human auditory characteristics.

[0174] 5. Overall improvement of system robustness.

[0175] Training-inference scenario mismatch solution: The dynamic parameterization mechanism covers noise types (such as applause, calls, and machine sounds) not seen in the training set, significantly enhancing the generalization ability.

[0176] Effect: In a multi-class untrained noise environment (including subway broadcasts and restaurant noise), the WER fluctuation range is significantly reduced compared to related technologies.

[0177] The application overcomes the industry problems of speech enhancement distortion, rapid decline of recognition rate and high computing load under non-stationary noise through the technical closed loop of environment perception, dynamic parameter adjustment and joint optimization, and provides high-robustness speech interaction support for conference, vehicle-mounted and industrial scenarios.

[0178] Figure 11 A structure schematic diagram of a speech recognition device based on environment noise enhancement is provided for the application, and the device comprises: The determination module 11 is configured to obtain a target speech signal to be recognized, perform frame processing on the target speech signal to obtain at least one frame of speech signal, determine a first power spectrum of each of the at least one frame of speech signal, determine a Mel spectrum of each of the at least one frame of speech signal according to the first power spectrum of each of the at least one frame of speech signal, input the Mel spectrum of each of the at least one frame of speech signal into a noise perception model, determine a noise feature vector of each of the at least one frame of speech signal based on the noise perception model, input the noise feature vector of each of the at least one frame of speech signal into a parameter generation model, determine a target model parameter based on the parameter generation model, and update a model parameter of a speech enhancement model by using the target model parameter. The recognition module 12 is configured to input the first power spectrum of each of the at least one frame of speech signal into the speech enhancement model, determine a second power spectrum of each of the at least one frame of speech signal based on the speech enhancement model, input the second power spectrum of each of the at least one frame of speech signal into a speech recognition model, and determine a target text sequence corresponding to the target speech signal based on the speech recognition model.

[0179] The determination module 11 is further configured to perform pre-emphasis processing on the target speech signal, wherein for at least one sampling point in the target speech signal, a pre-emphasis processed amplitude of the sampling point is determined according to a sampling amplitude of the sampling point, a sampling amplitude of a previous sampling point of the sampling point and a preset pre-emphasis coefficient.

[0180] The determination module 11 is specifically configured to determine a total frame number according to a total sampling point number of the target speech signal, a preset frame shift parameter and a preset frame length parameter, perform frame processing on the target speech signal according to the total frame number, the preset frame shift parameter and the preset frame length parameter to obtain at least one frame of speech signal.

[0181] The determination module 11 is specifically configured to perform windowing processing on the at least one frame of speech signal according to a window function, perform short-time Fourier transform on the at least one frame of speech signal after the windowing processing to obtain a frequency domain signal value of each of the at least one frame of speech signal, and determine the first power spectrum of each of the at least one frame of speech signal according to the frequency domain signal value of each of the at least one frame of speech signal.

[0182] The determining module 11 is specifically configured to determine the frequency domain signal value of each of the at least one frame of speech signals according to the formula · , wherein l represents a frame index, k represents a frequency index, X(l, k) represents the frequency domain signal value of the frame of speech signals, m represents a sampling point, = 0, 1, 2, 3,..., M-1, M represents a preset frame length parameter, represents the time domain signal value of the frame of speech signals after windowing processing, j is an imaginary unit, N FFT represents a preset FFT point number.

[0183] The determining module 11 is specifically configured to convert the linear physical frequency to the mel scale according to a mapping relationship between the mel scale and the linear physical frequency, set a mel filter on the mel scale, and determine a transfer function value of the mel filter; and determine the mel spectrum of each of the at least one frame of speech signals according to the first power spectrum of each of the at least one frame of speech signals and the transfer function value.

[0184] The identifying module 12 is specifically configured to input the first power spectrum of each of the at least one frame of speech signals into the speech enhancement model, determine the ratio mask of each of the at least one frame of speech signals based on the speech enhancement model, and determine the second power spectrum of each of the at least one frame of speech signals according to the first power spectrum and the ratio mask of each of the at least one frame of speech signals.

[0185] The device further comprises: ​​The model training module 13 is used to perform frame segmentation processing on the sample speech signal to obtain at least one frame of sample speech signal; determine the first sample power spectrum of each of the at least one frame of sample speech signal; determine the sample Mel spectrum of each of the at least one frame of sample speech signal based on the first sample power spectrum of each of the at least one frame of sample speech signal; input the sample Mel spectrum of each of the at least one frame of sample speech signal into the noise perception model to be trained, and determine the sample noise feature vector of each of the at least one frame of sample speech signal based on the noise perception model to be trained; input the sample noise feature vector of each of the at least one frame of sample speech signal into the parameter generation model to be trained, determine the sample model parameters based on the parameter generation model to be trained, and update the model parameters of the speech enhancement model using the sample model parameters; and process the at least one frame of sample speech signal into a sample power spectrum. The first sample power spectrum of each of the sample speech signals is input into the speech enhancement model, and the second sample power spectrum of each of the at least one frame of sample speech signals is determined based on the speech enhancement model; the second sample power spectrum of each of the at least one frame of sample speech signals is input into the speech recognition model, and the sample text sequence corresponding to the sample speech signal is determined based on the speech recognition model; a first loss value is determined according to the second sample power spectrum of each of the at least one frame of sample speech signals and the label sample power spectrum of each of the at least one frame of sample speech signals; a second loss value is determined according to the sample text sequence corresponding to the sample speech signal and the label text sequence corresponding to the sample speech signal; a fusion loss value is determined according to the first loss value and the second loss value; and the noise perception model and the parameter generation model are trained according to the fusion loss value.

[0186] This application also provides an electronic device, such as Figure 12 As shown, it includes: processor 21, communication interface 22, memory 23 and communication bus 24, wherein processor 21, communication interface 22 and memory 23 communicate with each other through communication bus 24; The memory 23 stores a computer program, which, when executed by the processor 21, causes the processor 21 to perform any of the above method steps.

[0187] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0188] The communication interface 22 is configured to enable communication between the electronic device and other devices.

[0189] The memory can include a Random Access Memory (RAM) and can also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.

[0190] The processor can be a general processor, including a central processing unit, a network processor (NP), etc. The processor can also be a Digital Signal Processor (DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, etc.

[0191] The present application also provides a computer storage readable storage medium, which stores a computer program executable by an electronic device, and when the program is run on the electronic device, the electronic device is caused to perform the above method steps.

[0192] The present application provides a computer program product, which includes an executable program, and when the executable program is executed by a processor, the method is implemented.

[0193] Although the preferred embodiments of the present application have been described, those skilled in the art who have the basic inventive concept can make further changes and modifications to the embodiments. Therefore, the appended claims are intended to include the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0194] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A speech recognition method based on ambient noise enhancement, characterized by, The method includes: The process involves acquiring a target speech signal to be identified, performing frame-segmentation on the target speech signal to obtain at least one frame of speech signal, determining the first power spectrum of each of the at least one frame of speech signal, determining the Mel spectrum of each of the at least one frame of speech signal based on the first power spectrum of each of the at least one frame of speech signal, inputting the Mel spectrum of each of the at least one frame of speech signal into a noise perception model, determining the noise feature vector of each of the at least one frame of speech signal based on the noise perception model, inputting the noise feature vector of each of the at least one frame of speech signal into a parameter generation model, determining the target model parameters based on the parameter generation model, and updating the model parameters of the speech enhancement model using the target model parameters. The first power spectrum of each of the at least one frame of speech signal is input into the speech enhancement model, and the second power spectrum of each of the at least one frame of speech signal is determined based on the speech enhancement model; the second power spectrum of each of the at least one frame of speech signal is input into the speech recognition model, and the target text sequence corresponding to the target speech signal is determined based on the speech recognition model.

2. The method of claim 1, wherein, After acquiring the target speech signal to be identified, and before performing frame-segmentation processing on the target speech signal to obtain at least one frame of speech signal, the method further includes: The target speech signal is pre-emphasized; wherein the pre-emphasis processing includes: For at least one sampling point in the target speech signal, the amplitude of the sampling point after pre-emphasis processing is determined based on the sampling amplitude of the sampling point, the sampling amplitude of the previous sampling point, and a preset pre-emphasis coefficient.

3. The method of claim 1, wherein, The target speech signal is subjected to frame segmentation processing to obtain at least one frame of speech signal, including: The total number of frames is determined based on the total number of sampling points of the target speech signal, the preset frame shift parameter, and the preset frame length parameter; the target speech signal is then divided into frames based on the total number of frames, the preset frame shift parameter, and the preset frame length parameter to obtain at least one frame of speech signal.

4. The method of claim 1, wherein, Determining the first power spectrum of each of the at least one frame of speech signal includes: The at least one frame of speech signal is windowed according to a window function; a short-time Fourier transform is performed on the windowed at least one frame of speech signal to obtain the frequency domain signal value of each of the at least one frame of speech signal; and the first power spectrum of each of the at least one frame of speech signal is determined according to the frequency domain signal value of each of the at least one frame of speech signal.

5. The method of claim 4, wherein, The step of performing a short-time Fourier transform on at least one frame of the windowed speech signal to obtain the frequency domain signal values ​​of each of the at least one frame of the speech signal includes: According to the formula = · , the respective frequency domain signal values of the at least one frame of speech signal are determined; wherein, I represents a frame index, k represents a frequency index, X(I, k) represents a frequency domain signal value of a frame voice signal, m represents a sampling point, = 0, 1, 2, 3,..., M-1, M represents a preset frame length parameter, represents a time domain signal value of a frame voice signal after windowing processing, j is an imaginary unit, N FFT is a preset FFT point number.

6. The method of claim 1, wherein, Determining the Mel spectrum of each of the at least one frame of speech signal based on its respective first power spectrum includes: Based on the mapping relationship between the Mel scale and the linear physical frequency, the linear physical frequency is converted to the Mel scale; a Mel filter is set on the Mel scale, and the transfer function value of the Mel filter is determined; The Mel spectrum of each of the at least one frame of speech signal is determined based on the first power spectrum of each frame and the transfer function value.

7. The method of claim 1, wherein, inputting the first power spectrum of each of the at least one frame of speech signals into the speech enhancement model, and determining the second power spectrum of each of the at least one frame of speech signals based on the speech enhancement model comprises: inputting the first power spectrum of each of the at least one frame of speech signals into the speech enhancement model, and determining the second power spectrum of each of the at least one frame of speech signals based on the speech enhancement model comprises:

8. The method of claim 1, wherein, The training process of the noise perception model and the parameter generation model comprises: frame processing of the sample speech signals to obtain at least one frame of sample speech signals, determining the first sample power spectrum of each of the at least one frame of sample speech signals, determining the sample Mel spectrum of each of the at least one frame of sample speech signals based on the first sample power spectrum of each of the at least one frame of sample speech signals, inputting the sample Mel spectrum of each of the at least one frame of sample speech signals into the noise perception model to be trained, determining the sample noise feature vector of each of the at least one frame of sample speech signals based on the noise perception model to be trained, inputting the sample noise feature vector of each of the at least one frame of sample speech signals into the parameter generation model to be trained, determining the sample model parameter based on the parameter generation model to be trained, and updating the model parameter of the speech enhancement model by using the sample model parameter; inputting the first sample power spectrum of each of the at least one frame of sample speech signals into the speech enhancement model, and determining the second sample power spectrum of each of the at least one frame of sample speech signals based on the speech enhancement model; inputting the second sample power spectrum of each of the at least one frame of sample speech signals into the speech recognition model, and determining the sample text sequence corresponding to the sample speech signals based on the speech recognition model; determining a first loss value based on the second sample power spectrum of each of the at least one frame of sample speech signals and the label sample power spectrum of each of the at least one frame of sample speech signals, determining a second loss value based on the sample text sequence corresponding to the sample speech signals and the label text sequence corresponding to the sample speech signals, determining a fusion loss value based on the first loss value and the second loss value, and training the noise perception model and the parameter generation model based on the fusion loss value.

9. An electronic device, comprising: The apparatus comprises a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory are in communication with each other via the communication bus. The memory is configured to store a computer program. The processor is configured to execute the program stored in the memory to implement the method of any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer program stored in the computer readable storage medium is configured to be executed by the processor to implement the method of any one of claims 1-8.

Images