High-fidelity end-to-end acoustic modeling method combining speech intelligibility reconstruction and autoregressive feedback optimization

Abstract

The invention belongs to the technical field of speech recognition and acoustic modeling, and particularly relates to a high-fidelity end-to-end acoustic modeling method combining speech intelligibility reconstruction and autoregressive feedback optimization, which comprises the following steps of: firstly, establishing an intelligibility loss function based on a human speech intelligibility model; subjective definition features of the voice are reconstructed through a frequency spectrum reconstruction network, intelligibility related features are extracted and embedded in combination with a perceptual weighted filter bank, and the intelligibility related features and Mel frequency spectrum features are input into an acoustic encoder in parallel; based on this, by introducing a voice intelligibility reconstruction and autoregressive feedback mechanism, while the conciseness of an end-to-end voice modeling framework is maintained, systematic improvement of voice sharpness, naturalness and stability is realized, and the method can be widely applied to scenes such as intelligent voice assistants, voice transcription, virtual anchors, voice restoration, cross-language voice generation and the like. And the method has extremely high practical application value and popularization prospect.

Description

Technical Field

[0001] This invention relates to the fields of speech recognition and acoustic modeling technology, specifically a high-fidelity end-to-end acoustic modeling method that combines speech intelligibility reconstruction and autoregressive feedback optimization. Background Technology

[0002] With the development of artificial intelligence and deep learning technologies, end-to-end speech recognition and speech synthesis technologies have become an important research direction in the field of speech processing. Traditional speech recognition systems typically include modules such as acoustic models, language models, and pronunciation dictionaries. Their training process relies on a large amount of labeled data and complex parameter tuning, resulting in a cumbersome system structure. Furthermore, their recognition performance significantly degrades in noisy environments or cross-language scenarios.

[0003] In recent years, end-to-end acoustic modeling methods have gradually replaced traditional staged architectures. By directly learning the mapping relationship between speech signals and text sequences, they achieve unified optimization from acoustic features to semantic output. However, existing end-to-end acoustic models still have the following shortcomings: First, while pursuing recognition accuracy, the models often neglect the subjective intelligibility of speech, resulting in speech outputs with high recognition rates but low semantic clarity, making it difficult to meet the auditory requirements of natural language interaction. Second, end-to-end systems generally adopt unidirectional or fixed feedforward prediction mechanisms, lacking dynamic feedback adjustment capabilities, and are prone to accumulating errors in long speech sequences or autoregressive inference, affecting the stability and consistency of generated speech.

[0004] Furthermore, existing speech reconstruction networks mostly rely on single spectral or temporal feature recovery methods, failing to comprehensively consider the balance between speech intelligibility and spectral integrity. This results in issues such as distortion at the detail level, phase discontinuity, and excessive smoothing in the output speech; especially in low signal-to-noise ratio environments, the robustness of the model decreases significantly. At the same time, most speech modeling methods cannot effectively handle cross-linguistic or multi-dialect inputs and lack a unified acoustic alignment and feedback mechanism, thus limiting their widespread application in multilingual speech recognition and high-fidelity speech generation.

[0005] Therefore, a high-fidelity end-to-end acoustic modeling method combining speech intelligibility reconstruction and autoregressive feedback optimization is invented. The method improves semantic clarity through intelligibility constraints and enhances generation stability by combining feedback optimization mechanisms, thereby achieving high-quality speech recognition and synthesis in multilingual, noisy, and low-resource environments. Summary of the Invention

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] A high-fidelity end-to-end acoustic modeling method combining speech intelligibility reconstruction and autoregressive feedback optimization includes the following specific steps:

[0008] S1, Speech Input Preprocessing: Acquire the raw speech signal, perform acoustic cleaning using multi-channel spectrogram denoising and dynamic time-frequency masking algorithms, and extract the fundamental frequency of the speech. The speech feature sequence input is obtained by segmenting speech segments using adaptive endpoint detection, based on the energy envelope and formant parameters.

[0009] S2, Speech Feature Perception and Intelligibility Modeling: First, an intelligibility loss function is established based on the human speech intelligibility model; then, the subjective clarity features of speech are reconstructed through a spectrum reconstruction network, and intelligibility-related feature embeddings are extracted by combining a perceptual weighted filter bank, and input into the acoustic encoder in parallel with the Mel spectrum features;

[0010] S3, High-fidelity end-to-end acoustic model training: First, an acoustic modeling network based on a hybrid architecture of Transformer and Conformer is adopted, with speech-text pairs as training samples. Then, a multi-task loss function, intelligibility reconstruction loss, and autoregressive prediction loss are adopted. By jointly optimizing the weights of the three, end-to-end mapping from speech to character sequences is achieved.

[0011] S4: Autoregressive Feedback Optimization: During the model inference stage, an autoregressive feedback module is introduced to recursively verify the prediction results of each frame of speech. The prediction distribution of historical frames is used to resample the input of the next frame to construct a time-dependent feedback loop to reduce inter-frame drift error. The model stability is improved by a dynamic learning rate adjustment mechanism based on KL divergence.

[0012] S5, Speech Output Reconstruction and Intelligibility Enhancement: The spectrum of the end-to-end model output is reconstructed into a speech signal, and the speech signal is reconstructed by combining a post-processing module based on an autoregressive residual network.

[0013] As a preferred embodiment of the high-fidelity end-to-end acoustic modeling method combining speech intelligibility reconstruction and autoregressive feedback optimization described in this invention, the dynamic time-frequency masking algorithm in S1 is based on a time-frequency dual attention mechanism, and combines a frequency-domain gated recurrent unit with a time-domain convolution module, which can maintain the integrity of the speech signal in an environment where the speech-to-noise ratio is less than 10dB.

[0014] As a preferred embodiment of the high-fidelity end-to-end acoustic modeling method combining speech intelligibility reconstruction and autoregressive feedback optimization described in this invention, the intelligibility modeling network in S2 uses a dual-channel perceptual subnetwork structure. The first channel takes the short-time Fourier transform amplitude spectrum as input, and the second channel takes the phase spectrum residual as input. The two are used to calculate the speech proficiency score through an adaptive fusion layer to achieve more accurate semantic intelligibility estimation.

[0015] As a preferred embodiment of the high-fidelity end-to-end acoustic modeling method combining speech intelligibility reconstruction and autoregressive feedback optimization described in this invention, wherein: the Conformer in S3 combines self-attention and convolutional enhancement layers, which can simultaneously capture long-range semantic dependencies and local acoustic changes; and the training objective function is:

[0016]

[0017] in, These represent the dynamic weights of the three types of losses, respectively.

[0018] As a preferred embodiment of the high-fidelity end-to-end acoustic modeling method combining speech intelligibility reconstruction and autoregressive feedback optimization described in this invention, wherein: the autoregressive feedback optimization in S4 adopts a two-stage mechanism:

[0019] Offline feedback correction is performed during the model training phase to reduce prediction drift through teacher-mandated correction.

[0020] The inference phase employs online feedback adjustment, correcting the current frame input by using the mean of the prediction distribution of the previous N frames, thereby enhancing prediction stability.

[0021] A high-fidelity end-to-end acoustic modeling system combining speech intelligibility reconstruction and autoregressive feedback optimization includes:

[0022] The speech preprocessing module is used to perform signal sampling, time-frequency transformation, noise reduction, and energy normalization;

[0023] The intelligibility modeling module is used to calculate speech intelligibility metrics and generate intelligibility embeddings;

[0024] The acoustic modeling module is used for end-to-end speech-to-text prediction based on the Transformer-Conformer hybrid structure.

[0025] The autoregressive feedback module is used to correct the prediction input of the current speech frame by using the historical prediction distribution, thereby achieving feedback self-optimization.

[0026] The speech reconstruction module is used to reconstruct a high-fidelity speech signal from the predicted spectral signal through residual enhancement filtering and inverse spectral domain convolution.

[0027] As a preferred embodiment of the high-fidelity end-to-end acoustic modeling system combining speech intelligibility reconstruction and autoregressive feedback optimization described in this invention, the intelligibility modeling module includes a perceptual analysis unit and a fusion computing unit. The perceptual analysis unit uses a dual-channel structure to extract features from the amplitude spectrum and phase spectrum. The fusion computing unit uses a multi-head attention mechanism to fuse the two types of features to generate an intelligibility embedding.

[0028] As a preferred embodiment of the high-fidelity end-to-end acoustic modeling system combining speech intelligibility reconstruction and autoregressive feedback optimization described in this invention, the autoregressive feedback module includes a temporal feedback unit and an error compensation unit. The temporal feedback unit recursively calculates the conditional distribution of the current prediction based on the probability distribution of historical frames, and the error compensation unit corrects the prediction error based on KL divergence and gradient variance estimation.

[0029] As a preferred embodiment of the high-fidelity end-to-end acoustic modeling system combining speech intelligibility reconstruction and autoregressive feedback optimization described in this invention, the speech reconstruction module includes an inverse spectral domain convolutional network and a residual enhancement filter. The inverse spectral domain convolutional network is used to reconstruct the time-domain waveform from the spectrum, and the residual enhancement filter is used to correct speech distortion regions and enhance speech naturalness.

[0030] Compared with existing technologies:

[0031] 1. Introduction of Speech Intelligibility Reconstruction Mechanism: Traditional end-to-end speech modeling only focuses on the numerical accuracy of the output waveform or spectrum, while ignoring the subjective perception characteristics of the human ear. This invention introduces speech intelligibility metrics (STOI, PESQ, etc.) into the training objective for the first time, and realizes the transformation from "minimizing objective error" to "optimizing subjective hearing" through the intelligibility reconstruction network, effectively improving speech clarity and semantic understanding.

[0032] 2. Autoregressive Feedback Optimization Architecture: Traditional end-to-end speech generation systems employ a unidirectional prediction structure, which is prone to drift and accumulated errors in long sequences. This invention proposes an autoregressive structure based on dynamic feedback adjustment. Through real-time error feedback and reinforcement learning control, it achieves self-correction in the continuous prediction process, improving temporal consistency and generation stability.

[0033] 3. Joint constraints of intelligibility and spectrum: This invention adopts a multi-task loss fusion mechanism to simultaneously minimize speech intelligibility error, spectrum reconstruction error and semantic matching error, forming a multi-dimensional collaborative optimization system, which fundamentally improves the naturalness, fidelity and fluency of speech.

[0034] 4. End-to-end multi-level joint training strategy: Through cross-module parameter sharing and unified optimization, this invention achieves end-to-end learning from speech input to high-fidelity output, no longer relying on traditional acoustic models, language models and dictionaries, significantly simplifying the system structure and reducing the need for manual annotation.

[0035] 5. Enhanced cross-language and robustness: Since the intelligibility reconstruction network performs language-independent modeling in the semantic space, this invention has high generalization ability for multilingual, dialect and noisy speech, and can maintain stable output under low signal-to-noise ratio and speech rate variation conditions.

[0036] 6. High-fidelity audio quality and low-latency output: The speech generated by this invention is superior to traditional systems in terms of formant, phase continuity and waveform smoothness, while significantly reducing generation latency, making it suitable for real-time voice interaction and intelligent broadcasting systems.

[0037] In summary, by introducing speech intelligibility reconstruction and autoregressive feedback mechanisms, this invention achieves a systematic improvement in speech clarity, naturalness, and stability while maintaining the simplicity of the end-to-end speech modeling framework. It can be widely applied in scenarios such as intelligent voice assistants, speech transcription, virtual anchors, speech restoration, and cross-language speech generation, and has extremely high practical application value and promotion prospects. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the overall structure of the present invention;

[0039] Figure 2 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0041] This invention provides a high-fidelity end-to-end acoustic modeling method that combines speech intelligibility reconstruction and autoregressive feedback optimization. Please refer to [link to relevant documentation]. Figures 1-2 The specific steps are as follows:

[0042] S1, Speech Input Preprocessing: Acquire the raw speech signal, perform acoustic cleaning using multi-channel spectrogram denoising and dynamic time-frequency masking algorithms, and extract the fundamental frequency of the speech. The system obtains the energy envelope and formant parameters, and segments the speech segments using adaptive endpoint detection (AED) to obtain the speech feature sequence input. Among them, the dynamic time-frequency masking algorithm is based on the time-frequency dual attention mechanism, and combines the frequency-domain gated recurrent unit (GRU) with the time-domain convolution module, which can maintain the integrity of the speech signal in an environment where the speech noise ratio is less than 10dB.

[0043] S2, Speech Feature Perception and Intelligibility Modeling: First, an intelligibility loss function is established based on the human speech intelligibility model (a combination of STOI and PESQ indices). Then, the subjective clarity features of speech are reconstructed through a spectrum reconstruction network, and intelligibility-related feature embeddings are extracted by combining a perceptual weighted filter bank. These embeddings are then input into the acoustic encoder in parallel with the Mel spectrum features. The intelligibility modeling network uses a dual-channel perceptual subnetwork structure. The first channel takes the short-time Fourier transform amplitude spectrum as input, and the second channel takes the phase spectrum residual as input. The two are used to calculate the speech clarity score through an adaptive fusion layer to achieve a more accurate semantic intelligibility estimate.

[0044] S3, High-fidelity end-to-end acoustic model training: First, an acoustic modeling network based on a hybrid Transformer and Conformer architecture is used, with speech-text pairs as training samples. Then, a multi-task loss function, an intelligibility reconstruction loss, and an autoregressive prediction loss (AR Feedback Loss) are employed. By jointly optimizing the weights of these three factors, an end-to-end mapping from speech to character sequences is achieved. The multi-task loss function includes a character / phoneme recognition loss (CTC Loss). The Conformer, combining self-attention and convolutional enhancement layers, can simultaneously capture long-range semantic dependencies and local acoustic variations. The training objective function is:

[0045]

[0046] in, These represent the dynamic weights of the three types of losses, respectively.

[0047] S4: Autoregressive Feedback Optimization: During the model inference stage, an autoregressive feedback module is introduced to recursively verify the prediction results of each frame of speech. The prediction distribution of historical frames is used to resample the input of the next frame, constructing a time-dependent feedback loop to reduce inter-frame drift error. Furthermore, a dynamic learning rate adjustment mechanism based on KL divergence is used to improve model stability. The autoregressive feedback optimization employs a two-stage mechanism:

[0048] Offline feedback correction is performed during the model training phase, and prediction drift is reduced through teacher-forcing.

[0049] The inference phase employs online feedback adjustment, correcting the current frame input by using the mean of the prediction distribution of the previous N frames, thereby enhancing prediction stability.

[0050] S5, Speech Output Reconstruction and Intelligibility Enhancement: The spectrum of the end-to-end model output is reconstructed into a speech signal, and the speech signal is reconstructed by combining a post-processing module based on an autoregressive residual network.

[0051] This invention employs a high-fidelity end-to-end acoustic modeling method that combines speech intelligibility reconstruction with autoregressive feedback optimization. This method achieves multi-dimensional optimization from the signal layer to the perception layer, solving the problems of insufficient intelligibility and prediction drift in traditional end-to-end speech modeling. It demonstrates excellent stability and scalability in low-resource, multilingual, and noisy environments, and has broad application and promotion value.

[0052] A high-fidelity end-to-end acoustic modeling system combining speech intelligibility reconstruction and autoregressive feedback optimization includes:

[0053] The speech preprocessing module is used to perform signal sampling, time-frequency transformation, noise reduction, and energy normalization;

[0054] The speech preprocessing module specifically includes:

[0055] This module is used to receive and preprocess raw speech data, extract multi-scale time-frequency features, including Mel-spectral features, cepstral coefficients (MFCC), log-energy distribution, and phase difference features. In noisy scenarios, this module achieves noise suppression through multi-channel adaptive filtering and uses a residual attention network to enhance the features of the speech signal, making the input features more discriminative and stable.

[0056] The speech perception module is primarily responsible for the acquisition, cleaning, and feature extraction of speech data. This module includes an input preprocessing unit, a noise suppression unit, and a feature encoding unit.

[0057] The input preprocessing unit employs an adaptive sampling and dynamic normalization algorithm to adjust the sampling rate and window function length based on the signal-to-noise ratio and energy distribution of the input speech, ensuring feature consistency under different speakers and different device sampling conditions.

[0058] The noise suppression unit removes environmental noise, breathing sounds, and background interference by combining multi-channel adaptive filtering with a frequency domain masking estimation algorithm.

[0059] The feature encoding unit uses a joint structure of three-layer convolution and bidirectional gated recurrent network (Bi-GRU) to extract multi-scale speech features, embedding short-term acoustic features and long-term semantic correlation into a unified representation space to provide input for subsequent modeling.

[0060] The intelligibility modeling module is used to calculate speech intelligibility metrics and generate intelligibility embeddings;

[0061] The intelligibility modeling module includes a perception analysis unit and a fusion computing unit. The perception analysis unit uses a dual-channel structure to extract features from the amplitude spectrum and the phase spectrum. The fusion computing unit uses a multi-head attention mechanism to fuse the two types of features to generate an intelligibility embedding.

[0062] The understandability modeling module specifically includes:

[0063] Based on a human auditory model and semantic representation space, a speech intelligibility constraint is constructed. Speech perceptual similarity metrics (such as STOI, PESQ, and CSIG) guide the acoustic model to prioritize preserving speech clarity and intelligibility during reconstruction. This module consists of two parallel sub-networks: an intelligibility prediction sub-network and a speech reconstruction sub-network. The former uses a contrastive learning mechanism to predict speech intelligibility scores, while the latter uses a reparameterized convolutional network for spectral recovery. The intelligibility feedback signal imposes constraints on the model generation process, achieving a balance between semantic preservation and waveform detail.

[0064] The innovation of the intelligibility reconstruction module lies in the introduction of semantic perception and intelligibility optimization mechanisms on the basis of traditional acoustic reconstruction, and the realization of "speech-semantics" consistency constraints through a two-way perception path.

[0065] The module input is the acoustic representation after feature encoding, and the output is the optimized high-fidelity spectrogram.

[0066] The module includes an intelligibility prediction subnetwork and a speech reconstruction subnetwork, which are trained collaboratively through a joint loss function;

[0067] The intelligibility prediction subnetwork is based on the Transformer architecture. It learns the mapping relationship between speech signals and text semantics through a cross-modal attention mechanism and outputs the predicted intelligibility score.

[0068] The speech reconstruction subnetwork uses a multi-scale residual convolutional network and a U-Net symmetric structure to reconstruct the distorted spectrum, ensuring a synergistic improvement in speech details and semantic clarity.

[0069] During training, a masking threshold constraint based on the human ear perception model is introduced to prevent excessive noise amplification when the model restores high-frequency details.

[0070] Through the above design, the intelligibility reconstruction module not only improves the clarity and semantic preservation of speech output, but also maintains stable semantic intelligibility in low signal-to-noise ratio environments, making the speech reconstruction results more in line with the natural auditory experience.

[0071] The acoustic modeling module is used for end-to-end speech-to-text prediction based on the Transformer-Conformer hybrid structure.

[0072] The acoustic modeling module is as follows:

[0073] An end-to-end architecture is adopted to jointly model acoustic features with target text or semantic tags. The model employs a multi-branch encoder-decoder structure, where the main encoder extracts temporal features, the auxiliary encoder captures vocal tract resonance information, and the decoder uses a combination of self-attention mechanism and gated recurrent units (GRUs) to generate continuous spectrogram outputs. This module jointly optimizes intelligibility loss, perceptual alignment loss, and spectral reconstruction loss through a multi-task learning framework, ensuring that the generated speech conforms to both physical acoustic laws and meets the requirements of language intelligibility.

[0074] The acoustic modeling module is built upon a deep self-attention structure, directly mapping speech features to target text or spectrograms. This module consists of a multi-layer encoder, decoder, and joint aligner.

[0075] The encoder part combines multi-head attention with a convolutional feedforward network to capture long-distance speech dependencies, while introducing positional coding to enhance temporal sensitivity;

[0076] The aligner part adopts an improved time alignment algorithm based on a hybrid mechanism of CTC and Transformer to achieve adaptive correspondence between speech frames and text tags;

[0077] The decoder combines Conditional Random Field (CRF) and Gated Cyclic Unit (GRU) to generate target features frame by frame through a context gating mechanism, ensuring spectral continuity and semantic consistency.

[0078] The module employs joint loss during the training phase, including: intelligibility loss, spectral reconstruction loss, and perceptual alignment loss, to ensure the model's comprehensive performance under multi-dimensional objectives.

[0079] The autoregressive feedback module is used to correct the prediction input of the current speech frame by using the historical prediction distribution, thereby achieving feedback self-optimization.

[0080] The autoregressive feedback module includes a time-series feedback unit and an error compensation unit. The time-series feedback unit recursively calculates the conditional distribution of the current prediction based on the probability distribution of historical frames, and the error compensation unit corrects the prediction error based on KL divergence and gradient variance estimation.

[0081] The autoregressive feedback module is specifically as follows:

[0082] This module is designed for dynamic error correction and temporal stability control during the generation phase. It introduces an autoregressive loop structure, feeding back the output spectrogram prediction results from the previous time step to the modeling unit of the current frame, thus establishing an autoregressive memory path. During inference, the system can automatically adjust the prediction distribution based on the feedback error signal, preventing the problem of long-sequence cumulative errors in traditional end-to-end systems. Simultaneously, the module utilizes reinforcement learning strategies to dynamically adjust the feedback step size, achieving an adaptive balance between generation speed and accuracy.

[0083] This invention introduces an autoregressive feedback mechanism into end-to-end acoustic modeling, enabling the model to adaptively adjust to time series generation errors. This module includes a feedback prediction unit and an error correction unit.

[0084] When generating each frame of the sound spectrum, the feedback prediction unit reads the prediction result of the previous frame and calculates the residual vector, and feeds the residual information back to the main modeling network to dynamically correct the prediction offset.

[0085] The error correction unit is based on a reinforcement learning strategy. It adjusts the feedback step size according to the trend of the generated error. When the accumulated error is large, the feedback frequency is increased to enhance stability.

[0086] Meanwhile, the module introduces "semantic alignment gating" during the inference phase, which automatically determines whether to trigger the feedback mechanism by analyzing the consistency of adjacent frames in the predicted semantic stream, thereby improving speech coherence without increasing the amount of computation.

[0087] Through an autoregressive feedback mechanism, this invention can significantly reduce the drift phenomenon in the generation of long-sequence speech, achieve seamless splicing of continuous speech segments, and improve the output stability and naturalness of the end-to-end model.

[0088] The speech reconstruction module is used to reconstruct a high-fidelity speech signal from the predicted spectral signal through residual enhancement filtering and inverse spectral domain convolution.

[0089] The speech reconstruction module includes an inverse spectral domain convolutional network and a residual enhancement filter. The inverse spectral domain convolutional network is used to reconstruct the time-domain waveform from the spectrum, and the residual enhancement filter is used to correct speech distortion areas and enhance the naturalness of the speech. The residual enhancement filter adopts a two-layer gated residual structure, with each layer containing one-dimensional convolution, sigmoid gating, and skip connections. The output signal is smoothed in time and frequency by the inverse spectral domain reconstruction module to restore the details of the speech waveform to the greatest extent.

[0090] The system also includes a data self-distillation module, which generates pseudo-label data from the high-resource language model and updates the parameters of the low-resource language acoustic model to improve the system's recognition performance in dialect and noisy environments.

[0091] Based on the above, the system supports multilingual input and adaptive inference mode. It can automatically load the corresponding acoustic parameter configuration according to the input language and adjust the inference rate through an online feedback mechanism to achieve real-time speech recognition. Meanwhile, the system can be deployed in embedded or cloud environments and can be applied to smart terminals, in-vehicle voice systems, conference transcription devices, and low-resource dialect recognition scenarios.

[0092] Although the present invention has been described above with reference to embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of the invention. In particular, as long as there is no structural conflict, the features in the disclosed embodiments can be combined with each other in any manner. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A high-fidelity end-to-end acoustic modeling method combining speech intelligibility reconstruction and autoregressive feedback optimization, characterized in that, The specific steps are as follows: S1, Speech Input Preprocessing: Acquire the raw speech signal, perform acoustic cleaning using multi-channel spectrogram denoising and dynamic time-frequency masking algorithms, and extract the fundamental frequency of the speech. The speech feature sequence input is obtained by segmenting speech segments using adaptive endpoint detection, based on the energy envelope and formant parameters. S2, Speech Feature Perception and Intelligibility Modeling: First, an intelligibility loss function is established based on the human speech intelligibility model; then, the subjective clarity features of speech are reconstructed through a spectrum reconstruction network, and intelligibility-related feature embeddings are extracted by combining a perceptual weighted filter bank, and input into the acoustic encoder in parallel with the Mel spectrum features; S3, High-fidelity end-to-end acoustic model training: First, an acoustic modeling network based on a hybrid architecture of Transformer and Conformer is adopted, with speech-text pairs as training samples. Then, a multi-task loss function, intelligibility reconstruction loss, and autoregressive prediction loss are adopted. By jointly optimizing the weights of the three, end-to-end mapping from speech to character sequences is achieved. S4: Autoregressive Feedback Optimization: During the model inference stage, an autoregressive feedback module is introduced to recursively verify the prediction results of each frame of speech. The prediction distribution of historical frames is used to resample the input of the next frame to construct a time-dependent feedback loop to reduce inter-frame drift error. The model stability is improved by a dynamic learning rate adjustment mechanism based on KL divergence. S5, Speech Output Reconstruction and Intelligibility Enhancement: The spectrum of the end-to-end model output is reconstructed into a speech signal, and the speech signal is reconstructed by combining a post-processing module based on an autoregressive residual network.

2. The high-fidelity end-to-end acoustic modeling method combining speech intelligibility reconstruction and autoregressive feedback optimization as described in claim 1, characterized in that, The dynamic time-frequency masking algorithm in S1 is based on a time-frequency dual attention mechanism, which combines a frequency-domain gated recurrent unit with a time-domain convolution module. It can maintain the integrity of the speech signal in an environment where the speech-to-noise ratio is less than 10dB.

3. The high-fidelity end-to-end acoustic modeling method combining speech intelligibility reconstruction and autoregressive feedback optimization as described in claim 1, characterized in that, The intelligibility modeling network in S2 uses a dual-channel perceptual subnet structure. The first channel takes the short-time Fourier transform amplitude spectrum as input, and the second channel takes the phase spectrum residual as input. The two are used to calculate the speech clarity score through an adaptive fusion layer to achieve more accurate semantic intelligibility estimation.

4. The high-fidelity end-to-end acoustic modeling method combining speech intelligibility reconstruction and autoregressive feedback optimization as described in claim 1, characterized in that, In S3, the Conformer combines self-attention and convolutional enhancement layers, enabling it to simultaneously capture long-range semantic dependencies and local acoustic variations; and the training objective function is: in, These represent the dynamic weights of the three types of losses, respectively.

5. The high-fidelity end-to-end acoustic modeling method combining speech intelligibility reconstruction and autoregressive feedback optimization as described in claim 1, characterized in that, The autoregressive feedback optimization in S4 employs a two-stage mechanism: Offline feedback correction is performed during the model training phase to reduce prediction drift through teacher-mandated correction. The inference phase employs online feedback adjustment, correcting the current frame input by using the mean of the prediction distribution of the previous N frames, thereby enhancing prediction stability.

6. A high-fidelity end-to-end acoustic modeling system combining speech intelligibility reconstruction and autoregressive feedback optimization, characterized in that, include: The speech preprocessing module is used to perform signal sampling, time-frequency transformation, noise reduction, and energy normalization; The intelligibility modeling module is used to calculate speech intelligibility metrics and generate intelligibility embeddings; The acoustic modeling module is used for end-to-end speech-to-text prediction based on the Transformer-Conformer hybrid structure. The autoregressive feedback module is used to correct the prediction input of the current speech frame by using the historical prediction distribution, thereby achieving feedback self-optimization. The speech reconstruction module is used to reconstruct a high-fidelity speech signal from the predicted spectral signal through residual enhancement filtering and inverse spectral domain convolution.

7. A high-fidelity end-to-end acoustic modeling system combining speech intelligibility reconstruction and autoregressive feedback optimization as described in claim 6, characterized in that, The intelligibility modeling module includes a perception analysis unit and a fusion computing unit. The perception analysis unit uses a dual-channel structure to extract features from the amplitude spectrum and the phase spectrum. The fusion computing unit uses a multi-head attention mechanism to fuse the two types of features to generate an intelligibility embedding.

8. A high-fidelity end-to-end acoustic modeling system combining speech intelligibility reconstruction and autoregressive feedback optimization as described in claim 6, characterized in that, The autoregressive feedback module includes a temporal feedback unit and an error compensation unit. The temporal feedback unit recursively calculates the conditional distribution of the current prediction based on the probability distribution of historical frames, and the error compensation unit corrects the prediction error based on KL divergence and gradient variance estimation.

9. A high-fidelity end-to-end acoustic modeling system combining speech intelligibility reconstruction and autoregressive feedback optimization as described in claim 6, characterized in that, The speech reconstruction module includes an inverse spectral domain convolutional network and a residual enhancement filter. The inverse spectral domain convolutional network is used to reconstruct the time-domain waveform from the spectrum, and the residual enhancement filter is used to correct speech distortion areas and enhance the naturalness of the speech.

Images