Text-to-voice hardware acceleration system supporting dynamic input adaptation

Abstract

The invention discloses a text-to-voice hardware acceleration system supporting dynamic input adaptation, and belongs to a text-to-voice system on an embedded platform or edge computing equipment. The technical problems of poor operator compatibility, difficult calculation graph staticization and quantitative deployment precision loss existing in real-time synthesis of dynamic length voice input on a resource-limited hardware platform are solved. According to the technical principle, a fixed-length frame sequence is generated through an audio segmentation unit, and an encoder extracts Mel spectrum features; a re-parameterization technology is adopted to convert probability sampling into deterministic calculation; and in combination with static transformation of a dynamic operator and an ONNX node replacement mechanism, hardware compatibility of a model structure is realized. The method has the beneficial effects that dynamic length input real-time synthesis (RTF < = 1) is supported; the hardware reasoning speed is increased by exporting a static calculation graph; re-parameterization guarantees the voice quality; and INT8 / FP32 dual-mode quantization deployment is realized.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of text-to-speech systems on embedded platforms or edge computing devices, and particularly relates to a text-to-speech hardware acceleration system supporting dynamic input adaptation. BACKGROUND

[0002] Text-to-speech (TTS) technology has been widely used in recent years in the fields of voice assistants, smart hardware, vehicle-mounted systems, educational aids, etc. The current mainstream TTS models are based on deep neural networks, constructing an end-to-end synthesis process, which can achieve high-quality voice output on the cloud. The existing device deployment system has the following significant problems in technical implementation:

[0003] 1. Fixed input length, lack of dynamic adaptation capability: Most existing TTS models use fixed input structures (such as fixed token number or audio frame length) in the training and deployment process, which cannot directly adapt to the dynamic changes of user input length.

[0004] 2. Coupling of forward inference logic is complex, hindering module decoupling and reconstruction: Some models use highly encapsulated forward functions, making it difficult to separately export and deploy encoder / decoder, increasing the difficulty of model reconstruction and structure optimization.

[0005] 3. The system lacks a unified feedback loop and cannot dynamically optimize the output: Most existing TTS systems are one-way inference processes, lacking automatic evaluation and adjustment mechanisms for output speech quality, and cannot drive system self-optimization through feedback such as RTF value, voice quality score (such as NISQA), etc., resulting in difficulty in balancing voice quality and efficiency.

[0006] In summary, the current TTS hardware deployment system still has significant deficiencies in dynamic input processing, inference performance control, and feedback optimization mechanisms, and it is urgent to build a complete system architecture for edge devices that balances flexibility and performance to improve the deployment efficiency and running stability of TTS systems in actual applications. SUMMARY

[0007] The purpose of the present application is to provide a text-to-speech hardware acceleration system supporting dynamic input adaptation, which solves the technical problems of dynamic input processing, model structure conversion, inference performance control, and feedback optimization mechanism in the prior art.

[0008] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0009] A text-to-speech hardware acceleration system supporting dynamic input adaptation, comprising:

[0010] The performance preliminary test module detects and evaluates the hardware inference performance, model compatibility and input adaptation capability, and provides initial reference basis for subsequent parameter adjustment and model optimization, so as to ensure that the system has basic operation conditions and performance basis before being formally put into use.

[0011] The audio input unit is configured to receive raw audio and convert it into a standard sampling rate and channel number, and output it to the audio segmentation unit.

[0012] The audio segmentation unit divides the raw audio into a sequence of frames with a length of , i.e. For each frame of audio, the time length is equal to a fixed value , and the frame sequence is output to the encoder module.

[0013] The encoder module extracts the Mel spectrum features of each frame of audio and outputs the Mel features to the embedding padding module.

[0014] The embedding padding module uses a dynamic position embedding correction function for dynamic expansion, and outputs the embedded features to the decoder module.

[0015] The decoder module generates an output of a speech spectrogram from the latent variable z through a variational autoencoder or a flow-based model, and outputs the speech spectrogram to the hardware acceleration inference module.

[0016] The inference configuration module injects a set of parameters to control the model loading sequence generated by the decoder module, and provides configuration parameters to the parameter adjustment module.

[0017] The parameter adjustment module receives the configuration parameters, adjusts the generation process of the hardware acceleration inference module, sets the control parameters of the generation process, and the control parameters include .

[0018] The hardware acceleration inference module receives the static model of the encoder-decoder, calculates the total inference time and the real-time factor , and outputs it to the decoding reconstruction module.

[0019] The decoding reconstruction module converts the speech spectrogram output by the decoder into a final speech waveform signal, and outputs the synthesized speech to the performance monitoring module.

[0020] The performance monitoring module receives the synthesized speech, evaluates the model score for the generated speech, and feeds back the parameters to the parameter adjustment module.

[0021] The beneficial effects of this scheme are: ultra-low delay and efficient real-time speech synthesis are realized through hardware acceleration, performance prediction, and frame pipeline. Parameterized control and feedback-based closed-loop optimization mechanism are provided to ensure and continuously improve the naturalness and expressiveness of synthesized speech. The design of modularization, configuration-driven, and static model deployment ensures the stability, maintainability, and resource utilization efficiency of the system.

[0022] As a further improvement of the above technical solution: the decoder module contains incompatible dynamic operators , which are replaced by static equivalent operators through structural staticization transformation , that is: . , or replace the accumulation operation with a loop.

[0023] The beneficial effects of this scheme are: through operator reconstruction and accumulation operation conversion (dynamic summation → fixed for loop), the compatibility problem of hardware accelerators for dynamic computation graphs is solved.

[0024] As a further improvement of the above technical solution: when the hardware acceleration inference module runs on the RK3588 platform, it supports model conversion to RKNN format and allows to be set to or mode to control the quantization precision, and its conversion process meets the following input-output matching: .

[0025] The beneficial effects of this scheme are: to ensure the dimensional consistency of the model structure before and after conversion, combined with RTF (real-time factor) monitoring to ensure RTF < 1, meeting the demand for low-latency real-time speech generation in embedded scenarios. According to actual needs (speed priority or quality priority), dtype can be freely selected without the need to redeploy the model.

[0026] As a further improvement of the above technical solution: the encoder and decoder modules are decoupled into independent structures: and exported as and model files, respectively.

[0027] The beneficial effects of this scheme are: converting the random sampling process into a deterministic calculation that is derivable, which not only maintains the randomness of the generated model but also meets the requirements of hardware acceleration for static computation graphs.

[0028] As a further improvement of the above technical solution: the latent variable z is generated through reparameterization , where is a random disturbance term, Unit Covariance Matrix

[0029] The beneficial effect of this scheme is that decoupling the random sampling process into deterministic calculation eliminates random nodes through mathematical equivalent transformation, and the entire generation process can be represented as a static calculation graph.

[0030] As a further improvement of the above technical solution: the performance monitoring module contains an NISQA model scoring function: Where Q is the speech quality score (Q = 5) ), signal is the generated speech signal, and the scoring result is used to dynamically adjust the naturalness of the parameters to improve naturalness.

[0031] The beneficial effect of this scheme is that the generated speech is objectively quantified and scored, directly reflecting key indicators such as clarity and naturalness. The scoring result is fed back in real time, forming a "generation → evaluation → optimization" closed-loop system, continuously improving the naturalness of the speech.

[0032] As a further improvement of the above technical solution: the inference configuration module is used to load a script , which contains a parameter key-value pair configuration set, and the script parser automatically completes parameter injection and controls the model loading sequence.

[0033] The beneficial effect of this scheme is that automatic parameter injection reduces human error.

[0034] As a further improvement of the above technical solution: the encoder module performs log spectrum extraction on the original speech, and the conversion formula is: Where T is the time step, and 80 is the frequency band dimension; the is calculated by short-time Fourier transform .

[0035] The beneficial effect of this scheme is that the Mel extraction formula realizes feature standardization by fixing the frequency band dimension, solving the dynamic input adaptation problem of the TTS system.

[0036] As a further improvement of the above technical solution: the position embedding length in the embedding padding module satisfies , where is the actual sequence length, and the embedding truncation or padding operation is dynamically performed when the model is loaded: .

[0037] The beneficial effect of this scheme is that the position embedding formula realizes processing of sequences of any length by truncation / padding, adapting to audio sequences of any length.

[0038] As a further improvement of the above technical solution: the encoder module, the embedding padding module and the decoder module are model conversion modules, which automatically detect unsupported ONNX node types and replace or structure prune according to the preset mapping table.

[0039] The beneficial effects of this scheme are: to ensure that the generated ONNX model meets the hardware inference compatibility, and to improve the compatibility of the system.

[0040] Compared with the prior art, the present application has the following beneficial effects in terms of dynamic input adaptation and terminal deployment:

[0041] First, the system of the present application supports dynamic perception and processing of multi-lingual and variable-length input, improves the input adaptation capability of the model in general scenarios, and avoids model running failure or performance degradation caused by input length variation.

[0042] Second, the present application uses a structure decoupling method to split the complex speech synthesis model into independent modules for deployment, which facilitates flexible loading and dynamic calling in hardware platforms, and avoids hardware incompatible nodes through static replacement method to improve the model conversion success rate.

[0043] Third, the present application introduces a speech style parameter adjustment and performance feedback mechanism, which can dynamically adjust the speech naturalness, speed and intonation control parameters while ensuring real-time performance, generate speech results that better meet user expectations, and have high-quality audio output capability.

[0044] Fourth, the present application realizes real-time inference effect of RTF≤0.36 in Rockchip NPU platform deployment experiment, which verifies the performance advantage of system structure optimization and inference path reconstruction in edge platform. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is a decoder signal processing framework based on normalization and residual structure;

[0046] Figure 2 is a text-to-speech hardware acceleration system block diagram supporting dynamic input adaptation. DETAILED DESCRIPTION

[0047] The present application will be further described below in combination with the drawings and specific embodiments.

[0048] Embodiment 1: Dynamic input adaptation and hardware acceleration deployment method based on Whisper model.

[0049] This embodiment aims at the problems of OpenAI Whisper model structure solidification, strong dependence on fixed input length, and original model not suitable for RKNN inference. A complete structure decoupling, input adaptive preprocessing, and inference closed-loop optimization process are designed.

[0050] (I) Audio input and preprocessing:

[0051] The system first receives external input speech signal x(t), allowing the input length T to vary widely (e.g., 3 seconds to 60 seconds), and the sampling rate is standardized to .

[0052] After channel calibration, resampling, and amplitude normalization processing, the input audio enters the segmentation link. Set the frame length , window step , and generate sub-sections according to the overlapping sliding window strategy:

[0053] ;

[0054] The processed each sub-speech will be sent into the feature extraction process.

[0055] (II) Mel spectrum feature and input mapping:

[0056] The log Mel spectrum is calculated using the official configuration of Whisper, with a frame length of 25 ms, a frame shift of 10 ms, a Hann window function, and a spectrum dimension of 80. The spectrum matrix is generated for each frame of audio. If (e.g., 300), padding is performed; if , truncation is performed. After dynamic input mapping, the model embedding layer needs to reconstruct the position encoding:

[0057]

[0058] where , d is the embedding dimension (e.g., 512).

[0059] (III) Model structure modularization export:

[0060] The system disassembles the Whisper model into Encoder and Decoder:

[0061] ;

[0062] ;

[0063] The two parts are exported as encoder.onnx and decoder.onnx, respectively.

[0064] (IV) Incompatible node reconstruction processing:

[0065] For NonZero, Cumsum, Random and other dynamic operations in the original model:

[0066] -Cumsum is replaced by manual loop accumulation:

[0067] ;

[0068] - Replace the static random tensor Load;

[0069] -Dynamic shape is replaced by constant input;

[0070] The rewritten ONNX model successfully passes the RKNN-Toolkit2 converter verification.

[0071] (Five) Hardware deployment and inference closed-loop control:

[0072] (Six) encoder.rknn and decoder.rknn are deployed on RK3588 board, and are controlled through config.yaml:

[0073] (Seven) ;

[0074] The system records the inference time consumption and the speech duration in real time , and dynamically evaluates the real-time factor:

[0075] ;

[0076] If , adjust or reduce through the parameter adjuster to speed up generation, and finally achieve an average , meeting the terminal real-time performance.

[0077] (Six) Output speech quality score feedback:

[0078] Send the system output speech into the built-in NISQA model, and score in the interval If the score of the continuously generated sample is , trigger the parameter rollback mechanism, and roll back to the parameter group with the latest score , forming a parameter-sound quality feedback closed-loop optimization strategy.

[0079] Embodiment 2:

[0080] MMS-TTS model structure reconstruction and NPU inference optimization implementation scheme.

[0081] ​This embodiment aims at the problems of complex original structure, ONNX incompatibility and low deployment efficiency of MetaMMS-TTS model, and proposes a static graph rewriting scheme to realize high-performance speech generation of RKNN.

[0082] (I) Model rewriting and export:

[0083] The original MMS-TTS model forward function contains a large number of dynamic graphs: torch.cumsum, range, rand_like. This system disassembles forward into:

[0084] forward_encoder(input_ids,attention_mask)→μ(x),σ²(x) / / Static encoder function: input text ID→output latent variable distribution parameters;

[0085] forward_decoder(z,mask)→spectrogram→waveform / / Static decoder function: input latent variable→output spectrogram→waveform;

[0086] At the same time, the dynamic random disturbance item in the model:

[0087] ;

[0088] Change to: ε = preset tensor, saved in randn_like_latents.npy, fixed during export, to ensure ONNX stability.

[0089] (II) Node replacement and static expansion strategy:

[0090] -Range(i,j)→index=[i,i+1,...,j-1] / / Precompute index sequence;

[0091] -Cumsum→Manually implement sequence accumulation;

[0092] -attention_mask×input→All use broadcast multiplication expansion;

[0093] After the above conversion, the model exports encoder.onnx and decoder.onnx, and successfully converts to RKNN model.

[0094] (III) Board deployment and inference scheduling:

[0095] System running shell:

[0096] . / rknn_mms_tts_demoencoder.rknndecoder.rknn"The future is open."

[0097] Inference takes about 0.47 seconds, outputs 3.5 seconds of speech, RTF ≈ 0.134. Control parameters include:

[0098] ;

[0099] (Four) Feedback closed-loop score optimization:

[0100] The system will generate audio submission NISQA sub-module score:

[0101] ;

[0102] Compared with the historical average , if it continues to decline:

[0103] - Reduce sway (intonation), improve naturalness;

[0104] - Improve , enhance the consistency of the reference timbre;

[0105] The parameter adjustment module updates the YAML configuration, and the system re-inferences to iterate in this cycle. Finally, the average score of the output audio reaches 4.0, and the stability and naturalness are significantly enhanced.

[0106] In this paper, specific examples are used to illustrate the principles and implementation methods of the present application. The above examples are only used to help understand the method of the present application and its core idea. The above description is only the preferred embodiment of the present application. It should be noted that due to the limitation of language expression, there are objectively infinite specific structures. For ordinary technical personnel in this technical field, without departing from the principles of the present application, some improvements, refinements or changes can be made, or the above technical features can be combined in an appropriate way; these improvements, refinements, changes or combinations, or without improvement, directly apply the concept and technical solution of the present application to other occasions, should be regarded as the protection scope of the present application.

Claims

1. A text-to-speech hardware acceleration system supporting dynamic input adaptation, characterized in that, include: Audio input unit, used to receive raw audio. The data is then converted to a standard sampling rate and number of channels and output to the audio segmentation unit. Audio segmentation unit, which segments the original audio Frames are divided into frames with a length of sequence, that is For each frame of audio, its duration is equal to a fixed value. Output the frame sequence to the encoder module; The encoder module extracts the Mel spectrum features of each frame of audio and outputs the Mel features to the embedding and padding module. The embedding and padding module uses a dynamic position embedding correction function for dynamic expansion and outputs the embedded features to the decoder module. The decoder module generates a speech spectrogram from the latent variable z using a variational autoencoder or a flow-based model. Output the speech spectrogram to the hardware-accelerated inference module; The inference configuration module injects a set of parameters, controls the loading order of the model generated by the decoder module, and provides configuration parameters to the parameter tuning module. The parameter adjustment module receives configuration parameters, adjusts the generation process of the hardware-accelerated inference module, and sets control parameters for the generation process, including... ; The hardware-accelerated inference module receives the static model from the encoder and decoder, and calculates the total inference time and the real-time factor. The output is sent to the decoding and reconstruction module; The decoding and reconstruction module converts the speech spectrogram output by the decoder into the final speech waveform signal and outputs the synthesized speech to the performance monitoring module. The performance monitoring module receives the synthesized speech, evaluates the generated speech using the evaluation model, and feeds back the parameters to the parameter adjustment module.

2. The text-to-speech hardware acceleration system supporting dynamic input adaptation according to claim 1, characterized in that, The decoder module contains incompatible dynamic operators. It achieves this through structural static transformation. To substitute, that is: , For static equivalent operators; or using Replace the accumulation operation with a loop.

3. The text-to-speech hardware acceleration system supporting dynamic input adaptation according to claim 1, characterized in that, When the hardware-accelerated inference module runs on the RK3588 platform, it supports model conversion to RKNN format and allows... Set as or The mode controls the quantization precision, and its conversion process satisfies the following input-output matching: .

4. A text-to-speech hardware acceleration system supporting dynamic input adaptation according to claim 1, characterized in that, The encoder and decoder modules are decoupled into an independent structure: And export them separately as and Model file.

5. A text-to-speech hardware acceleration system supporting dynamic input adaptation according to claim 1, characterized in that, The latent variable z is generated through reparameterization. in For random disturbance terms, It is the unit covariance matrix.

6. A text-to-speech hardware acceleration system supporting dynamic input adaptation according to claim 1, characterized in that, The performance monitoring module includes the NISQA model scoring function: Where Q is the speech quality score, signal is the generated speech signal, and the scoring result is used for dynamic adjustment. Parameters are adjusted to enhance naturalness.

7. A text-to-speech hardware acceleration system supporting dynamic input adaptation according to claim 1, characterized in that, The inference configuration module is used to load scripts. It contains a set of parameter key-value pair configurations, and the script parser automatically injects parameters and controls the model loading order.

8. A text-to-speech hardware acceleration system supporting dynamic input adaptation according to claim 1, characterized in that, The encoder module performs a logarithmic operation on the original speech. Spectrum extraction and conversion formula: Where T is the time step and 80 is the frequency band dimension; By Short Time Fourier Transform Calculated.

9. A text-to-speech hardware acceleration system supporting dynamic input adaptation according to claim 1, characterized in that, Position embedding length in the embedding filling module satisfy ,in The actual sequence length is used, and embedding truncation or padding operations are dynamically performed when the model is loaded.

10. A text-to-speech hardware acceleration system supporting dynamic input adaptation according to claim 1, characterized in that, The encoder module, embedding and filling module, and decoder module are model conversion modules. The model conversion module automatically detects unsupported ONNX node types and performs replacement or structural pruning according to a preset mapping table.

Images