TTS audio super-division method and device based on text semantics, equipment and medium

Abstract

The invention discloses a TTS audio super-division method and device based on text semantics, equipment and a medium, and relates to the technical field of audio super-division, and the method comprises the steps: obtaining an initial TTS audio and initial text data, and carrying out the preprocessing of the initial TTS audio and initial text data, so as to obtain a preprocessed audio and a preprocessed text; performing feature extraction on the preprocessed text to obtain semantic features and rhythm features, and performing feature extraction on the preprocessed audio to obtain audio features; and fusing the semantic feature, the rhythm feature and the audio feature to obtain a fused feature, and generating a target TTS audio corresponding to the initial TTS audio by using the fused feature to realize audio super-division of the initial TTS audio. By introducing text features and rhythm features into audio super-division, the problem of semantic expression distortion caused by a pure audio driving method is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio super-resolution, and particularly relates to a TTS (Text To Speech) audio super-resolution method and device based on text semantics, equipment and a medium. BACKGROUND

[0002] Audio super-resolution is a key technology for recovering high-quality audio from low-quality audio signals using deep learning techniques, and is widely used in voice communication, music production and other fields.

[0003] The current audio super-resolution method uses a pure audio driving framework to optimize high-frequency details based only on the spectral features of low-frequency audio. The high-resolution audio generated by this method has the problem of expression distortion. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a TTS audio super-resolution method and device based on text semantics, which can solve the problem of semantic expression distortion caused by the pure audio driving method by introducing text features and prosody features into audio super-resolution. The specific scheme is as follows:

[0005] In a first aspect, the present application provides a TTS audio super-resolution method based on text semantics, applied to a target audio super-resolution model, comprising:

[0006] Obtaining an initial TTS audio and initial text data corresponding to the initial TTS audio, and pre-processing the initial TTS audio and the initial text data to obtain corresponding pre-processed audio and pre-processed text;

[0007] Extracting features from the pre-processed text to obtain corresponding semantic features and prosody features, and extracting features from the pre-processed audio to obtain corresponding audio features;

[0008] Fusing the semantic features, prosody features and audio features to obtain fused features, and generating a target TTS audio corresponding to the initial TTS audio using the fused features to achieve audio super-resolution of the initial TTS audio.

[0009] Optionally, the prosody features include stress weight, tone change trend and pause marker; the stress weight represents the degree of emphasis of the pronunciation of the words in the text, the tone change trend represents the tone change trend of the sentences in the text, and the pause marker represents the pause position of the voice corresponding to the text.

[0010] Optionally, the pre-processing of the initial TTS audio and the initial text data comprises:

[0011] determine a target frame length and a target frame shift corresponding to the initial TTS audio, frame and window the initial TTS audio by using a short-time Fourier transform, the target frame length and the target frame shift, to achieve preprocessing of the initial TTS audio;

[0012] perform word segmentation on the initial text data, determine target weights corresponding to each word in the initial text data by using a target weight calculation algorithm, and determine keywords in the initial text data according to the target weights;

[0013] label each of the keywords, and map sentiment labels in the initial text data into one-hot vectors, to achieve preprocessing of the initial text data; wherein the keywords are words whose corresponding target weights are greater than a preset weight threshold.

[0014] Optionally, the feature extraction on the preprocessed text to obtain corresponding semantic features and prosody features comprises:

[0015] perform feature extraction on the preprocessed text to obtain corresponding semantic features, and predict the prosody features according to the semantic features and the one-hot vectors.

[0016] Optionally, the fusion of the semantic features, the prosody features and the audio features comprises:

[0017] perform normalization processing on the semantic features, the prosody features and the audio features respectively to obtain normalized semantic features, normalized prosody features and normalized audio features;

[0018] calculate an association weight between the normalized audio features and the normalized semantic features based on an attention weighting mechanism, weight the normalized semantic features by using the association weight to obtain corresponding weighted semantic features, and concatenate the normalized audio features, the normalized prosody features and the weighted semantic features.

[0019] Optionally, the generation of the target TTS audio corresponding to the initial TTS audio by using the fused features comprises:

[0020] compress and encode the fused features by using a preset encoder to obtain corresponding compressed features, and perform denoising processing on the compressed features to obtain corresponding denoised features;

[0021] perform frequency spectrum reconstruction on the denoised features by using a preset decoder to obtain a target mel spectrum, and perform phase recovery and time domain reconstruction on the target mel spectrum to obtain the target TTS audio.

[0022] Optionally, after the target TTS audio corresponding to the initial TTS audio is generated by using the fused features, the method further includes:

[0023] performing volume normalization processing on the target TTS audio to obtain a corresponding normalized audio, and performing format conversion on the normalized audio, so as to perform video dubbing by using the normalized audio after format conversion.

[0024] In a second aspect, the present application provides a TTS audio super-resolution device based on text semantics, applied to a target audio super-resolution model, including:

[0025] a data preprocessing module, configured to obtain an initial TTS audio and initial text data corresponding to the initial TTS audio, and perform preprocessing on the initial TTS audio and the initial text data to obtain corresponding preprocessed audio and preprocessed text;

[0026] a feature extraction module, configured to perform feature extraction on the preprocessed text to obtain corresponding semantic features and prosodic features, and perform feature extraction on the preprocessed audio to obtain corresponding audio features;

[0027] an audio super-resolution module, configured to fuse the semantic features, the prosodic features and the audio features to obtain fused features, and generate a target TTS audio corresponding to the initial TTS audio by using the fused features, so as to realize audio super-resolution of the initial TTS audio.

[0028] In a third aspect, the present application provides an electronic device, including:

[0029] a memory, configured to save a computer program;

[0030] a processor, configured to execute the computer program to realize the aforementioned TTS audio super-resolution method based on text semantics.

[0031] In a fourth aspect, the present application provides a computer readable storage medium, configured to save a computer program, and the computer program is executed by a processor to realize the aforementioned TTS audio super-resolution method based on text semantics.

[0032] The application first acquires initial TTS audio and initial text data corresponding to the initial TTS audio, and pre-processes the initial TTS audio and the initial text data to obtain corresponding pre-processed audio and pre-processed text, then extracts features from the pre-processed text to obtain corresponding semantic features and prosodic features, and extracts features from the pre-processed audio to obtain corresponding audio features, and finally fuses the semantic features, the prosodic features and the audio features to obtain fused features, and generates target TTS audio corresponding to the initial TTS audio by using the fused features, to realize audio super-resolution of the initial TTS audio. As can be seen, the application introduces text information into the audio super-resolution process by extracting semantic features and prosodic features of the text, so that the system can perceive the semantic content and the prosodic pattern of the text; by fusing the semantic features, the prosodic features and the audio features extracted from the audio, the generation process is optimized according to both the audio quality and the text semantics; finally, the target audio is generated based on the fused features, so that the tone, the stress and the text semantics of the generated audio are consistent while the audio quality is improved, solving the semantic expression distortion problem caused by the pure audio driven method. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on the provided drawings.

[0034] Figure 1 A TTS audio super-resolution method based on text semantics disclosed by the present application is shown in the flow chart;

[0035] Figure 2 An audio super-resolution flow chart disclosed by the present application is shown in the flow chart;

[0036] Figure 3 A TTS audio super-resolution flow chart based on text semantics disclosed by the present application is shown in the flow chart;

[0037] Figure 4 A TTS audio super-resolution device structure diagram based on text semantics disclosed by the present application is shown in the flow chart;

[0038] Figure 5 An electronic device structure diagram disclosed by the present application is shown in the flow chart. DETAILED DESCRIPTION

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] Current audio super-resolution methods suffer from distortion in the generated high-resolution audio. To address this, this application provides a text-based semantic TTS audio super-resolution method. By incorporating textual and prosodic features into the audio super-resolution, it solves the semantic distortion problem caused by purely audio-driven methods.

[0041] See Figure 1 As shown, this embodiment of the invention discloses a text-based TTS audio super-resolution method, applied to a target audio super-resolution model, including:

[0042] Step S11: Obtain the initial TTS audio and the initial text data corresponding to the initial TTS audio, and preprocess the initial TTS audio and the initial text data to obtain the corresponding preprocessed audio and preprocessed text.

[0043] The core approach of this embodiment is to add a "text semantic-prosodic guidance module" and a "prosodic consistency constraint mechanism" to the original AudioLBM architecture. This addresses existing problems through three key actions: extracting text semantic features and corresponding target prosodic features to provide "semantic anchors" for super-resolution; fusing text-prosodic features with audio spectral features, allowing the generator to perceive text semantic requirements when optimizing high-frequency details; and adding a prosodic consistency loss function to forcibly constrain the prosodic match between the super-resolution audio and the text semantics. Specifically, the text semantic-prosodic guidance module provides a reference for "which prosodices to retain," while the prosodic consistency constraint mechanism ensures that "the actual retained prosodices meet the requirements." Together, they improve sound quality without compromising the consistency between "text semantics → vocal prosodics."

[0044] The original AudioLBM was a latent diffusion model (LDM) for general audio super-resolution, with the core objective of "recovering high-frequency details from low-frequency audio to improve sound quality." Its complete architecture, module interactions, and data flow are as follows: Figure 2 As shown, the process includes: acquiring low-resolution TTS audio, preprocessing the audio, and extracting Mel-spectral features of the audio to generate low-dimensional latent variable features.

[0045] Addressing the core issue of "impaired semantic-prosodic consistency" in TTS scenarios, this embodiment adds two core modules (feature fusion module and text semantic-prosodic guidance module) and improves one key module (loss function module) on the original AudioLBM architecture, while maintaining the main structure of the original model (reducing complexity). The updated model (i.e., the target audio super-resolution model) performs audio super-resolution as follows: Figure 3 As shown, the process includes: preprocessing the audio and text separately to obtain audio features, semantic features and prosodic features; combining the features to generate fused features; generating high-resolution target TTS audio based on the fused features; and finally updating the loss function based on the high-resolution reference audio and backpropagating to optimize all module parameters.

[0046] It should be noted that the text semantic-prosodic guidance module is the core of "semantic-prosodic binding". It adopts a lightweight structure of "preprocessing + dual-branch extraction". Its core goal is to extract "semantic anchors" and "target prosodic references" from the text, providing a basis for "which prosodices to retain" for super-resolution, ensuring low latency and low complexity.

[0047] Core inputs: the original text corresponding to TTS (e.g., "Did you really go?"), and optional sentiment tags (e.g., "questioning", "joyful", "serious");

[0048] Core outputs: text semantic features (dimension [T×768]) and target prosodic features (dimension [T×3], including stress weights, F0 trends, and pause markers).

[0049] In addition, the specific details of the loss function module are as follows:

[0050] The original module only constrained "sound quality similarity" and could not guarantee prosodic consistency. Therefore, a "prosodic consistency loss term" was added. The total loss function takes into account both sound quality optimization and prosodic preservation to avoid "clear sound but semantic distortion".

[0051] Total loss function formula: TotalLoss = ×SpectralLoss + ×PerceptualLoss + ×ProsodyLoss;

[0052] Detailed explanation of each loss item:

[0053] Original spectrum MSE loss (Spectral Loss):

[0054] Calculation logic: Compare the generated HR Mel spectrum, i.e., the MSE values ​​of the high-resolution Mel spectrum and the reference HR Mel spectrum at each time step and frequency point, and take the average value;

[0055] Core function: To ensure the spectral similarity of the super-resolution audio, laying the foundation for sound quality;

[0056] Weight settings: =1.0, constraint objective: the MSE of the generated spectrum and the reference spectrum ≤1e-3.

[0057] Existing VGGish Perceptual Loss:

[0058] Calculation logic: Input the generated HR audio and the reference HR audio into the pre-trained VGGish model (input sampling rate 16kHz), extract the features of the 4th layer convolution (dimension 128), calculate the L2 loss of the features and take the mean;

[0059] Core function: To ensure the naturalness of the sound after super-resolution, and to avoid audio with similar spectrum but strange sound.

[0060] Weight settings: =0.3, constraint objective: perceptual loss ≤0.1.

[0061] Added prosodic consistency loss (ProsodyLoss):

[0062] Actual prosodic feature extraction (P_pred): Three classes of features are extracted from the generated HR audio using the Liborosa tool and aligned with the target prosodic features (P_gt):

[0063] Fundamental frequency F0: After extraction, the interpolation mapping is -1 / 0 / 1, corresponding to the F0 trend;

[0064] Frame energy: after normalization, it is mapped to 0-1, corresponding to accent weights;

[0065] Silent segment detection: Frames with energy ≤ threshold are marked as 1, corresponding to a pause marker;

[0066] Time step alignment: The Dynamic Time Warping (DTW) algorithm is used to ensure that the time steps of P_pred and P_gt are aligned (error ≤ 5ms).

[0067] Computational logic: L1 loss is used, focusing on stress weights and F0 trend (the core prosodic dimension of semantic delivery), the formula is: ProsodyLoss = (1 / T) × [ |w_pred(t) - w_gt(t)| + |f0_pred(t) -f0_gt(t)|], where w = accent weight, f0 = F0 trend, and t is the time step.

[0068] Core function: Forced super-resolution audio rhythm and text semantic matching to avoid problems such as pitch shifting from rising to level and weakening of accents;

[0069] Weight settings: =0.2, to avoid excessive constraints that could limit sound quality optimization. Constraint target: Prosody preservation accuracy ≥90%.

[0070] The above |w_pred(t) represents the predicted stress weight at time step t. Source: One of the actual prosodic features extracted from the generated high-resolution audio. Specifically, the extraction method uses the Librosa tool to calculate the energy (or intensity) of the audio frame, then normalizes it to a range of 0 to 1. A higher value indicates that the corresponding speech part of that frame is more likely to be stressed. Goal: When generating audio, the model attempts to make w_pred(t) close to w_gt(t).

[0071] The above w_gt(t) represents the target (true) stress weight at time step t. Source: This is obtained from the target prosodic features output by the text semantic-prosodic guidance module. This module predicts the stress weights that should exist at each time step based on the input raw text and sentiment tags (e.g., higher weights for time steps corresponding to keywords and lower weights for non-keywords). It represents the stress pattern required by the text semantics. Function: As the target value that w_pred(t) needs to approximate.

[0072] The above f0_pred(t) represents the predicted fundamental frequency (F0) trend at time step t. Source: This is one of the actual prosodic features extracted from the generated high-resolution (HR) audio. Specifically, the extraction method uses the Librosa tool to calculate the fundamental frequency (F0) of the audio, then interpolates it and quantizes it into discrete values: -1 represents a falling tone, 0 represents a level tone, and 1 represents a rising tone. This reflects the intonation fluctuations of the speech. Goal: When generating audio, the model attempts to make f0_pred(t) close to f0_gt(t).

[0073] The above f0_gt(t)| represents the fundamental frequency (F0) trend of the target (true) at time step t. Source: This is obtained from the target prosodic features output by the text semantic-prosodic guidance module. This module predicts the F0 trend (-1, 0, or 1) that should exist at each time step based on the input raw text (e.g., punctuation marks: question marks usually correspond to rising intonation) and sentiment tags (e.g., the sentiment of "question" requires rising intonation at the end of a sentence). Function: As the target value that f0_pred(t) needs to approximate.

[0074] In addition, to ensure compatibility with the output of the newly added modules, this embodiment makes lightweight modifications to the three modules of the original AudioLBM without changing the core network structure and algorithm logic:

[0075] The VQ (Vector Quantization) - AE (Autoencoder) latent variable encoding module expands the input dimension from "T×80" to "T×851" (fusion feature dimension), and simultaneously adjusts the number of input layer convolutional kernels (from 64 to 128), keeping the channel number / input dimension ratio consistent. The core structure of other convolutional layers, VQ codebook, etc., remains unchanged.

[0076] Diffusion generator module: The input / output dimension is expanded from "T×256" to "T×1027" (1027=256+768+3). The number of input and output channels of U-Net is adjusted to 256 (maintaining the feature compression ratio of the original model). The intermediate layer self-attention module and convolutional structure are completely retained.

[0077] VQ-AE decoding module: The input dimension is expanded from "T×256" to "T×1027", the number of input channels of the deconvolution layer is adjusted from 64 to 128, the number of output channels remains at 80 (corresponding to 4F=320 dimensions of the HR Mel spectrum), and the logic of the Griffin-Lim phase recovery algorithm remains unchanged.

[0078] As is understandable, this embodiment requires training the target audio super-resolution model:

[0079] Training data construction (TTS scenario-specific dataset): To adapt to the "text-derived speech" characteristic of TTS and avoid adaptation bias caused by using natural speech data, a dedicated training set is constructed:

[0080] LR TTS audio: 150 hours, downsampled to 16kHz (16bit, mono) by linear interpolation from HR TTS audio, covering the output of three mainstream TTS models: Tacotron 2, VITS, and FastSpeech 2;

[0081] Text + Emotion Tags: 1.5 million, one-to-one correspondence with LR TTS audio, covering video dubbing scenarios such as news broadcasts, emotional dialogues, advertising voice-overs, and narration. The text contains complete punctuation, and the emotion tags are marked with 5 core emotions.

[0082] HR TTS audio: 150 hours of original high-fidelity TTS audio (48kHz, 24bit, mono), signal-to-noise ratio ≥45dB, no background noise, used as a super-resolution reference;

[0083] Manually annotated prosodic data: 100,000 records, manually annotated with "stress position, F0 trend, and pause position" from HR TTS audio, with an annotation accuracy of time step error ≤10ms and stress annotation accuracy ≥95%, used for pre-training the text semantic-prosodic guidance module.

[0084] The entire training process (server-side execution, phased training):

[0085] Phase 1: Pre-trained text semantic-prosodic guidance module (independent training, completed in 3 days);

[0086] Data loading: Load 100,000 data entries of "text + sentiment tags + manually annotated prosody", and divide them into training set and validation set in an 8:2 ratio;

[0087] Model initialization: Initialize the parameters of the lightweight BERT (Bidirectional Encoder Representations from Transformers, a pre-trained language model) model and MLP (Multilayer Perceptron) network using the Xavier initialization method;

[0088] Training configuration: Using the AdamW optimizer, learning rate = 5e-5, batch size = 32, training epochs = 10, with "prosodic prediction accuracy" as the core metric;

[0089] Model saving: When the prosody prediction accuracy on the validation set is ≥92%, training is stopped and the pre-trained weights of the modules are saved.

[0090] Phase 2: Joint training of the overall model (server-side distributed training, completed in 7 days);

[0091] Weight initialization: Load the original AudioLBM general audio super-resolution pre-trained weights, load the text semantic-prosodic guidance module pre-trained weights, and initialize the parameters of the feature fusion module and the improved loss function module;

[0092] Data loading: Load a 150-hour TTS text-audio pairing training set, and divide it into training batches with a batch size of 16;

[0093] Single-batch training steps: a. Text branch processing: Input text + sentiment tag → text preprocessing → text semantic features + target prosodic features; b. Audio branch processing: Input LR TTS audio → audio preprocessing → LR Mel spectral features; c. Feature fusion: Input LR Mel spectral features, text semantic features, and target prosodic features into the feature fusion module to generate fused features; d. Generation and decoding: Fusion features → VQ-AE latent variable encoding → diffusion generator denoising → VQ-AE decoding → HR Mel spectrum → Griffin-Lim algorithm → HR audio; e. Loss calculation: Calculate the spectral MSE loss, VGGish perceptual loss, and prosodic consistency loss separately, and sum them by weight to obtain the total loss; f. Backpropagation: Use the AdamW optimizer (learning rate = 1e-4, weight decay = 1e-6) to backpropagate the total loss to all modules and update the parameters;

[0094] Validation and saving: After each training round, validate the model using 500 test set data (signal-to-noise ratio ≥35dB, prosody retention rate ≥90%). After 20 training rounds, save the final model weights (model file size ≤2GB).

[0095] This embodiment first performs data preprocessing (executed on the client side, taking ≤0.02 seconds); audio preprocessing: DC offset removal and normalization are performed on the input LR TTS audio (16kHz) to ensure that the format is consistent with the training data; text preprocessing: word segmentation and punctuation retention are performed on the input text, and sentiment tags are mapped to one-hot vectors.

[0096] Accordingly, the preprocessing process for the initial TTS audio and initial text data includes: determining the target frame length and target frame shift corresponding to the initial TTS audio; performing frame segmentation and windowing on the initial TTS audio using short-time Fourier transform, target frame length, and target frame shift to achieve preprocessing of the initial TTS audio; performing word segmentation on the initial text data; determining the target weight corresponding to each word in the initial text data using a target weight calculation algorithm; determining the keywords in the initial text data based on the target weights; labeling each keyword; and mapping the sentiment tags in the initial text data to one-hot vectors to achieve preprocessing of the initial text data; wherein, keywords are words whose corresponding target weights are greater than a preset weight threshold.

[0097] Specifically, the audio preprocessing module converts time-domain LR audio (sampling rate ≤ 16kHz) into frequency-domain features to adapt to the model input. It employs STFT (Short-Time Fourier Transform) (frame length 20ms, frame shift 10ms) + Mel filtering (80-dimensional frequency), outputting standardized Mel spectral features (dimensions of [time step T × frequency dimension F]). The frame length and frame shift mentioned above can be dynamically set according to the actual application scenario; the specific data above is only an example.

[0098] The text data preprocessing process is handled by the text preprocessing submodule within the text semantic-prosody guidance module:

[0099] Word segmentation: Chinese words are segmented using the jieba precise mode, retaining punctuation marks that are strongly related to rhythm, such as question marks, exclamation marks, and commas, to avoid deviations in rhythm judgment due to the loss of punctuation.

[0100] Keyword tagging: The weight of each word is calculated using the TF-IDF (Term Frequency-Inverse Document Frequency) algorithm (i.e., the target weight calculation method), and the words with the top 20% weights are tagged as keywords (such as "good", "important", etc.) for subsequent stress prediction;

[0101] Sentiment label mapping: The five sentiment labels "question", "joy", "serious", "anger" and "neutral" are mapped to one-dimensional one-hot vectors (dimension = 5) to provide sentiment constraints for prosody prediction.

[0102] Step S12: Extract features from the preprocessed text to obtain corresponding semantic and prosodic features, and extract features from the preprocessed audio to obtain corresponding audio features.

[0103] The prosodic features in this embodiment include stress weight, tone change trend, and pause marker; stress weight represents the degree of emphasis of the pronunciation of words in the text, tone change trend represents the tone change trend of sentences in the text, and pause marker represents the pause position of the speech in the text.

[0104] In this embodiment, the pre-trained text semantic-prosodic guidance module weights are loaded, the pre-processed text plus sentiment tags are input, and the text semantic features (T×768) and target prosodic features (T×3) are output.

[0105] Inference acceleration: TensorRT FP16 precision optimization is adopted to reduce inference latency.

[0106] Audio spectral feature extraction (client / server execution, time ≤ 0.03 seconds):

[0107] Using the original AudioLBM audio preprocessing logic, the input is the normalized LR TTS audio, and the output is the LR Mel spectral features, i.e., the audio features (T×80).

[0108] The process of feature extraction from the preprocessed text is handled by the text semantic feature extraction submodule within the text semantic-prosodic guidance module:

[0109] Model selection: A lightweight pre-trained BERT model (bert-base-chinese-mini) was adopted, with only 40M parameters (70% smaller than the standard BERT), balancing semantic extraction capability and computational efficiency;

[0110] Feature extraction: Input the preprocessed text into the lightweight BERT and extract the output of the 6th hidden layer as word-level semantic features (each word corresponds to a 768-dimensional vector);

[0111] Time step alignment: Since the number of words in the text is inconsistent with the audio time step T, linear interpolation is used to adjust the time step of the semantic features to T, ensuring that it is completely matched with the time dimension of the LR Mel spectral features (alignment error ≤ 10ms).

[0112] Target prosodic feature prediction submodule:

[0113] Feature concatenation: The text semantic features (T×768) are concatenated with the sentiment one-hot vector (expanded to T×5 by time step replication) to obtain a fused input feature of T×773;

[0114] MLP Prediction Network: Employs a 2-layer fully connected neural network (MLP), with a hidden layer dimension of 256 and ReLU activation function. The output layer uses activation methods designed for the three prosodic features.

[0115] Accent weight: Output a value between 0 and 1, with a value ≥0.7 for keywords and ≤0.3 for non-keywords, using Sigmoid activation;

[0116] F0 trend (i.e., tone change trend): Output -1 (falling tone), 0 (level tone), 1 (rising tone), output 1 at the end of a question, output 0 at the end of a statement, and output -1 at the beginning of an exclamation. After Tanh activation, it is quantized into discrete values.

[0117] Pause marker: Output 0 (no pause) or 1 (pause), output 1 at comma and period positions, using Sigmoid activation (threshold = 0.5).

[0118] Module training method: independent pre-training + overall fine-tuning. First, pre-training is performed using 100,000 pieces of data consisting of "text + sentiment tags + manually annotated prosody" to ensure that the prosody prediction accuracy is ≥92%. Then, the data is integrated into the overall model for joint fine-tuning to avoid training conflicts.

[0119] Accordingly, in this embodiment, feature extraction is performed on the preprocessed text to obtain corresponding semantic features and prosodic features, including: feature extraction is performed on the preprocessed text to obtain corresponding semantic features, and prosodic features are predicted based on semantic features and one-hot vectors.

[0120] By extracting semantic features and predicting prosodic features, constraints are provided for audio super-resolution, ensuring the intonation consistency between the generated target audio and the original low-resolution TTS audio.

[0121] Step S13: The semantic features, prosodic features and audio features are fused to obtain fused features, and the fused features are used to generate the target TTS audio corresponding to the initial TTS audio, so as to achieve audio super-resolution of the initial TTS audio.

[0122] In this embodiment, the process of fusing semantic features, prosodic features, and audio features includes: normalizing the semantic features, prosodic features, and audio features respectively to obtain normalized semantic features, normalized prosodic features, and normalized audio features; calculating the association weight between the normalized audio features and the normalized semantic features based on an attention weighting mechanism; using the association weight to weight the normalized semantic features to obtain corresponding weighted semantic features; and concatenating the normalized audio features, normalized prosodic features, and weighted semantic features.

[0123] The aforementioned feature fusion process is handled by the aforementioned feature fusion module. The core objective of this module is to deeply bind audio spectral features with text-prosodic features, enabling the generator to "perceive" text semantics and target prosodic when optimizing high-frequency details. It adopts a fusion strategy of "attention weighting + dimensional concatenation" without destroying the original audio feature information.

[0124] Core inputs: LR Mel spectral features (T×F, F=80), text semantic features (T×768), target prosodic features (T×3);

[0125] Core output: Fusion features (T×851, 851=80+768+3);

[0126] Detailed processing procedure:

[0127] Feature normalization: Perform LayerNorm normalization (mean = 0, variance = 1) on the three types of input features respectively. =1e-5), to eliminate the differences in the numerical range of different modal features and avoid the suppression of certain features;

[0128] Attention weight calculation: A single-head self-attention mechanism is adopted, using LR Mel spectral features as the query (Q, dimension T×80) and text semantic features as the key (K, dimension T×768) and value (V, dimension T×768). The attention weight of the semantic features is calculated using the formula: Attention weight = Softmax( / ), where d_k=80 (the dimension of Q), the purpose of which is to give higher weight to semantic features related to the current audio frame;

[0129] Weighted semantic feature generation: The text semantic features are multiplied by the attention weights to obtain "semantic features that are strongly correlated with the audio spectrum" (dimension T×768), ensuring that the semantic information is accurately matched with the audio frame;

[0130] Dimensional concatenation: The LR Mel spectral features (T×80), weighted semantic features (T×768), and target prosodic features (T×3) are concatenated along the frequency dimension (the second dimension) to obtain the final fused feature (T×851), which simultaneously carries audio spectral information, text semantic constraints, and target prosodic requirements.

[0131] Module complexity control: The number of parameters is only 4.8M, accounting for 3% of the total number of parameters in the original model, which does not significantly increase the computational burden.

[0132] In addition, in this embodiment, generating the target TTS audio corresponding to the initial TTS audio using the fused features includes: compressing and encoding the fused features using a preset encoder to obtain the corresponding compressed features, and denoising the compressed features to obtain the corresponding denoised features; reconstructing the spectrum of the denoised features using a preset decoder to obtain the target Mel spectrum, and performing phase recovery and time domain reconstruction on the target Mel spectrum to obtain the TTS audio.

[0133] Specifically, the VQ-AE latent variable encoding module (i.e., the preset encoder) after the fusion feature input is adapted generates fusion latent variable features;

[0134] The latent variable feature input diffusion generator is integrated to perform 500 steps of denoising (originally 1000 steps, a speedup of 50%).

[0135] The denoised latent variable features are input into the adapted VQ-AE decoding module, and the output is the HR Mel spectrum (i.e. the target Mel spectrum).

[0136] The phase is recovered using the Griffin-Lim algorithm, and 48kHz, 24bit HR TTS audio is output.

[0137] Post-processing of output (executed on the client side, taking ≤0.01 seconds).

[0138] In addition, after generating the target TTS audio corresponding to the initial TTS audio using the fused features, the method further includes: performing volume normalization processing on the target TTS audio to obtain the corresponding normalized audio, and performing format conversion on the normalized audio so as to use the format-converted normalized audio for video dubbing.

[0139] In other words, it performs volume normalization on HR audio (peak value ≤ 0dB), supports MP3 / WAV format conversion, and outputs audio files that can be directly used for video dubbing.

[0140] Therefore, this application extracts semantic and prosodic features from the text and introduces text information into the audio super-resolution process, enabling the system to perceive the semantic content and prosodic pattern of the text. By fusing semantic and prosodic features with audio features extracted from the audio, the generation process is optimized based on both audio quality and text semantics. Finally, the target audio is generated based on the fused features, thereby improving sound quality while maintaining the consistency of intonation, stress, and text semantics in the generated audio, thus solving the problem of semantic expression distortion caused by pure audio-driven methods.

[0141] See Figure 4 As shown, this embodiment of the invention discloses a text-based semantic TTS audio super-resolution device, applied to a target audio super-resolution model, comprising:

[0142] The data preprocessing module 11 is used to acquire the initial TTS audio and the initial text data corresponding to the initial TTS audio, and to preprocess the initial TTS audio and the initial text data to obtain the corresponding preprocessed audio and preprocessed text.

[0143] The feature extraction module 12 is used to extract features from the preprocessed text to obtain corresponding semantic features and prosodic features, and to extract features from the preprocessed audio to obtain corresponding audio features.

[0144] The audio super-resolution module 13 is used to fuse the semantic features, the prosodic features and the audio features to obtain fused features, and use the fused features to generate the target TTS audio corresponding to the initial TTS audio, so as to achieve audio super-resolution of the initial TTS audio.

[0145] In some specific embodiments, the data preprocessing module 11 may specifically include:

[0146] An audio framing and windowing unit is used to determine the target frame length and target frame shift corresponding to the initial TTS audio, and to perform framing and windowing on the initial TTS audio using short-time Fourier transform, the target frame length, and the target frame shift to achieve preprocessing of the initial TTS audio;

[0147] The keyword determination unit is used to perform word segmentation on the initial text data, determine the target weight corresponding to each word in the initial text data using a target weight calculation algorithm, and determine the keywords in the initial text data based on the target weights.

[0148] The tag mapping unit is used to tag each of the keywords and map the sentiment tags in the initial text data into one-hot vectors to achieve preprocessing of the initial text data; wherein, the keywords are words whose corresponding target weights are greater than a preset weight threshold.

[0149] In some specific embodiments, the feature extraction module 12 may specifically include:

[0150] The feature prediction unit is used to extract features from the preprocessed text to obtain corresponding semantic features, and to predict the prosodic features based on the semantic features and the one-hot vector.

[0151] In some specific embodiments, the audio super-resolution module 13 may specifically include:

[0152] The feature normalization unit is used to normalize the semantic features, the prosodic features and the audio features respectively to obtain normalized semantic features, normalized prosodic features and normalized audio features;

[0153] The feature splicing unit calculates the association weight between the normalized audio features and the normalized semantic features based on the attention weighting mechanism, uses the association weight to weight the normalized semantic features to obtain the corresponding weighted semantic features, and splices the normalized audio features, the normalized prosodic features, and the weighted semantic features.

[0154] In some specific embodiments, the audio super-resolution module 13 may specifically include:

[0155] The feature denoising unit is used to compress and encode the fused features using a preset encoder to obtain the corresponding compressed features, and to denoise the compressed features to obtain the corresponding denoised features.

[0156] The audio acquisition unit is used to perform spectral reconstruction on the denoised features using a preset decoder to obtain the target Mel spectrum, and to perform phase recovery and time-domain reconstruction on the target Mel spectrum to obtain the target TTS audio.

[0157] In some specific embodiments, the audio super-resolution module 13 further includes:

[0158] An audio format conversion unit is used to perform volume normalization processing on the target TTS audio to obtain the corresponding normalized audio, and to perform format conversion on the normalized audio so as to use the format-converted normalized audio for video dubbing.

[0159] Furthermore, embodiments of this application also disclose an electronic device, Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0160] Figure 5 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the text-based semantic TTS audio super-resolution method disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be a computer.

[0161] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0162] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0163] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the text-based semantic TTS audio super-resolution method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0164] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned text semantic-based TTS audio super-resolution method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0165] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0166] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0167] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0168] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0169] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A text-based semantic TTS audio super-resolution method, characterized in that, Applied to target audio super-resolution models, including: Obtain the initial TTS audio and the initial text data corresponding to the initial TTS audio, and preprocess the initial TTS audio and the initial text data to obtain the corresponding preprocessed audio and preprocessed text; Feature extraction is performed on the preprocessed text to obtain corresponding semantic and prosodic features, and feature extraction is performed on the preprocessed audio to obtain corresponding audio features; The semantic features, prosodic features, and audio features are fused to obtain fused features, and the fused features are used to generate the target TTS audio corresponding to the initial TTS audio, so as to achieve audio super-resolution of the initial TTS audio.

2. The TTS audio super-resolution method based on text semantics according to claim 1, characterized in that, The prosodic features include stress weight, tone change trend, and pause markers; the stress weight represents the degree of emphasis of the pronunciation of words in the text, the tone change trend represents the tone change trend of sentences in the text, and the pause markers represent the pause positions of the speech in the text.

3. The TTS audio super-resolution method based on text semantics according to claim 1, characterized in that, The preprocessing of the initial TTS audio and the initial text data includes: The target frame length and target frame shift corresponding to the initial TTS audio are determined, and the initial TTS audio is segmented and windowed using short-time Fourier transform, the target frame length, and the target frame shift to achieve preprocessing of the initial TTS audio; The initial text data is segmented into words, and the target weights corresponding to each word in the initial text data are determined using a target weight calculation algorithm. Keywords in the initial text data are then determined based on the target weights. Each of the keywords is labeled, and the sentiment tags in the initial text data are mapped to one-hot vectors to achieve preprocessing of the initial text data; wherein, the keywords are words whose corresponding target weights are greater than a preset weight threshold.

4. The TTS audio super-resolution method based on text semantics according to claim 3, characterized in that, The step of extracting features from the preprocessed text to obtain corresponding semantic and prosodic features includes: Feature extraction is performed on the preprocessed text to obtain corresponding semantic features, and the prosodic features are predicted based on the semantic features and the one-hot vector.

5. The TTS audio super-resolution method based on text semantics according to claim 1, characterized in that, The fusion of the semantic features, the prosodic features, and the audio features includes: The semantic features, prosodic features, and audio features are respectively normalized to obtain normalized semantic features, normalized prosodic features, and normalized audio features; The association weight between the normalized audio features and the normalized semantic features is calculated based on the attention weighting mechanism. The association weight is used to weight the normalized semantic features to obtain the corresponding weighted semantic features. The normalized audio features, the normalized prosodic features and the weighted semantic features are then concatenated.

6. The TTS audio super-resolution method based on text semantics according to claim 1, characterized in that, The step of generating the target TTS audio corresponding to the initial TTS audio using the fused features includes: The fused features are compressed using a preset encoder to obtain the corresponding compressed features, and the compressed features are then denoised to obtain the corresponding denoised features. The denoised features are reconstructed using a preset decoder to obtain the target Mel spectrum, and the target Mel spectrum is then phase-recovered and reconstructed in the time domain to obtain the target TTS audio.

7. The TTS audio super-resolution method based on text semantics according to claim 1, characterized in that, After generating the target TTS audio corresponding to the initial TTS audio using the fused features, the method further includes: The target TTS audio is volume normalized to obtain the corresponding normalized audio, and the normalized audio is format converted so that the format-converted normalized audio can be used for video dubbing.

8. A text-to-speech (TTS) audio super-resolution device based on text semantics, characterized in that, Applied to target audio super-resolution models, including: The data preprocessing module is used to acquire the initial TTS audio and the initial text data corresponding to the initial TTS audio, and to preprocess the initial TTS audio and the initial text data to obtain the corresponding preprocessed audio and preprocessed text. The feature extraction module is used to extract features from the preprocessed text to obtain corresponding semantic features and prosodic features, and to extract features from the preprocessed audio to obtain corresponding audio features. The audio super-resolution module is used to fuse the semantic features, the prosodic features and the audio features to obtain fused features, and use the fused features to generate the target TTS audio corresponding to the initial TTS audio, so as to achieve audio super-resolution of the initial TTS audio.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the text semantics-based TTS audio super-resolution method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the text-based semantic TTS audio super-resolution method as described in any one of claims 1 to 7.

Images