A system for realizing audio synthesis of minority languages ​​based on neural network model

Abstract

The present invention discloses a system for realizing audio synthesis of a minority language based on a neural network model, and relates to the field of audio synthesis. The system is realized by a data acquisition module, a processing module, a synthesis module, an integration module, and a training module. The present invention has a synthesis method based on a neural network that can automatically learn audio features and reduce dependence on manual parameter adjustment, and improve the naturalness and coherence of the synthesized audio. The excellent data acquisition system can alleviate the problem of scarcity of minority language data, introduce a more advanced acoustic feature prediction model, improve the accuracy and reliability of acoustic feature prediction, and thus be able to adapt well to the characteristic differences of different minority languages. The overall audio synthesis system has excellent connection from text input to audio waveform output, thereby improving the overall efficiency and performance of the system. The audio generated by the system for audio synthesis of minority languages ​​has reached a better level in sound quality and clarity.

Description

Technical Field

[0001] The present invention relates to the field of audio synthesis, and in particular to a system for realizing audio synthesis of a minority language based on a neural network model. Background Art

[0002] In the field of audio synthesis, with the rapid development of deep learning technology, a more intelligent, fast and effective language audio synthesis system is needed. The language audio synthesis system can be simply summarized as converting text into sound, but for audio synthesis of small languages, the existing method system is not effective.

[0003] The existing audio synthesis systems for minority languages ​​may have problems such as data scarcity, complex language characteristics, and uneven resource allocation. In other words, compared with mainstream languages, the corpus of minority languages ​​is smaller and the quality is uneven, which leads to overfitting when training neural network systems, limiting the generalization ability of the model and the naturalness of the generated audio, which affects the effect of audio synthesis. In addition, traditional audio synthesis methods often rely on complex signal processing technology and a large amount of manual parameter adjustment, which is not only time-consuming and labor-intensive, but also has uneven audio effects after synthesis. In addition, the acoustic feature prediction model in the existing technology may not be able to accurately capture the unique acoustic characteristics of minority languages, resulting in deficiencies in the sound quality and clarity of the generated audio. Summary of the invention

[0004] The purpose of the present invention is to provide a system for realizing audio synthesis of minority languages ​​based on a neural network model, which solves the problems raised in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides a system for realizing audio synthesis of a minority language based on a neural network model, comprising:

[0006] Data collection module: used to collect text data of minority languages ​​and input the text data into the processing module;

[0007] Processing module: used to clean text data, perform phoneme annotation, extract features based on Mel spectrum, and output processed text data;

[0008] Synthesis module: used to input processed text data, output the encoded text representation at position t and dimension p, the predicted Mel spectrogram features at position t and feature dimension f, and the audio feature representation at the nth audio frame;

[0009] Integration module: used to integrate the text representation encoded at position t and dimension p, the Mel spectrogram features predicted at position t and feature dimension f, and the audio feature representation at the nth audio frame into the model;

[0010] Training module: used to load text data from the data acquisition module, cyclically train the model synthesized by the integration module, and regularly evaluate the performance of the model during the training process.

[0011] Optionally, the synthesis module includes an encoding submodule, a feature submodule, a generation submodule and a loop optimization submodule.

[0012] Optionally, the encoding submodule processing process is as follows:

[0013] ECU t，p =ReLU[A 1，p ×LSTM(ECUA t，p ,ECU t-1，p )+a 1，p ];

[0014]

[0015] in:

[0016] ECU t，p Refers to the text representation encoded at position t and dimension p, CCD t Refers to conditional parameters;

[0017] ECUA t，p Refers to the embedded representation of the processed text data at position t and dimension p, ECU t-1，p Refers to the text representation encoded at position t-1 and dimension p;

[0018] t refers to position, p refers to dimension, and P refers to the total number of dimensions encoded;

[0019] ReLU refers to activation function 1, LSTM refers to long short-term memory network, ⊙ refers to element-by-element multiplication, W cond，p Refers to the weight matrix, bs, that generates conditional parameters on dimension p cond Refers to CCD t The bias term of

[0020] A 1，p and a 1，p They refer to weight 1 and weight 2 of the encoding layer of dimension p respectively;

[0021] Initialize the parameters of the encoder network in the neural network model construction, and convert each element in the processed text data sequence into the embedded representation ECUA of the processed text data at position t and dimension p t，p , ECUA t，p As input, it is propagated through the encoder network and based on the activation function ReLU, the long short-term memory network LSTM outputs the encoded text representation ECU at position t and dimension p t，p, and represents the ECU based on the text encoded at the output position t and dimension p t，p Generate condition parameters CCD t .

[0022] Optionally, the feature submodule processing process is as follows:

[0023] Mel t，f =sigmoid[B 3，f ×Relu(B 4，f ×ECU t，p +CS i +b 4，f )]⊙exp[-1 / 2

[0024] (B 5，f ×SRNP t，f +b 5，f ) 2 +b 6，f ];

[0025] in:

[0026] Mel t，f Refers to the predicted Mel spectrum features at position t and feature dimension f, Mel t，f Generally refers to acoustic characteristics;

[0027] CS i Refers to the conditional vector of the i-th iteration, SRNP t，f Refers to the same feature dimension as Mel at position t and feature dimension f t，f predict relevant acoustic features;

[0028] sigmoid refers to activation function 2, exp refers to exponential function, and f refers to feature dimension;

[0029] B 3，f , B 4，f , B 5，f 、b 4，f 、b 5，f and b 6，f They refer to weight matrix 1, weight matrix 2, weight matrix 3, bias item 1, bias item 2 and bias item 3 on the feature dimension f respectively;

[0030] The text encoded at position t and dimension p represents the ECU t，p and Mel at position t and feature dimension f t，f Prediction of acoustic signatures associated with SRNP t，f Input into the feature submodule and based on the conditional vector CS of the i-th iteration i, the activation function sigmoid and the exponential function exp capture complex acoustic features and output the predicted Mel spectrum feature Mel at position t and feature dimension f t，f .

[0031] Optionally, the generation submodule processing process is as follows:

[0032] AUD n =sigmoid[C r ×(Mel t，f ⊙CCD t +c r )⊙exp(-|CL×Mel t，f +

[0033] bs| 2 )];

[0034] in:

[0035] AUD n Refers to the audio feature representation on the nth audio frame;

[0036] C r ,CL,c r and bs refer to Mel t，f and CCD t Combine to generate AUD n Bias term one, bias term two, bias term three and bias term four;

[0037] The vocoder model is constructed based on the neural network model, and the Mel spectrum feature Mel predicted at position t and feature dimension f is used. t，f and conditional parameters CCD t As input, the generation submodule outputs the audio feature representation AUD on the nth audio frame. n .

[0038] Optionally, the loop optimization submodule processing process is as follows:

[0039] First: Set the calculation formula CS i+1 =ActivationFunction(OU×[CS i ;AUD n ]+s);

[0040] Second: Set the loop termination condition:

[0041] Condition 1: The number of iterations i is 95;

[0042] Condition 2: |CS i+1 -CS i |<γ,γ=0.00042;

[0043] in:

[0044] CS i+1 Refers to the conditional vector after the i+1th iteration;

[0045] ActivationFunction refers to activation function 3, OU refers to weight matrix, s refers to adjustment factor;

[0046] [CS i+1 ;AUD n ] refers to CS i With AUD n Vector concatenation of ;

[0047] Representing AUD based on the audio features on the nth audio frame n and the conditional vector CS of the i-th iteration i The conditional vector CS of the i-th iteration is adjusted by adjusting the factor s and the weight matrix OU i The degree of change, output the conditional vector CS after the i+1th iteration i+1 , the conditional vector CS after the i+1th iteration i+1 Replace the conditional vector CS of the i-th iteration in the feature submodule i , to continuously adjust Mel t，f and AUD n , the purpose of loop termination is achieved through the loop termination condition.

[0048] Optionally, the text data includes text data and voice data;

[0049] The text data is a minority language text corpus, specifically sentences, minority language phrases and minority language words in a minority language;

[0050] The voice data is a minority language voice recording, specifically a minority language voice recording corresponding to a minority language text corpus.

[0051] Optionally, the voice data is required to have clear sound quality and be noise-free, and cover multiple voice characteristics, where the multiple voice characteristics include different genders, ages, and speaking speeds.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] 1. The iterative synthesis method based on neural network of the present invention can automatically learn audio features, reduce dependence on manual parameter adjustment, and improve the naturalness and coherence of synthesized audio.

[0054] 2. The present invention obtains the Mel spectrum feature ECUA predicted at position t and feature dimension f through the encoding submodule t，pThe encoding submodule is more flexible through the text encoding and conditional generation method of the neural network, which can handle a wider range of text inputs and more complex language phenomena, and ECU t，p It captures the long-term dependencies in the text sequence, thereby forming a deep understanding of the entire sentence or paragraph, making the overall system model flexible and scalable. Compared with existing systems that may be based on statistical or rule-based text analysis methods, this neural network model can better handle complex language phenomena.

[0055] 3. The present invention obtains the Mel spectrum feature Mel predicted at position t and feature dimension f through the feature submodule t，f This feature submodule directly predicts the Mel spectrum features, and the overall system can bypass complex acoustic models and vocoders to improve the efficiency of audio generation. The system can more accurately predict the acoustic features that match the text content by utilizing the powerful fitting capabilities of the deep learning model. The overall system model of the multi-layer neural network can capture the complex mapping relationship between text and acoustic features to generate high-quality acoustic features. This neural network model can better handle nonlinear relationships and complex data distributions, and compared with other acoustic feature prediction methods, prediction methods based on neural networks usually have higher accuracy.

[0056] 4. The present invention obtains the audio feature representation AUD on the nth audio frame by generating a submodule n This sub-module realizes the conversion from acoustic features to audio waveforms, and the system can achieve nonlinear scaling of Mel spectrogram features, thereby more finely controlling the sound quality and dynamic range of the audio. The element-by-element multiplication operation enables the Mel spectrogram features and conditional parameters to interact with each other at each time step and frequency point, enhancing the flexibility and expressiveness of audio generation. The final generated audio samples are close to natural speech in sound quality, with high audibility and comfort.

[0057] 5. The present invention iterates the parameters in the feature submodule through the loop optimization submodule, and the system can dynamically adapt to the changes in the audio synthesis process to generate audio output that is more in line with the expectations. Therefore, the discontinuity and unnaturalness in the audio synthesis can be reduced, thereby improving the overall quality of the generated audio. By introducing the iteration and feedback mechanism, the overall model can learn more about the audio characteristics of small languages ​​and enhance its generalization ability in new data or unseen scenarios, which is particularly important for the synthesis of small language audio. The internal state of the model can be adjusted more accurately through iteration, and the system can converge to the optimal solution faster, which helps to shorten the model training time and thus improve development efficiency. The iterative synthesis method based on neural networks can automatically learn audio features, reduce dependence on manual parameter adjustment and improve the naturalness and coherence of synthesized audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a flowchart of the method steps of the system for realizing audio synthesis of minority languages ​​based on the neural network model;

[0059] Figure 2 This is a schematic diagram of the overall structure of the system for realizing audio synthesis of minority languages ​​based on the neural network model;

[0060] Figure 3 This is a schematic diagram of the structure of the synthesis module in the system that implements minority language audio synthesis based on the neural network model. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] The system for synthesizing audio of minority languages ​​based on the neural network model is different from the existing synthesis system. When the existing synthesis system is used for audio synthesis of minority languages, the corpus size is smaller than that of mainstream languages, and the quality is uneven. This leads to overfitting when training the neural network model, which limits the generalization ability of the model and the naturalness of the generated audio. In addition, minority languages ​​often have unique pronunciation rules, intonation patterns, and vocabulary structures. These characteristics are often difficult to fully capture and express in existing audio synthesis systems. For example, the pronunciation of vowels and consonants in some minority languages ​​is very different from that of mainstream languages, requiring more sophisticated modeling technology.

[0063] The sub-module of this systematic method for realizing audio synthesis of minority languages ​​based on neural network models targets the unique pronunciation rules and intonation patterns of minority languages. Through a special encoder structure and conditional generation mechanism, it ensures that the model can accurately capture and express these language characteristics. It also introduces a more advanced acoustic feature prediction model and optimizes it in combination with the acoustic characteristics of minority languages, which significantly improves the accuracy and reliability of the predicted Mel spectrogram and other acoustic features. It also adopts excellent data acquisition modules and data enhancement technology, which effectively alleviates the problem of scarcity of minority language data and improves the generalization and robustness of the model.

[0064] For example, see Figures 1 to 3 This embodiment provides a system for realizing audio synthesis of a minority language based on a neural network model, including a data acquisition module, a processing module, a synthesis module, an integration module and a training module, and specifically includes the following steps:

[0065] Step 1: collect text data of minority languages ​​based on the data collection module, and input the text data into the processing module;

[0066] Step 2: Clean the text data through the processing module, perform phoneme annotation and extract features based on Mel spectrum, and output the processed text data;

[0067] Among them: in the processing module, the text data is cleaned, segmented, and phoneme-tagged, and the internal voice data is processed by sampling rate unification, denoising, framing, windowing, etc., and characteristic Mel spectrum features are extracted. The text data is specifically processed by removing irrelevant symbols and punctuation marks, segmenting, word form restoration, etc., and phoneme tagging is also required. The voice data is specifically processed by sampling rate unification, denoising, framing, windowing, etc.;

[0068] And: When building a system model for minority language audio synthesis based on a neural network model, it is necessary to design encoders, decoders, vocoders and model integration. This is the existing technology for model construction. This system is briefly described. Encoder design is to design encoder network structures, such as LSTM and Transformer, which are used to convert text or phoneme sequences into encoded representations.

[0069] Responsible for converting the input text or phoneme sequence into a high-dimensional coded representation. Decoder design is to design the decoder network structure, which is used to convert the coded representation into acoustic features, that is, Mel spectrum features. Vocoder design is to design the vocoder network structure, such as WaveNet and WaveGlow, which is used to convert acoustic features into audio waveforms. In the model construction integration, the encoder, decoder and vocoder are integrated into a complete model, and the forward propagation and backpropagation processes of the model are defined. The decoder is responsible for converting the coded representation into acoustic features, and the vocoder generates the final audio waveform based on these acoustic features, and then corresponds to the synthesis module of the present invention in the model construction process. Each submodule in this module participates in the model construction process;

[0070] Step 3: Input the processed text data into the synthesis module, output the text representation encoded at position t and dimension p, the Mel spectrum map features predicted at position t and feature dimension f, and the audio feature representation on the nth audio frame, and integrate the text representation encoded at position t and dimension p, the Mel spectrum map features predicted at position t and feature dimension f, and the audio feature representation on the nth audio frame into the model by the integration module;

[0071] Step 4: Load text data from the data acquisition module based on the training module, cyclically train the model synthesized by the integration module, and regularly evaluate the performance of the model during the training process;

[0072] The synthesis module includes an encoding submodule, a feature submodule, a generation submodule and a loop optimization submodule.

[0073] In this embodiment, the combination of multiple sub-modules in the synthesis module constitutes a complete audio synthesis process, from text input, that is, the data acquisition module, to the encoding representation, that is, the encoding sub-module, to the feature sub-module, that is, the acoustic feature prediction, and finally the generation of the audio signal, that is, the generation sub-module. This process realizes end-to-end audio synthesis without the need for additional intermediate processing steps, thereby improving the overall efficiency and performance of the system. In addition, the unique sub-module based on the present invention enables the system to learn the unique characteristics of small languages, including pronunciation rules, intonation patterns, etc. This high adaptability and flexibility enables the system to easily cope with the differences between different small languages ​​and generate audio outputs that conform to their respective language habits. In addition, based on the feature sub-module, the system can capture key information in the audio signal, such as pitch, timbre and rhythm, etc. These information are converted into high-quality audio signals under the action of the vocoder, so that the system can generate natural and realistic speech outputs.

[0074] See also Figures 1 to 3 , the encoding submodule processing process is as follows:

[0075] ECU t，p =ReLU[A 1，p ×LSTM(ECUA t，p ,ECU t-1，p )+a 1，p ];

[0076]

[0077] in:

[0078] ECU t，p Refers to the text representation encoded at position t and dimension p;

[0079] CCD t Refers to the conditional parameters, that is, ECU t，p The encoded conditional parameters;

[0080] ECUA t，p Refers to the embedded representation of the processed text data at position t and dimension p. Embedding is the process of converting text data into a numerical vector to facilitate processing by machine learning models. t，p That is, the processed text data;

[0081] P refers to the total number of dimensions encoded;

[0082] ECU t-1，p Refers to the text representation encoded at position t-1 and dimension p;

[0083] t refers to the position, which refers to the tth character in the text sequence, and p refers to the dimension, which refers to the pth element of the embedding vector;

[0084] ReLU refers to activation function 1. ReLU is used to introduce nonlinearity to help the model learn complex feature representations.

[0085] LSTM stands for Long Short-Term Memory Network. LSTM is used to process sequence data and capture long-term dependencies in text.

[0086] ⊙ refers to the element-by-element multiplication, which is used to multiply the encoded representation with the weight matrix;

[0087] W cond，p Refers to the weight matrix, bs, that generates conditional parameters on dimension p cond Refers to CCD t The bias term of

[0088] A 1，p and a 1，p They refer to weight 1 and weight 2 of the encoding layer of dimension p respectively;

[0089] Initialize the parameters of the encoder network in the neural network model construction, and convert each element in the processed text data sequence into the embedded representation ECUA of the processed text data at position t and dimension p t，p , ECUA t，p As input, it is propagated through the encoder network and based on the activation function ReLU, the long short-term memory network LSTM outputs the encoded text representation ECU at position t and dimension p t，p , and represents the ECU based on the text encoded at the output position t and dimension p t，p Generate condition parameters CCD t .

[0090] In this embodiment: This sub-model is based on the embedding layer of the model building components, and ECUA t，p Convert it into a vector representation of fixed dimension, and use sequence models such as LSTM to capture the long-term dependencies in the text to generate the encoded text representation ECUt,p. This process effectively converts text information into a numerical form that can be processed by the neural network, and based on ECU t，p Conditional parameter CCD is generated through a series of linear transformations and nonlinear activations t This conditional parameter not only contains the semantic information of the text, but also contains specific style, intonation and other features related to audio generation, providing a basis for subsequent audio synthesis;

[0091] Compared with traditional rule-based text-to-speech systems such as TTS systems, neural network-based text encoding and conditional generation methods are more flexible and robust, and can handle a wider range of text inputs and more complex language phenomena. t，p Each word or character in the input text sequence is converted into a vector in a high-dimensional space. These vectors capture the semantic and contextual information of the word. Through recurrent neural networks such as LSTM, ECU t，p It captures the long-term dependencies in the text sequence and forms a deep understanding of the entire sentence or paragraph. t As a further refinement of the encoding result, it not only contains the semantic information of the text, but also may incorporate the speaker's style, emotion and other characteristics, providing conditional constraints for subsequent acoustic feature prediction;

[0092] Through the embedding and encoding process, the system can accurately represent the semantic content of the text. The overall system model is flexible and scalable, and can adjust parameters such as the embedding dimension and the number of LSTM layers according to specific tasks. Compared with traditional statistical or rule-based text analysis methods, neural network models can better handle complex language phenomena, such as polysemy and contextual ambiguity. Compared with other neural network-based text representation methods, recurrent neural networks such as LSTM have unique advantages in processing sequence data;

[0093] ECU t，p Usually based on an encoder, it can convert the input text, phonemes or other linguistic features into a high-dimensional, neural network-friendly encoding representation, which captures the semantic and contextual information of the input data and provides a basis for subsequent audio synthesis. In the audio synthesis system, ECU t，p It can be regarded as deep feature extraction of input text or linguistic features, which are crucial for generating natural and fluent speech, and for audio synthesis of minority languages, ECU t，p The encoding method needs to be able to adapt well to the characteristics of different languages, including vocabulary, grammar, and pronunciation rules. By training the encoder, the system can learn the unique characteristics of small languages ​​and generate audio output that conforms to the language habits.

[0094] See also Figures 1 to 3 , the feature submodule processing process is as follows:

[0095] Mel t，f =sigmoid[B 3，f ×Relu(B 4，f ×ECU t，p +CS i +b 4，f )]⊙exp[-1 / 2

[0096] (B5，f ×SRNP t，f +b 5，f ) 2 +b 6，f ];

[0097] in:

[0098] Mel t，f Refers to the predicted Mel-spectrogram features at position t and feature dimension f;

[0099] CS i Refers to the conditional vector of the i-th iteration, SRNP t，f Refers to the same feature dimension as Mel at position t and feature dimension f t，f predict relevant acoustic features;

[0100] Sigmoid refers to activation function 2, which is used to limit the output between 0 and 1;

[0101] exp refers to the exponential function, f refers to the feature dimension;

[0102] B 3，f , B 4，f , B 5，f 、b 4，f 、b 5，f and b 6，f They refer to weight matrix 1, weight matrix 2, weight matrix 3, bias item 1, bias item 2 and bias item 3 on the feature dimension f respectively;

[0103] The text encoded at position t and dimension p represents the ECU t，p and Mel at position t and feature dimension f t，f Prediction of acoustic signatures associated with SRNP t，f Input into the feature submodule and based on the conditional vector CS of the i-th iteration i , the activation function sigmoid and the exponential function exp capture complex acoustic features and output the predicted Mel spectrum feature Mel at position t and feature dimension f t，f .

[0104] In this embodiment: by directly predicting the Mel spectrogram features, the system can bypass the complex acoustic model and vocoder, improve the efficiency of audio generation, and use the powerful fitting ability of the deep learning model to more accurately predict the acoustic features that match the text content. Through the multi-layer neural network, the overall system model can capture the complex mapping relationship between the text and the acoustic features, generate high-quality acoustic features, and use the powerful fitting ability of the deep learning model to achieve accurate prediction of acoustic features. By predicting the Mel spectrogram features, the interpretability of the acoustic feature level is retained;

[0105] Compared with the HMM acoustic model system in the prior art, the neural network model can better handle nonlinear relationships and complex data distributions, and compared with other acoustic feature prediction methods, the prediction method based on the neural network usually has higher accuracy and robustness;

[0106] Mel spectrogram feature Mel predicted at position t and feature dimension f t，f , which simulates the way the human ear perceives sound, and effectively reduces the dimension and extracts features from audio signals. These features are used to guide the vocoder to generate specific audio waveforms in audio synthesis. Mel spectrum features can capture key information in audio signals, such as pitch, timbre, and rhythm, thereby helping the system to generate more natural and realistic speech. For audio synthesis of small languages, this helps to improve the sound quality and intelligibility of synthesized speech. Although there are differences in pronunciation rules and speech characteristics of different languages, Mel spectrum features, as a universal acoustic feature representation method, can maintain cross-language consistency to a certain extent. This helps the system achieve smooth transition and synthesis between different languages, with a certain degree of cross-language consistency.

[0107] See also Figures 1 to 3 , the process of generating submodules is as follows:

[0108] AUD n =sigmoid[C r ×(Mel t，f ⊙CCD t +c r )⊙exp(-|CL×Mel t，f +

[0109] bs| 2 )];

[0110] in:

[0111] AUD n Refers to the audio feature representation on the nth audio frame;

[0112] C r ,CL,c r and bs refer to Mel t，f and CCD t Combine to generate AUD n Bias term one, bias term two, bias term three and bias term four;

[0113] The vocoder model is constructed based on the neural network model, and the Mel spectrum feature Mel predicted at position t and feature dimension f is used. t，f and conditional parameters CCD tAs input, the generation submodule outputs the audio feature representation AUD on the nth audio frame. n , that is, audio waveform samples.

[0114] In this embodiment: this sub-module combines the Mel spectrum feature Mel predicted at position t and feature dimension f t，f and conditional parameters CCD t , and uses nonlinear operations such as sigmoid activation function, element-by-element multiplication, exponential function, etc. to generate the final audio feature representation AUD on the nth audio frame n , that is, audio waveform samples. This process realizes the conversion from acoustic features to audio waveforms. By introducing CL and bs and exponential functions, the system can achieve nonlinear scaling of Mel spectrogram features, thereby more finely controlling the sound quality and dynamic range of the audio. The element-by-element multiplication operation enables the Mel spectrogram features and conditional parameters to interact with each other at each time step and frequency point, enhancing the flexibility and expressiveness of audio generation.

[0115] Compared with traditional synthesis systems, the audio waveform generation method based on neural networks can more directly use acoustic features to generate high-quality audio waveforms while retaining more details and dynamic changes. Through nonlinear scaling and element-by-element multiplication operations, the system can achieve fine control of audio quality and dynamic range, and the generated audio samples are close to natural speech in sound quality, with high audibility and comfort. For real-time audio synthesis systems, AUD n The generation speed and naturalness are important indicators of system performance. This system needs to be able to quickly and accurately generate speech output that meets user expectations while maintaining the natural fluency of the speech and AUD n As the output of the system, it interacts directly with the user by continuously optimizing the AUD n The generation algorithms and model parameters can improve user experience and satisfaction, and promote the widespread application and development of audio synthesis technology for small languages.

[0116] See also Figures 1 to 3 , the loop optimization submodule processing process is as follows:

[0117] First: Set the calculation formula CS i+1 =ActivationFunction(OU×[CS i ;AUD n ]+s);

[0118] Second: Set the loop termination condition:

[0119] Condition 1: The number of iterations i is 95;

[0120] Condition 2: |CS i+1 -CS i |<γ,γ=0.00042;

[0121] in:

[0122] CS i+1 Refers to the conditional vector after the i+1th iteration;

[0123] ActivationFunction refers to activation function three, where this activation function three specifically refers to the ReLU function. The ReLU function is simple and popular for its effectiveness in alleviating the gradient vanishing problem. The ReLU function can converge quickly in neural networks, and due to its linear relationship, both forward propagation and back propagation are relatively fast.

[0124] OU refers to the weight matrix, s refers to the adjustment factor;

[0125] [CS i+1 ;AUD n ] refers to CS i With AUD n Vector concatenation of ;

[0126] In neural networks and deep learning, [] is used here to represent feature concatenation, which is to connect two or more vectors or matrices along a certain dimension, and () is used to indicate the priority of mathematical operations, ensuring that operations such as matrix multiplication and addition are performed before applying activation functions;

[0127] (OU×[CS i ;AUD n ]+s) means multiplying the concatenated vector by the weight matrix OU.

[0128] Representing AUD based on the audio features of the nth audio frame n and the conditional vector CS of the i-th iteration i The conditional vector CS of the i-th iteration is adjusted by adjusting the factor s and the weight matrix OU i The degree of change, output the conditional vector CS after the i+1th iteration i+1 , the conditional vector CS after the i+1th iteration i+1 Replace the conditional vector CS of the i-th iteration in the feature submodule i , to continuously adjust Mel t，f and AUD n , the purpose of loop termination is achieved through the loop termination condition.

[0129] In this embodiment: AUD is represented by the audio features on the nth audio frame nFeedback adjusts the condition vector CS of the i-th iteration i , the system can dynamically adapt to changes in the audio synthesis process. This mechanism enables the model to generate more desired audio output based on the quality or characteristics of the generated audio. i It is continuously optimized during the iteration process, based on its Mel t，f It will also be more accurate and coherent, which helps reduce discontinuity and unnaturalness in audio synthesis and improve the overall quality of generated audio. By introducing iteration and feedback mechanisms, the model can learn more about the characteristics of minority language audio, thereby enhancing its generalization ability in new data or unseen scenarios. This is especially important for minority language audio synthesis, because minority language data resources are often relatively scarce. This iterative form can be flexibly integrated into different neural network models, such as RNNsGRUs and Transformer. In addition, with the development of technology, new optimization algorithms can be easily introduced or the iterative process can be improved to adapt to more complex audio synthesis tasks. The audio generated based on this iterative form will be closer to the real minority language pronunciation in terms of sound quality, intonation, speaking speed, etc., which will help improve the user experience and make the generated audio more natural and easy to understand.

[0130] The internal state of the model can be adjusted more accurately through iteration, and the system can converge to the optimal solution more quickly, which helps to shorten the model training time and improve development efficiency. Traditional audio synthesis methods often rely on complex signal processing technology and a large amount of manual parameter adjustment. In contrast, the iterative synthesis method based on neural networks can automatically learn audio features, reduce dependence on manual parameter adjustment, and improve the naturalness and coherence of synthesized audio. Audio synthesis systems without iteration may not be able to fully adapt to changes in audio characteristics. By introducing an iterative mechanism, the overall system can adjust the internal state in real time according to the quality of the generated audio, thereby generating higher quality audio output.

[0131] In the specific implementation process, the three sub-modules in this method are used to form a minority language audio synthesis system, by converting each element in the processed text data sequence into an embedded representation ECUA of the processed text data at position t and dimension p. t，p Input to the encoding submodule, output the text representation ECU encoded at position t and dimension p t，p and conditional parameters CCD t , and thus the text encoding and conditional generation method of the present invention through the neural network is more flexible and can handle a wider range of text inputs and more complex language phenomena, ECU t，p It captures the long-term dependencies in the text sequence and forms a deep understanding of the entire sentence or paragraph. tAs a further refinement of the encoding result, it not only contains the semantic information of the text, but also incorporates the speaker's style, emotion and other characteristics, thereby providing conditional constraints for the subsequent acoustic feature prediction. Through the embedding and encoding process, the system can accurately represent the semantic content of the text. The overall system model is flexible and scalable, and parameters such as the embedding dimension and the number of LSTM layers can be adjusted according to specific tasks. Compared with traditional statistical or rule-based text analysis methods, neural network models can better handle complex language phenomena;

[0132] By encoding the text representation ECU at position t and dimension p t，p Input to the feature submodule, output the predicted Mel spectrum feature Mel at position t and feature dimension f t，f The present invention directly predicts the Mel spectrogram features, and the overall system can bypass the complex acoustic model and vocoder to improve the efficiency of audio generation. The system can more accurately predict the acoustic features that match the text content by using the powerful fitting ability of the deep learning model. The overall system model of the multi-layer neural network can capture the complex mapping relationship between the text and the acoustic features to generate high-quality acoustic features. The neural network model can better handle nonlinear relationships and complex data distributions. Compared with other acoustic feature prediction methods, the prediction method based on the neural network usually has higher accuracy and robustness.

[0133] By taking the predicted Mel spectrum feature Mel at position t and feature dimension f t，f and conditional parameters CCD tp Input to the generation submodule, output the audio feature representation AUD on the nth audio frame n , this sub-module realizes the conversion from acoustic features to audio waveforms, and the system can realize nonlinear scaling of Mel spectrogram features, thereby more finely controlling the sound quality and dynamic range of the audio. The element-by-element multiplication operation enables the Mel spectrogram features and conditional parameters to interact at each time step and frequency point, enhancing the flexibility and expressiveness of audio generation, and the audio waveform generation method based on neural networks can more directly use acoustic features to generate high-quality audio waveforms, and retain more details and dynamic changes. The nonlinear scaling and element-by-element multiplication operation system can realize fine control of audio quality and dynamic range, and the generated audio samples are close to natural speech in sound quality, with high audibility and comfort;

[0134] Representing AUD based on the audio features on the nth audio frame n The conditional vector CS for the i-th iteration iBy performing cyclic iterations, the system can dynamically adapt to changes in the audio synthesis process to generate audio output that is more in line with expectations, which helps reduce discontinuity and unnaturalness in audio synthesis and improve the overall quality of generated audio. By introducing iteration and feedback mechanisms, the model can learn more about the characteristics of minority language audio, thereby enhancing its generalization ability in new data or unseen scenarios. This is particularly important for minority language audio synthesis. The internal state of the model can be adjusted more accurately through iteration, and the system can converge to the optimal solution more quickly, which helps shorten the model training time and improve development efficiency. The iterative synthesis method based on neural networks can automatically learn audio features, reduce dependence on manual parameter adjustment, and improve the naturalness and coherence of synthesized audio.

[0135] This allows the various sub-modules to cooperate with each other in calculations, and to perform overall cycles and iterations, so that the overall system has the effect of automatic optimization and updating, and thus better adaptability.

[0136] Example 2: Please refer to Figure 1 , Figure 2 and Figure 3 , text data includes text data and voice data;

[0137] The text data is a minority language text corpus, specifically sentences, phrases and words in a minority language; the voice data is a minority language voice recording, specifically a minority language voice recording corresponding to the minority language text corpus. The voice data is required to have clear sound quality and be noise-free, and cover multiple voice features, including: different genders, ages and speaking speeds.

[0138] In this embodiment, specific text data can be collected from books, websites, and social media to increase the breadth of data collection, thereby improving the accuracy of model creation and system construction. When collecting voice data, multi-voice characteristics must be covered. Multi-voice characteristics refer to different genders, ages, and speaking speeds, thereby further improving the stability and accuracy of the audio synthesis system for small languages.

[0139] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A system for realizing audio synthesis of a minority language based on a neural network model, characterized in that: include: Data collection module: used to collect text data of minority languages ​​and input the text data into the processing module; Processing module: used to clean the text data, perform phoneme annotation and extract features based on Mel spectrum, and output the processed text data, wherein the text data includes text data and voice data, wherein the text data is a minority language text corpus, specifically sentences, phrases and words in a minority language, and the voice data is a minority language voice recording, specifically a minority language voice recording corresponding to the minority language text corpus; Synthesis module: used to input processed text data, output the encoded text representation at position t and dimension p, the predicted Mel spectrogram features at position t and feature dimension f, and the audio feature representation at the nth audio frame; Integration module: used to integrate the text representation encoded at position t and dimension p, the Mel spectrogram features predicted at position t and feature dimension f, and the audio feature representation at the nth audio frame into the model; Training module: used to load text data from the data acquisition module, cyclically train the model synthesized by the integration module, and regularly evaluate the performance of the model during the training process; The synthesis module includes a coding submodule, a feature submodule, a generation submodule and a loop optimization submodule; The calculation formula of the encoding submodule is as follows: ECU t，p =ReLU[A 1，p ×LSTM(EQA t，p ,ECU t-1，p )+a 1，p ]: in: ECU t，p Refers to the text representation encoded at position t and dimension p, CCD t Refers to conditional parameters; ECUA t，p Refers to the embedded representation of the processed text data at position t and dimension p, ECU t-1，p Refers to the text representation encoded at position t-1 and dimension p; t refers to position, p refers to dimension, and P refers to the total number of dimensions encoded; ReLU refers to activation function 1, LSTM refers to long short-term memory network, ⊙ refers to element-by-element multiplication, W cond，p Refers to the weight matrix, bs, that generates conditional parameters on dimension p cond Refers to CCD t The bias term of A 1，p and a 1，p They refer to weight 1 and weight 2 of the encoding layer of dimension p respectively; The processing process of the encoding submodule is as follows: Initialize the parameters of the encoder network in the neural network model construction, and convert each element in the processed text data sequence into the embedded representation ECUA of the processed text data at position t and dimension p t，p , ECUA t，p As input, it is propagated through the encoder network and based on the activation function ReLU, the long short-term memory network LSTM outputs the encoded text representation ECU at position t and dimension p t，p , and represents the ECU based on the text encoded at the output position t and dimension p t，p Generate condition parameters CCD t ; The calculation formula of the feature submodule is as follows: Mel t，f =sigmoid[B 3，f ×Relu(B 4，f ×ECU t，p +CS i +b 4，f )]⊙exp[-1 / 2 (B 5，f ×SRNP t，f +b 5，f ) 2 +b 6，f ]; in: Mel t，f Refers to the predicted Mel-spectrogram features at position t and feature dimension f; CS i Refers to the conditional vector of the i-th iteration, SRNP t，f Refers to the same feature dimension as Mel at position t and feature dimension f t，f predict relevant acoustic features; sigmoid refers to activation function 2, exp refers to exponential function, and f refers to feature dimension; B 3，f , B 4，f , B 5，f 、b 4，f 、b 5，f and b 6，f They refer to weight matrix 1, weight matrix 2, weight matrix 3, bias item 1, bias item 2 and bias item 3 on the feature dimension f respectively; The processing process of the feature submodule is as follows: The text encoded at position t and dimension p represents the ECU t，p and Mel at position t and feature dimension f t，f Prediction of acoustic signatures associated with SRNP t，f Input into the feature submodule and based on the conditional vector CS of the i-th iteration i , the activation function sigmoid and the exponential function exp capture complex acoustic features and output the predicted Mel spectrum feature Mel at position t and feature dimension f t，f ; The calculation formula of the generation submodule is as follows: AUD n =sigmoid[C r ×(Mel t，f ⊙CCD t +c r )⊙exp(-|CL×Mel t，f + bs| 2 )]; in: AUD n Refers to the audio feature representation on the nth audio frame; C r ,CL,c r and bs refer to Mel t，f and CCD t Combine to generate AUD n Bias term one, bias term two, bias term three and bias term four; The processing process of generating the submodule is as follows: The vocoder model is constructed based on the neural network model, and the Mel spectrum feature Mel predicted at position t and feature dimension f is used. t，f and conditional parameters CCD t As input, the generation submodule outputs the audio feature representation AUD on the nth audio frame. n .

2. The system for realizing audio synthesis of a minority language based on a neural network model according to claim 1, characterized in that: The processing process of the loop optimization submodule is as follows: First: Set the calculation formula CS i+1 =ActivationFunction(OU×[CS i ;AUD n ]+s); Second: Set the loop termination condition: Condition 1: The number of iterations i is 95; Condition 2: |CS i+1 -CS i |<γ,γ=0.00042; in: CS i+1 Refers to the conditional vector after the i+1th iteration; ActivationFunction refers to activation function 3, OU refers to weight matrix, s refers to adjustment factor; [CS i+1 ;AUD n ] refers to CS i With AUD n Vector concatenation of ; Representing AUD based on the audio features of the nth audio frame n and the conditional vector CS of the i-th iteration i The conditional vector CS of the i-th iteration is adjusted by adjusting the factor s and the weight matrix OU i The degree of change, output the conditional vector CS after the i+1th iteration i+1 , the conditional vector CS after the i+1th iteration i+1 Replace the conditional vector CS of the i-th iteration in the feature submodule i , to continuously adjust Mel t，f and AUD n , and the loop termination purpose is achieved by setting the loop termination condition.

3. The system for realizing audio synthesis of a minority language based on a neural network model according to claim 1, characterized in that: The voice data is required to have clear sound quality and be free of noise, and to cover multiple voice characteristics, including different genders, ages, and speech speeds.

Images