Sound cloning system for real-time emotion simulation

Abstract

The invention relates to the technical field of voice, and discloses a real-time emotion simulation sound cloning system, which comprises a voice feature extraction module used for extracting key pitch, tone and spectrum features from preprocessed voice data to provide a basis for emotion modeling, the voice feature extraction module is connected with an emotion modeling module, and the emotion modeling module is connected with the voice feature extraction module. The extracted features are used as input of emotion modeling; the speech synthesis module is used for generating features according to the emotion modeling module; and the quality evaluation and optimization module is used for carrying out quality evaluation on the generated voice and optimizing the system according to an evaluation result and user feedback. High-fidelity voice is generated through the generative adversarial network and the recurrent neural network model, voice output is close to real voice in tone quality and naturalness, the common mechanical and monotonous problems in a traditional voice synthesis technology are avoided, and the advantage is particularly suitable for scenes needing real voice experience, and the method is suitable for popularization and application. Such as virtual assistants, online education and customer services.

Description

Technical Field

[0001] The invention relates to the field of speech technology, and in particular to a voice cloning system for real-time emotion simulation. Background Art

[0002] Voice cloning based on voice conversion technology uses voice conversion technology based on statistical models to convert the features of a piece of speech into the features of the target sound. Common methods include hidden Markov models (HMM) and recurrent neural networks (RNN). These technologies can achieve basic voice conversion, but often have deficiencies in naturalness and emotional expression. Speech synthesis based on generative adversarial networks (GANs). In recent years, generative adversarial networks (GANs) have been widely used in the field of speech synthesis, especially speech cloning and emotional synthesis. GAN generates high-quality speech samples through adversarial training, and has made significant progress in the naturalness and diversity of generated speech. However, the training complexity of the GAN model is high, and there are still certain challenges in processing long-term speech or complex emotional expressions. Speech synthesis based on multimodal fusion: Multimodal fusion technology combines multiple modal information such as speech, images, and text to generate richer speech expressions. Although this type of technology has made breakthroughs in improving the naturalness and emotional expression of speech, its implementation process is complex, the data demand is large, and the real-time performance is poor.

[0003] Although the speech generated by existing technologies in voice cloning is realistic, it still has shortcomings in naturalness and emotional expression. Especially when dealing with complex emotional expressions, the speech tends to appear mechanical and lacks vividness, and has limited emotional expression ability. In scenarios that require delicate emotional expression, it is often unable to accurately capture and reproduce the emotional intonation of the target voice. Although deep learning models such as GAN can generate high-quality speech, their training process is complex, consumes a lot of computing resources, and requires a large amount of labeled data for training, which to a certain extent limits the popularization and application of the technology. The real-time performance is insufficient. Many existing speech synthesis technologies perform poorly in scenarios with high real-time requirements such as real-time voice conversion or online interaction due to their high computational complexity. The generation delay is significant, affecting the user experience and data privacy and security risks. In the process of voice cloning, since the user's voice data is involved, there is a risk of data leakage during data storage and transmission. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a real-time emotion simulation voice cloning system to solve the problems of insufficient speech naturalness, limited emotion expression ability, high model training complexity, and risks of data privacy and data leakage.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: a real-time emotional simulation voice cloning system, comprising: Data collection and preprocessing module, speech feature extraction module, emotion modeling module, speech synthesis module, quality assessment and optimization module and user output interface module; The data acquisition and preprocessing module is used to obtain the user's original voice data and perform denoising, normalization, and frame processing operations. The data acquisition and preprocessing module is connected to the voice feature extraction module, and the preprocessed data is transmitted to the feature extraction module; The speech feature extraction module is used to extract key pitch, tone, and spectrum features from the preprocessed speech data to provide a basis for emotion modeling. The speech feature extraction module is connected to the emotion modeling module, and the extracted features are used as input for emotion modeling; The emotion modeling module is used to perform emotion modeling on the extracted speech features by using a generative adversarial network and a recurrent neural network to generate a speech model with specific emotion features. The emotion modeling module is connected to the speech synthesis module, and the modeled emotion speech features are transmitted to the speech synthesis module; The speech synthesis module is used to synthesize speech with emotion according to the features generated by the emotion modeling module. The speech synthesis module is connected to the user output interface module, and the generated speech will be transmitted to the user interface for user use; The quality assessment and optimization module is used to perform quality assessment on the generated speech and optimize the system according to the assessment results and user feedback. The quality assessment and optimization module is connected to the speech synthesis module and the user feedback system, and the assessment results are used to optimize the speech synthesis module. The user output interface module is used to display the generated speech to the user and provide download and play functions for interaction with the user. The user output interface module is connected to the speech synthesis module and the quality assessment module, and the final speech is displayed and played by this module.

[0006] Preferably, the data acquisition and preprocessing module includes an original data acquisition unit and a noise removal unit, wherein the original data acquisition unit is responsible for collecting the user's initial voice information, and the noise removal unit is used to eliminate environmental noise and other interfering sounds in the collected voice data.

[0007] Preferably, the data acquisition and preprocessing module further includes a data standardization unit and a data cache unit. The data standardization unit is used to standardize the preprocessed voice data so that it has a unified format and range for subsequent processing. The data cache unit is used to temporarily store the collected original voice data to prevent data loss or processing interruption. Preferably, the speech feature extraction module includes a feature extraction algorithm unit, a feature screening unit and a feature storage unit. The feature extraction algorithm unit is used to use the relevant algorithm of the convolutional neural network to extract key features from the preprocessed speech data. The feature screening unit is used to screen the extracted features and remove redundant or unimportant features. The feature storage unit is used to store the screened key speech features for subsequent module calls.

[0008] Preferably, the emotion modeling module includes an emotion learning unit and a model building unit. The emotion learning unit is used to use a generative adversarial network and a recurrent neural network to learn a large number of speech samples with emotion labels to acquire knowledge for emotion modeling. The model building unit is used to build an emotion speech model based on the learned knowledge.

[0009] Preferably, the emotion modeling module also includes a model adjustment unit and a model verification unit, wherein the model verification unit is used to verify the validity and accuracy of the constructed and adjusted emotion model, and the model adjustment unit is used to fine-tune the constructed model to improve the accuracy of emotion expression.

[0010] Preferably, the speech synthesis module includes a synthesis algorithm application unit, a sound quality optimization unit and a synthesized speech output unit. The synthesis algorithm application unit is used to synthesize speech using relevant algorithms of parameter speech synthesis technology and neural network speech synthesis technology. The sound quality optimization unit is used to optimize the sound quality of the generated speech to make it clear and natural. The synthesized speech output unit is used to output the synthesized speech to subsequent modules or users.

[0011] Preferably, the quality assessment and optimization module includes a quality assessment index setting unit and an assessment execution unit, wherein the quality assessment index setting unit is used to evaluate specific indicators of the generated speech quality, wherein the indicators include clarity, naturalness and accuracy of emotional expression, and the assessment execution unit is used to evaluate the generated speech according to the set indicators.

[0012] Preferably, the quality assessment and optimization module also includes a historical data storage unit and an optimization strategy formulation unit. The optimization strategy formulation unit is used to formulate an optimization strategy based on the assessment results to improve system performance. The historical data storage unit is used to store the results of previous quality assessments and the corresponding optimization strategies for trend analysis and experience learning.

[0013] Preferably, the user output interface module includes a voice display unit, a user operation response unit and a feedback collection unit. The voice display unit is used to display the generated voice on the user interface for the user to view intuitively. The user operation response unit is used to respond to the user's download and play operation requests. The feedback collection unit is used to collect the user's feedback information on the generated voice, and the feedback information is used for system improvement.

[0014] The present invention provides a real-time emotional simulation voice cloning system, which has the following beneficial effects: 1. The present invention can generate highly realistic speech through optimized generative adversarial networks and recurrent neural network models. The speech output is close to real-person speech in terms of sound quality and naturalness, avoiding the mechanization and monotony problems commonly found in traditional speech synthesis technology. This advantage is particularly suitable for scenarios that require a realistic voice experience, such as virtual assistants, online education, and customer service.

[0015] 2. The present invention has advanced emotion modeling capabilities and can accurately capture and reproduce the speaker's emotional intonation. This feature makes the generated speech not only highly consistent in speech content, but also can achieve accurate simulation in emotional expression, thereby enhancing the user's interactive experience. It is suitable for application scenarios that require emotional expression, such as movie dubbing and voice assistants.

[0016] 3. The present invention greatly improves the real-time performance of speech generation through optimized speech processing and generation algorithms. The system can quickly generate synthesized speech under high concurrency conditions to meet the application requirements of instant messaging, online conferencing, etc. that require real-time speech generation. This advantage overcomes the limitations of existing technologies in real-time speech generation.

[0017] 4. The present invention provides a flexible personalized customization function. Users can adjust the voice style, speaking speed and intonation according to their needs, thereby generating customized voice that meets personal or brand needs. This feature makes the system have broad application prospects in the fields of advertising dubbing, brand promotion and personalized services.

[0018] 5. The present invention adopts advanced data encryption technology and strict access control mechanism in the data processing process to ensure the security and privacy of user voice data. The system design complies with the requirements of relevant laws and regulations to prevent data leakage and abuse and fully protect the rights and interests of users.

[0019] 6. The system of the present invention is not only applicable to the traditional field of speech synthesis, but can also play a role in multiple industries such as customer service, online education, entertainment production, medical assistance, etc. Its high-quality speech generation capabilities and flexible customization options enable the system to provide targeted solutions for users in different fields, significantly improving the application effect and user experience.

[0020] 7. Through the present invention, many jobs that originally required real-person dubbing can be automatically completed by the system, thereby significantly reducing labor costs. Especially in film and television dubbing, advertising production and language teaching, the system's high efficiency and automation capabilities can also significantly improve production efficiency and reduce production time. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A system architecture diagram of a voice cloning system for real-time emotion simulation according to the present invention; Figure 2 This is a data acquisition and preprocessing module architecture diagram of a sound cloning system for real-time emotion simulation of the present invention; Figure 3 This is a diagram of the architecture of a speech feature extraction module of a voice cloning system for real-time emotion simulation according to the present invention; Figure 4 This is an architecture diagram of an emotion modeling module of a sound cloning system for real-time emotion simulation according to the present invention; Figure 5 This is a diagram of the architecture of a speech synthesis module of a voice cloning system for real-time emotion simulation according to the present invention; Figure 6 This is a quality assessment and optimization module architecture diagram of a voice cloning system for real-time emotion simulation according to the present invention; Figure 7 This is a user output interface module architecture diagram of a voice cloning system for real-time emotion simulation according to the present invention; Figure 8 This is a module relationship diagram of a voice cloning system for real-time emotion simulation according to the present invention. DETAILED DESCRIPTION

[0022] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] Implementation 1: Sound cloning system based on deep learning Please refer to the attached Figure 1 -Attached Figure 8 The embodiment of the present invention provides a voice cloning system for real-time emotion simulation, comprising: Data collection and preprocessing module, speech feature extraction module, emotion modeling module, speech synthesis module, quality assessment and optimization module and user output interface module; The data acquisition and preprocessing module is used to obtain the user's original voice data and perform denoising, normalization, and frame processing operations. The data acquisition and preprocessing module is connected to the voice feature extraction module, and the preprocessed data is transmitted to the feature extraction module; The speech feature extraction module is used to extract key pitch, tone, and spectrum features from the preprocessed speech data to provide a basis for emotion modeling. The speech feature extraction module is connected to the emotion modeling module, and the extracted features are used as input for emotion modeling; The emotion modeling module is used to perform emotion modeling on the extracted speech features using a generative adversarial network and a recurrent neural network to generate a speech model with specific emotion features. The emotion modeling module is connected to the speech synthesis module, and the modeled emotion speech features will be transmitted to the speech synthesis module; The speech synthesis module is used to synthesize speech with emotion according to the features generated by the emotion modeling module. The speech synthesis module is connected to the user output interface module, and the generated speech will be transmitted to the user interface for user use; The quality assessment and optimization module is used to assess the quality of the generated speech and optimize the system based on the assessment results and user feedback. The quality assessment and optimization module is connected to the speech synthesis module and the user feedback system, and the assessment results are used to optimize the speech synthesis module. The user output interface module is used to display the generated speech to the user and provide download and play functions for interaction with the user. The user output interface module is connected to the speech synthesis module and the quality assessment module, and the final speech is displayed and played by this module.

[0024] The data collection and preprocessing module includes an original data collection unit and a noise removal unit. The original data collection unit is responsible for collecting the user's initial voice information, and the noise removal unit is used to eliminate environmental noise and other interfering sounds in the collected voice data.

[0025] Specifically, in the data acquisition and preprocessing module, the user uploads the original voice data through the system interface. The system first preprocesses the data. The preprocessing steps include noise reduction, normalization, framing and other operations to ensure the quality and consistency of the input data. When the user uploads the voice data, the system automatically starts the preprocessing module, which removes background noise through a filtering algorithm and normalizes the audio signal to ensure that the volume and frequency of each voice segment are within the same standard range. Finally, the processed audio signal is framed to prepare for subsequent feature extraction; The original data acquisition unit captures the voice signal emitted by the user through an audio input device such as a microphone. It uses analog-to-digital conversion technology to convert continuous analog voice signals into discrete digital signals. During the acquisition process, the voice is sampled and quantized according to a certain sampling frequency and quantization accuracy to ensure that various voice features can be accurately recorded. The original voice information of the user can be accurately and comprehensively obtained, providing a rich data foundation for subsequent processing, ensuring the originality and integrity of the voice data, so that subsequent analysis and processing can be based on real, undistorted voice sources, thereby improving the performance and accuracy of the entire system; Noise removal unit, the noise removal unit first performs spectrum analysis on the collected voice signal to determine the frequency range and intensity characteristics of the noise, and then uses filtering techniques such as low-pass filtering, high-pass filtering, band-pass filtering, etc. to attenuate or remove the signal components within the noise frequency range. At the same time, an adaptive filtering algorithm may also be used to dynamically adjust the filtering parameters according to the real-time changes of the voice signal and noise to more effectively remove the noise. In addition, a statistical model-based method is also used to estimate the characteristics of the noise and subtract the estimated value of the noise from the voice signal, which significantly improves the quality and clarity of the voice data, reduces the impact of environmental noise and other interfering sounds on voice feature extraction and subsequent processing, enhances the intelligibility and recognizability of the voice signal, and provides subsequent processing modules with purer and more valuable voice data, which helps to improve the performance and accuracy of the entire system and enhance the user experience.

[0026] The data acquisition and preprocessing module also includes a data standardization unit and a data cache unit. The data standardization unit is used to standardize the preprocessed voice data so that it has a unified format and range for subsequent processing. The data cache unit is used to temporarily store the collected original voice data to prevent data loss or processing interruption.

[0027] Specifically, the data standardization unit first performs feature analysis on the preprocessed speech data to determine the distribution range and feature values ​​of the data, and then uses common standardization methods such as Z-score standardization or Min-Max standardization. Z-score standardization converts the data into a distribution with a mean of 0 and a standard deviation of 1 by subtracting the mean and dividing by the standard deviation; Min-Max standardization maps the data to a specified interval, such as [0,1]. During the standardization process, each feature dimension of the speech data is processed independently to ensure that the data of each dimension has a unified scale and range, so that speech data from different sources and features have a unified format and range, eliminating the problem of dimensional differences and excessive numerical differences between data, facilitating the subsequent processing modules to perform consistent analysis and processing of the data, improving the stability and accuracy of the algorithm, facilitating data interaction and sharing between different modules, and reducing errors and complexity caused by inconsistent data formats; Data cache unit, the data cache unit is composed of high-speed storage media (such as memory or flash memory) and has a certain storage capacity. When the original voice data is collected, they will be quickly written into the cache area. The cache unit adopts first-in-first-out (FIFO) or other cache management strategies to ensure that the newly collected data can be stored in time, and the old data is overwritten or deleted when necessary. During the data processing process, if the processing speed cannot keep up with the collection speed or the processing is interrupted, the cache unit can temporarily save the unprocessed data and continue to provide data when waiting for the processing to resume, effectively preventing data loss caused by the mismatch between data collection speed and processing speed, ensuring the continuity of data processing, and maintaining data integrity even when the system has a short-term failure or performance fluctuation, reducing repeated collection and processing due to data interruption, improving system efficiency and stability, and providing reliable data support for real-time processing and analysis of voice data to ensure the normal operation of the system.

[0028] The speech feature extraction module includes a feature extraction algorithm unit, a feature screening unit and a feature storage unit. The feature extraction algorithm unit is used to use the relevant algorithms of the convolutional neural network to extract key features from the preprocessed speech data. The feature screening unit is used to screen the extracted features and remove redundant or unimportant features. The feature storage unit is used to store the screened key speech features for subsequent module calls.

[0029] Specifically, in the speech feature extraction module, the system uses a convolutional neural network to extract key speech features, such as pitch, tone, and spectrum, from the preprocessed speech data. These features can effectively capture the details and emotional information of the speech. After the preprocessing is completed, the audio data will be passed to the speech feature extraction module, which performs multi-layer convolution operations on the audio through the CNN model to extract the frequency and time domain features of the speech layer by layer. The extracted feature vector will be passed as input to the emotion modeling module. Feature Extraction Algorithm Unit The convolutional neural network (CNN) in the feature extraction algorithm unit processes the preprocessed speech data by constructing multiple convolutional layers and pooling layers. The convolution kernel in the convolutional layer slides on the input speech data to extract features from the local area. The weight of the convolution kernel is learned through training and can capture different patterns and features in the speech data, such as pitch changes, frequency distribution, etc. The pooling layer downsamples the output of the convolutional layer to reduce the data dimension while retaining the main features. It can automatically extract representative and discriminative key features from complex speech data. These features can reflect the essential attributes and patterns of speech, improve the efficiency and accuracy of feature extraction, avoid the complexity and limitations of manual feature design, provide high-quality feature representation for subsequent processing, and help improve the performance of the entire system; The feature screening unit uses various feature evaluation indicators and algorithms to analyze the extracted features. Common methods include calculating the variance, correlation, information gain, etc. of the features. According to the set threshold or rule, it determines which features are redundant and contribute less to classification or modeling. Then, these unimportant features are removed, and only the features with significant distinguishing ability and information content are retained, which reduces the dimension and redundancy of the features, reduces the computational complexity and storage requirements, improves the efficiency and accuracy of subsequent processing, avoids the interference and misleading of the model by unimportant features, enables the model to focus more on key features, and improves the generalization ability and performance of the model. Feature Storage Unit The feature storage unit uses efficient data storage structures, such as databases, arrays or caches, to save the screened key speech features. According to the type and quantity of features, the storage space is reasonably allocated, and indexes are established for fast retrieval and calling. During the storage process, the features are also compressed and encoded to reduce the storage space occupied, providing subsequent modules with the ability to quickly access and call features, reducing the time overhead of repeated calculations and data transmission, ensuring the security and integrity of features, preventing data loss or damage, facilitating the management and updating of features, and providing support for continuous optimization and improvement of the system.

[0030] The emotion modeling module includes an emotion learning unit and a model building unit. The emotion learning unit is used to use generative adversarial networks and recurrent neural networks to learn a large number of speech samples with emotion labels and acquire knowledge for emotion modeling. The model building unit is used to build an emotion speech model based on the learned knowledge.

[0031] Specifically, the emotion modeling module uses generative adversarial networks (GANs) and recurrent neural networks (RNNs) to model speech emotions. GANs generate realistic speech samples, while RNNs capture the time series characteristics of speech to ensure that the generated speech matches real-person speech in terms of emotional expression and coherence. The extracted speech feature vectors enter the emotion modeling module. First, the GAN generator generates preliminary emotional speech based on these features, and the discriminator evaluates the authenticity and naturalness of the generated speech. The RNN module performs time series modeling on the generated speech to ensure the coherence and naturalness of emotional expression. After multiple rounds of adversarial training, highly emotional speech features are finally generated. The generative adversarial network (GAN) in the emotion learning unit consists of a generator and a discriminator. The generator tries to generate speech samples with emotional characteristics, while the discriminator determines whether the input sample is a real speech sample with emotion labels or generated by the generator. Through continuous confrontation and optimization between the two, the generator gradually learns the characteristic distribution of real emotional speech, and the recurrent neural network (RNN) can process sequence data and remember historical information, so as to better capture the temporal dynamics of emotional expression in speech. By inputting a large number of speech samples with clear emotion labels into the combined model of GAN and RNN, and repeatedly training and adjusting, the model gradually learns the expression patterns and laws of different emotions in speech, acquires the knowledge of emotion modeling, can deeply mine the emotional information in speech samples, extract complex emotional features and patterns, and provide a rich and accurate knowledge base for subsequent model construction, improve the accuracy and reliability of emotion modeling, adapt to emotional expressions of different types and intensities, and enhance the generalization ability of the model; The model building unit determines the architecture and parameters of the model based on the knowledge acquired by the emotion learning unit, selects the appropriate type, number and connection method of the neural network layer, and builds a speech model that can accurately simulate emotional expression. It uses the learned parameters such as weights and biases to process and transform the input speech features to generate output with specific emotional characteristics. During the construction process, it will continue to debug and optimize to ensure the performance and accuracy of the model.

[0032] The emotion modeling module also includes a model adjustment unit and a model verification unit. The model verification unit is used to verify the validity and accuracy of the constructed and adjusted emotion model, and the model adjustment unit is used to fine-tune the constructed model to improve the accuracy of emotion expression.

[0033] Specifically, the model adjustment unit first analyzes the performance of the model on the training data and the validation data to determine possible deficiencies and deviations, and then optimizes the performance of the model by adjusting the model's hyperparameters, such as the learning rate, the number of layers, the number of nodes, etc., or adopts more advanced optimization algorithms, such as Adagrad, Adadelta, etc., to improve the model's training process. In addition, regularization techniques, such as L1 and L2 regularization, can be introduced to prevent the model from overfitting. For complex parts with inaccurate emotional expressions, more training data may be added or data enhancement operations may be performed to enrich the model's learning materials, and the constructed model can be refined to improve the model's accuracy and adaptability to various emotional expressions, effectively avoid overfitting or underfitting of the model, and enable the model to have better generalization capabilities on new and unseen data, and gradually optimize the model's performance to make it more in line with the needs of practical applications. The model verification unit applies the constructed and adjusted emotion model to a set of independent verification data sets, which contain various speech samples with clear emotion annotations. The model's performance is measured by comparing the model's output results with the actual emotion annotations and calculating a series of evaluation indicators, such as accuracy, recall, F1 value, etc. At the same time, a visual analysis is performed to observe the model's performance in different emotion categories and its ability to handle marginal and complex emotion samples. In addition, cross-validation and other techniques may be used to increase the reliability and stability of verification, objectively and accurately evaluate the effectiveness and accuracy of the emotion model, provide a strong basis for further improvement of the model, help discover the advantages and disadvantages of the model in different emotion categories and scenarios, and provide direction for targeted optimization, to ensure that the model can reliably identify and express emotions in practical applications and improve the overall performance of the system and user experience.

[0034] The speech synthesis module includes a synthesis algorithm application unit, a sound quality optimization unit and a synthesized speech output unit. The synthesis algorithm application unit is used to synthesize speech using relevant algorithms of parameter speech synthesis technology and neural network speech synthesis technology. The sound quality optimization unit is used to optimize the sound quality of the generated speech to make it clear and natural. The synthesized speech output unit is used to output the synthesized speech to subsequent modules or users.

[0035] Specifically, the speech synthesis module generates high-quality synthesized speech based on the speech features after emotion modeling. The synthesis process combines parametric speech synthesis technology and neural network speech synthesis technology to ensure the accuracy of the sound quality and emotional expression of the generated speech. After the emotion modeling is completed, the speech features are passed to the speech synthesis module, which processes the speech features through a parametric synthesis method and optimizes the synthesized speech using a neural network to ensure high fidelity of the sound quality and delicate expression of emotions. The generated synthesized speech is then passed to the quality assessment module; The synthesis algorithm application unit first analyzes and processes the input speech features and related text information. The parametric speech synthesis technology is based on the acoustic model and the excitation model, and describes the characteristics of the speech by generating acoustic parameters, such as fundamental frequency, duration, spectrum, etc. The neural network speech synthesis technology uses deep neural networks to learn the characteristic patterns and mapping relationships of speech. The two technologies are combined to comprehensively consider the rhythm, intonation, timbre and other factors of the speech to generate a preliminary synthetic speech signal. The sound quality optimization unit significantly improves the sound quality of the generated speech, making it clearer, more natural, and smoother, reducing synthesis traces, enhancing the intelligibility and audibility of the speech, improving the user's auditory experience, making the synthesized speech more in line with the laws of human speech expression, and enhancing its fidelity and affinity; The synthesized speech output unit provides the synthesized high-quality speech accurately and timely to the subsequent modules for further processing or directly presents it to the user, meeting the requirements of different systems and devices for speech data format, achieving wide application and compatibility, ensuring the stability and fluency of speech output, and improving the performance of the entire system and user satisfaction.

[0036] The quality assessment and optimization module includes a quality assessment index setting unit and an assessment execution unit. The quality assessment index setting unit is used to evaluate specific indicators of the generated speech quality, including clarity, naturalness and accuracy of emotional expression. The assessment execution unit is used to evaluate the generated speech according to the set indicators.

[0037] Specifically, the system uses an automated quality assessment tool to evaluate the generated speech and optimizes it based on the assessment results to ensure that the naturalness and clarity of the speech meet the expected standards. After speech synthesis, the generated speech will be evaluated by a quality assessment module, which uses a deep learning-based assessment algorithm to score indicators such as the naturalness of the speech and the accuracy of emotional expression, and provides optimization suggestions. The system iterates and optimizes based on these suggestions to ensure the high quality of the output speech. The quality assessment indicator setting unit provides a clear and scientific basis for accurately evaluating the quality of generated speech, ensuring the comprehensiveness and objectivity of the evaluation, avoiding subjective arbitrariness and one-sidedness, and helping to focus on key quality dimensions and improve the quality of speech generation in a targeted manner; The evaluation execution unit can objectively and accurately evaluate the generated speech according to predetermined indicators, provide reliable data support for subsequent optimization, timely discover quality problems in the speech generation process, provide a clear direction for improvement, promote the continuous optimization and improvement of speech generation technology, and improve the performance and competitiveness of the system.

[0038] The quality assessment and optimization module also includes a historical data storage unit and an optimization strategy formulation unit. The optimization strategy formulation unit is used to formulate optimization strategies based on the assessment results to improve system performance. The historical data storage unit is used to store the results of previous quality assessments and corresponding optimization strategies for trend analysis and experience learning.

[0039] Specifically, the historical data storage unit provides a rich data foundation for the long-term optimization of the system, supports in-depth analysis of the trend of system performance changes, facilitates data backtracking and comparison, helps to discover potential rules and problems, promotes the accumulation and inheritance of experience, and provides valuable reference for new optimization work; The optimization strategy formulation unit can formulate effective optimization strategies in a targeted manner, quickly resolve existing quality issues, improve the efficiency and accuracy of system optimization, avoid blind attempts and ineffective improvement work, and promote the continuous improvement of system performance to adapt to ever-changing user needs and technological developments.

[0040] The user output interface module includes a voice display unit, a user operation response unit and a feedback collection unit. The voice display unit is used to display the generated voice on the user interface for the user to view intuitively. The user operation response unit is used to respond to the user's download and play operation requests. The feedback collection unit is used to collect the user's feedback information on the generated voice, and the feedback information is used for system improvement. Specifically, the user output interface module is responsible for presenting the final synthesized speech to the user and supports speech download and playback functions. The optimized speech is transmitted to the user output interface, and the user can play or download the synthesized speech through the interface, or make further speech adjustments and optimizations. The voice display unit enables users to intuitively see the relevant information of the generated voice, enhance the perception and understanding of the generated results, provide convenient operation control, facilitate users to play according to their own needs, improve the interactive experience between users and the system, and increase user satisfaction with the system; The user operation response unit can respond to the user's operation request in a timely and accurate manner, provide a smooth user experience, ensure the stability and reliability of download and playback operations, avoid operation failures or freezes, enhance the interactivity and usability of the system, and improve user satisfaction with system functions; The feedback collection unit provides direct and real opinions and suggestions from users for system improvement, which has important reference value, helps to discover problems and deficiencies in the system, clarify the direction and focus of improvement, enhance user participation, and increase user attention and trust in the system.

[0041] Implementation Method 2: Real-time Voice Cloning System Real-time data collection and preprocessing: The system collects voice data through real-time microphone input and performs preprocessing while collecting to ensure the real-time nature of voice generation. During real-time calls or online meetings, voice data is directly input into the system through the microphone. The system performs preprocessing operations while collecting, and the processed voice segments are immediately passed to the voice feature extraction module to reduce processing delays. Fast emotion modeling and synthesis: This solution uses an acceleration algorithm in the emotion modeling and speech synthesis process to ensure instant speech generation. After feature extraction, the system uses a lightweight emotion modeling algorithm to quickly model speech emotions. The generated emotion features are immediately passed to the speech synthesis module, and fast speech generation is achieved through an optimized neural network, ensuring that users get a smooth speech experience in real-time interaction. Real-time user feedback and automatic optimization: Working principle: The system obtains user feedback in real time, automatically adjusts speech generation parameters, and continuously optimizes the quality of speech output. During the real-time generation process, the system monitors user feedback in real time, collects feedback data and automatically adjusts speech synthesis parameters, such as speaking speed and intonation, to ensure that the speech output meets user expectations.

[0042] Implementation 3: Multimodal Fusion Emotional Speech Cloning System Multimodal data collection and fusion: The system not only collects voice data, but also combines image and text data for multimodal fusion to enhance the accuracy of emotional expression. When users upload voice data, they can also upload corresponding expression images and text descriptions. The system processes the image data and text data synchronously with the voice data, and generates richer emotional speech through multimodal fusion algorithms. Multimodal emotion modeling: The system uses multimodal data for emotion modeling to ensure that the generated speech is highly consistent with the user's emotions not only at the speech level but also at the expression and semantic levels. The feature extraction module extracts not only speech features but also expression features in images and emotional information in text. The multimodal fusion model integrates these features to generate a unified emotion expression model for speech synthesis. Multimodal speech synthesis: The multimodal speech synthesis module combines audio, image and text features to generate high-fidelity emotional speech, further enhancing the user experience. After the emotion modeling is completed, the multimodal synthesis module combines the audio features with the image and text features to generate more vivid speech output. Users can not only hear the speech, but also see the corresponding expressions and text descriptions through the interface.

[0043] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A real-time emotional simulation voice cloning system, characterized in that: include: Data collection and preprocessing module, speech feature extraction module, emotion modeling module, speech synthesis module, quality assessment and optimization module and user output interface module; The data acquisition and preprocessing module is used to obtain the user's original voice data and perform denoising, normalization, and frame processing operations. The data acquisition and preprocessing module is connected to the voice feature extraction module, and the preprocessed data is transmitted to the feature extraction module; The speech feature extraction module is used to extract key pitch, tone, and spectrum features from the preprocessed speech data to provide a basis for emotion modeling. The speech feature extraction module is connected to the emotion modeling module, and the extracted features are used as input for emotion modeling; The emotion modeling module is used to perform emotion modeling on the extracted speech features by using a generative adversarial network and a recurrent neural network to generate a speech model with specific emotion features. The emotion modeling module is connected to the speech synthesis module, and the modeled emotion speech features are transmitted to the speech synthesis module; The speech synthesis module is used to synthesize speech with emotion according to the features generated by the emotion modeling module. The speech synthesis module is connected to the user output interface module, and the generated speech will be transmitted to the user interface for user use; The quality assessment and optimization module is used to perform quality assessment on the generated speech and optimize the system according to the assessment results and user feedback. The quality assessment and optimization module is connected to the speech synthesis module and the user feedback system, and the assessment results are used to optimize the speech synthesis module. The user output interface module is used to display the generated speech to the user and provide download and play functions for interaction with the user. The user output interface module is connected to the speech synthesis module and the quality assessment module, and the final speech is displayed and played by this module.

2. The voice cloning system for real-time emotion simulation according to claim 1, characterized in that: The data collection and preprocessing module includes an original data collection unit and a noise removal unit. The original data collection unit is responsible for collecting the user's initial voice information, and the noise removal unit is used to eliminate environmental noise and other interfering sounds in the collected voice data.

3. The real-time emotional simulation voice cloning system according to claim 1, characterized in that: The data acquisition and preprocessing module also includes a data standardization unit and a data cache unit. The data standardization unit is used to standardize the preprocessed voice data so that it has a unified format and range to facilitate subsequent processing. The data cache unit is used to temporarily store the collected original voice data to prevent data loss or processing interruption.

4. The real-time emotional simulation voice cloning system according to claim 1, characterized in that: The speech feature extraction module includes a feature extraction algorithm unit, a feature screening unit and a feature storage unit. The feature extraction algorithm unit is used to use the relevant algorithm of the convolutional neural network to extract key features from the preprocessed speech data. The feature screening unit is used to screen the extracted features and remove redundant or unimportant features. The feature storage unit is used to store the screened key speech features for subsequent module calls.

5. The real-time emotional simulation voice cloning system according to claim 1, characterized in that: The emotion modeling module includes an emotion learning unit and a model building unit. The emotion learning unit is used to use a generative adversarial network and a recurrent neural network to learn a large number of speech samples with emotion labels to acquire knowledge for emotion modeling. The model building unit is used to build an emotion speech model based on the learned knowledge.

6. The real-time emotional simulation voice cloning system according to claim 1, characterized in that: The emotion modeling module also includes a model adjustment unit and a model verification unit. The model verification unit is used to verify the validity and accuracy of the constructed and adjusted emotion model, and the model adjustment unit is used to fine-tune the constructed model to improve the accuracy of emotion expression.

7. The real-time emotional simulation voice cloning system according to claim 1, characterized in that: The speech synthesis module includes a synthesis algorithm application unit, a sound quality optimization unit and a synthesized speech output unit. The synthesis algorithm application unit is used to synthesize speech using algorithms related to parameter speech synthesis technology and neural network speech synthesis technology. The sound quality optimization unit is used to optimize the sound quality of the generated speech to make it clear and natural. The synthesized speech output unit is used to output the synthesized speech to subsequent modules or users.

8. The real-time emotional simulation voice cloning system according to claim 1, characterized in that: The quality assessment and optimization module includes a quality assessment index setting unit and an assessment execution unit. The quality assessment index setting unit is used to evaluate specific indicators of the generated speech quality, including clarity, naturalness and accuracy of emotional expression. The assessment execution unit is used to evaluate the generated speech according to the set indicators.

9. The real-time emotional simulation voice cloning system according to claim 1, characterized in that: The quality assessment and optimization module also includes a historical data storage unit and an optimization strategy formulation unit. The optimization strategy formulation unit is used to formulate an optimization strategy based on the assessment results to improve system performance. The historical data storage unit is used to store the results of previous quality assessments and the corresponding optimization strategies for trend analysis and experience learning.

10. The real-time emotional simulation voice cloning system according to claim 1, characterized in that: The user output interface module includes a voice display unit, a user operation response unit and a feedback collection unit. The voice display unit is used to display the generated voice on the user interface for the user to view intuitively. The user operation response unit is used to respond to the user's download and play operation requests. The feedback collection unit is used to collect the user's feedback information on the generated voice, and the feedback information is used for system improvement.

Images