Emotional speech modeling method based on deep learning

Abstract

The invention relates to the technical field of emotional voice modeling, and discloses an emotional voice modeling method based on deep learning, which comprises the following steps: S1, system starting and initialization: when a system is started, firstly initializing a data preprocessing module, and inputting original voice data into an emotional feature extraction module; s2, emotion modeling and distinguishing, wherein an emotion modeling module receives feature data; s3, speech synthesis and optimization: the optimized emotional speech sample is transmitted to a speech synthesis module; and S4, outputting and feeding back: presenting the generated emotional voice to a user through a user interface module and a user interface or an output system. Through combination of the GAN and the RNN, the system can carefully capture subtle emotional changes in the speech, the generated speech is more natural and vivid in emotional expression, the emotional state of a speaker can be accurately reflected, and the advantage is especially suitable for application scenes requiring high emotional expression.

Description

Technical Field

[0001] The present invention relates to the technical field of emotional speech modeling, and specifically to an emotional speech modeling method based on deep learning. Background Art

[0002] Emotion modeling technology is an important part of achieving natural speech synthesis. Current emotion modeling methods are mainly based on deep learning and traditional speech processing technology. Although these methods have made certain progress in speech synthesis, there are still many problems and shortcomings in the accuracy, naturalness of emotional expression and the processing of complex emotions.

[0003] Emotional modeling based on hidden Markov model (HMM). Hidden Markov model (HMM) is one of the commonly used modeling methods in early speech synthesis technology. This method relies on statistical models and realizes the generation of emotional speech by modeling the statistical laws of training data. Emotional speech generation based on recurrent neural network (RNN). With the development of deep learning technology, recurrent neural network (RNN) is gradually applied to emotional speech generation. RNN can capture the time series characteristics of speech and has a good processing effect on speech coherence and emotional expression. Application of generative adversarial network (GAN) in emotion modeling: Generative adversarial network (GAN) is widely used in the field of speech synthesis.

[0004] In the existing technology, HMM is not good enough in processing the delicacy and diversity of emotional expression, and the generated speech often appears mechanical and lacks vividness. RNN has limitations in processing long sequence dependencies, and the generated emotional speech is prone to incoherence and lack of stability in emotional expression during long conversations. GAN is still insufficient in the delicacy of emotional expression and the ability to process multiple emotions, especially in complex emotional transitions and multi-modal emotional expression. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides an emotional speech modeling method based on deep learning to solve the problems of insufficient subtlety of emotional expression, limited ability to handle complex emotional transitions, low naturalness of speech, high complexity of model training and limitations of multimodal emotional expression.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: A deep learning-based emotional speech modeling method comprises the following steps: S1, system startup and initialization: when the system starts, the data preprocessing module is initialized first, the original speech data is input into the emotion feature extraction module, and then after the feature extraction module completes the extraction operation, the extracted feature data is passed to the emotion modeling module. At the same time, the data enhancement module enhances the original data, and then the model training module prepares the data and parameters required for training, and stores the relevant data through the data storage module; S2, emotion modeling and discrimination: the emotion modeling module receives feature data, generates emotion speech samples using adversarial networks, and uses recurrent neural networks to perform time series modeling on the generated samples. The generated speech samples are then fed into the emotion discrimination module, which performs evaluation and feeds the evaluation results back to the modeling module. The modeling module is used for speech optimization, and the data storage module saves intermediate data. S3, speech synthesis and optimization: The optimized emotional speech samples are passed to the speech synthesis module. The synthesis algorithm application unit uses the relevant algorithm to generate the final speech output. The generated speech is then finally evaluated and optimized by the quality assessment and optimization module. The data storage module records the relevant data. The model training module adjusts the training strategy according to the evaluation results. S4. Output and feedback: The generated emotional speech is presented to the user through the user interface module, the user interface or the output system, and user feedback is collected at the same time. The user feedback is used to further optimize the system performance. The data storage module stores the user feedback data. The model training module performs targeted training based on the feedback. The data enhancement module adjusts the enhancement strategy based on the feedback.

[0007] A deep learning-based emotional speech modeling method, comprising: The emotion feature extraction module is used to extract key features that reflect emotions from the original speech, the key features include pitch and tone, the emotion feature extraction module receives the speech data from the data preprocessing module, and passes the extracted feature data to the emotion modeling module; The emotion modeling module is used to receive the feature data from the emotion feature extraction module, generate an emotion speech sample through an adversarial network, and use a recurrent neural network model to model the time series of the speech, and pass the generated emotion speech model data to the speech synthesis module; The emotion discrimination module is used to connect with the emotion modeling module, receive the generated emotion speech samples, and transmit the feedback data back to the emotion modeling module for optimization after evaluation; The speech synthesis module is used to receive the processed data from the emotion modeling module and generate the final speech output, and the generated speech data is passed to the quality assessment module for final assessment; The quality evaluation and optimization module is used to connect with the speech synthesis module and the user feedback system, receive the output data after speech synthesis for evaluation, and pass it to each module for optimization according to the feedback results; The data enhancement module is used to increase the diversity of the original speech data, improve the generalization ability of the model, receive the preprocessed data, and pass the enhanced data to the emotion feature extraction module; The model training module is used to train and update the parameters of the relevant model, train the model performance through the parameters, connect with the emotion modeling module and the emotion discrimination module, and receive training data and optimization instructions; The data storage module is used to store various system-related data and models, and is used for calling at any time, connected to various modules, and receiving and providing data that needs to be stored and read; The user interface module is used to realize the communication between the user and the system. The user can input data and instructions, and can also see the generated results of the system. It is connected with each module, transmits user instructions and data, and receives the results generated by the system.

[0008] Preferably, the emotion feature extraction module includes a feature acquisition unit, a feature analysis unit and a feature extraction execution unit. The feature acquisition unit is responsible for collecting key information related to emotion from the original speech data, the feature analysis unit is used to analyze the collected information to analyze which are the key emotion features, and the feature extraction execution unit is used to actually perform the feature extraction operation and output the result.

[0009] Preferably, the emotion modeling module includes a model generation unit, a time series processing unit and a model parameter adjustment unit. The model generation unit is used to generate a preliminary emotion speech model using an adversarial network. The time series processing unit is used to model and optimize the time series of speech using a recurrent neural network. The model parameter adjustment unit is used to adjust the parameters of the model according to the input characteristics and generation effects.

[0010] Preferably, the emotion discrimination module includes an evaluation standard setting unit, an evaluation execution unit and a feedback generation unit. The evaluation standard setting unit is used to evaluate the specific standards and indicators of the generated speech, the evaluation execution unit is used to perform actual evaluation of the generated emotional speech, and the feedback generation unit is used to generate feedback information based on the evaluation results and pass it to the emotion modeling module.

[0011] Preferably, the speech synthesis module includes a synthesis algorithm application unit, a speech optimization unit and a synthesis result output unit. The synthesis algorithm application unit is used to perform synthesis operations using parameterized and neural network speech synthesis algorithms. The speech optimization unit is used to optimize the sound quality and emotional expression of the synthesized speech. The synthesis result output unit is used to output the final synthesized emotional speech to the quality assessment module.

[0012] Preferably, the quality assessment and optimization module includes a quality assessment index calculation unit, an optimization strategy formulation unit and an optimization execution supervision unit. The quality assessment index calculation unit is used to calculate specific index values ​​for evaluating the generated voice quality, the optimization strategy formulation unit is used to formulate a system optimization strategy based on the assessment results, and the optimization execution supervision unit is used to supervise the execution of the optimization strategy.

[0013] Preferably, the data enhancement module includes a data expansion strategy formulation unit, an enhancement algorithm execution unit and an enhancement data screening unit. The data expansion strategy formulation unit is used to formulate a strategy for expanding and enhancing the original data, the enhancement algorithm execution unit is used to execute a specific data enhancement algorithm to generate new data, and the enhancement data screening unit is used to screen and filter the enhanced data to ensure its quality and effectiveness.

[0014] Preferably, the model training module includes a training data preparation unit, a training process control unit and a training result evaluation unit. The training data preparation unit is used to collect, organize and preprocess data for model training. The training process control unit is used to control the progress of model training, which includes setting the number of training rounds and adjusting the learning rate. The training result evaluation unit is used to evaluate the progress of model training and determine whether further training or parameter adjustment is needed based on the progress.

[0015] Preferably, the data storage module includes a data classification storage unit, a data backup and recovery unit and a storage resource management unit. The data classification storage unit is used to classify and store different types of data, including original data, feature data and model data. The data backup and recovery unit is used to regularly back up important data and restore data when necessary. The storage resource management unit is used to manage storage resources, detect whether there is enough space to store data, and optimize storage efficiency and storage space. The user interface module includes a user input receiving unit, a result display unit and a user interaction response unit. The user input receiving unit is used to receive voice data and related information of instructions input by the user. The result display unit is used to display the results generated by the system to the user in an intuitive manner. The results include voice playback and graphic display. The user interaction response unit is used to promptly respond to the user's operations and feedback during the interaction process.

[0016] The present invention provides an emotional speech modeling method based on deep learning, which has the following beneficial effects: 1. The present invention achieves accurate modeling of speech emotions by combining the generative adversarial network (GAN) with the recurrent neural network (RNN). The system can carefully capture the subtle changes in emotions in the speech. The generated speech is more natural and vivid in emotional expression and can accurately reflect the speaker's emotional state. This advantage is particularly suitable for application scenarios that require high emotional expression, such as virtual assistants, online education, and emotional computing.

[0017] 2. The present invention has the ability to handle complex emotional transitions and can achieve natural transitions between multiple emotions during the speech generation process. For example, the emotional change from calm to anger can generate natural and coherent speech performance through the emotion modeling module of the present invention, avoiding the common problems of emotional mutations or incoherent expressions in the prior art.

[0018] 3. Through time series modeling and GAN optimization, the present invention ensures the naturalness and coherence of speech in long-term speech generation. Even in scenarios with long conversations or alternating emotions, the speech generated by the system can still maintain a high degree of naturalness and fluency, thereby enhancing the user's auditory experience and sense of interaction.

[0019] 4. The present invention adopts an optimized deep learning algorithm and an efficient data processing method, which significantly reduces the time and computing resources required for model training. Compared with traditional emotional speech modeling technology, the system of the present invention is more efficient in the training and reasoning process and is suitable for large-scale practical application scenarios.

[0020] 5. The present invention supports the fusion of multimodal data. By combining multiple modal information such as voice, image, and text, the richness and accuracy of emotional expression are further enhanced. The system is not only suitable for voice emotional expression, but can also be extended to a wide range of application scenarios of multimodal emotional computing and synthesis.

[0021] 6. The emotion modeling module of the present invention is suitable for a variety of industry applications, such as voice assistants, virtual characters, online education, emotion computing, customer service, etc. Its high-quality emotional speech generation capability and good scalability enable this technology to play an important role in various fields and enhance user experience and interactive effects.

[0022] 7. The present invention significantly improves the user's auditory experience and interactive feeling by improving the emotional expression ability and naturalness of speech generation, especially in scenarios that require high personalization and emotionality, such as voice interaction and virtual assistants. The present invention can provide more humane voice services and enhance user satisfaction and loyalty. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A method flow chart of an emotional speech modeling method based on deep learning of the present invention; Figure 2 A connection relationship diagram of an emotional speech modeling method based on deep learning of the present invention; Figure 3 This is a system architecture diagram of an emotional speech modeling system based on deep learning of the present invention; Figure 4 This is an architecture diagram of an emotion feature extraction module of an emotion speech modeling system based on deep learning of the present invention; Figure 5 This is an architecture diagram of an emotion modeling module of an emotion speech modeling system based on deep learning of the present invention; Figure 6 This is an architecture diagram of an emotion discrimination module of an emotion speech modeling system based on deep learning of the present invention; Figure 7 This is a speech synthesis module architecture diagram of an emotional speech modeling system based on deep learning of the present invention; Figure 8 This is a quality assessment and optimization module architecture diagram of an emotional speech modeling system based on deep learning in the present invention; Fig. 9 This is a data enhancement module architecture diagram of an emotional speech modeling system based on deep learning of the present invention; Fig.10 This is a model training module architecture diagram of an emotional speech modeling system based on deep learning of the present invention; Fig.11 This is a data storage module architecture diagram of an emotional speech modeling system based on deep learning according to the present invention; Fig.12 This is an architecture diagram of a user interface module of an emotional speech modeling system based on deep learning of the present invention. DETAILED DESCRIPTION

[0024] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Example: Please see attached Figure 1 and attached Figure 2 , an emotional speech modeling method based on deep learning, comprising the following steps: S1, system startup and initialization: when the system starts, the data preprocessing module is initialized first, the original speech data is input into the emotion feature extraction module, and then after the feature extraction module completes the extraction operation, the extracted feature data is passed to the emotion modeling module. At the same time, the data enhancement module enhances the original data, and then the model training module prepares the data and parameters required for training, and stores the relevant data through the data storage module; S2, emotion modeling and discrimination: the emotion modeling module receives feature data, generates emotion speech samples using adversarial networks, and uses recurrent neural networks to perform time series modeling on the generated samples. The generated speech samples are then fed into the emotion discrimination module, which performs evaluation and feeds the evaluation results back to the modeling module. The modeling module is used for speech optimization, and the data storage module saves intermediate data. S3, speech synthesis and optimization: The optimized emotional speech samples are passed to the speech synthesis module. The synthesis algorithm application unit uses the relevant algorithm to generate the final speech output. The generated speech is then finally evaluated and optimized by the quality assessment and optimization module. The data storage module records the relevant data. The model training module adjusts the training strategy according to the evaluation results. S4. Output and feedback: The generated emotional speech is presented to the user through the user interface module, the user interface or the output system, and user feedback is collected at the same time. The user feedback is used to further optimize the system performance. The data storage module stores the user feedback data. The model training module performs targeted training based on the feedback. The data enhancement module adjusts the enhancement strategy based on the feedback.

[0026] Specifically, S1. System startup and initialization: When the system starts, the data preprocessing module cleans and preliminarily processes the original voice data. The emotional feature extraction module uses deep learning algorithms to accurately capture the emotional features in the voice. The data enhancement module increases data diversity through data expansion technology. The model training module prepares the data required for training. The data storage module records various related data, providing a high-quality, distinctive and rich data foundation for subsequent modeling work, ensuring the smooth startup of the entire system and the accuracy and integrity of the initial data, and creating favorable conditions for the efficient development of subsequent processes. S2. Emotion modeling and discrimination: The emotion modeling module uses the adversarial network to generate preliminary emotional speech samples, and the recurrent neural network finely models the time series of the samples. The emotion discrimination module strictly evaluates the generated samples according to the preset standards and feeds the results back to the modeling module to optimize the model parameters and structure. Through repeated generation, evaluation and optimization, the generated emotional speech samples are continuously improved in terms of accuracy, coherence and naturalness of emotional expression, gradually approaching real and contagious emotional speech; S3, speech synthesis and optimization: The optimized emotional speech samples are passed to the speech synthesis module. The synthesis algorithm application unit integrates multiple advanced algorithms to generate speech. The quality assessment and optimization module evaluates based on multi-dimensional indicators and adjusts the synthesis strategy accordingly. The data storage module and the model training module record data and adjust the training strategy respectively. The generated speech reaches a high quality in terms of sound quality, fluency, and emotional expression. After evaluation and optimization, defects are further reduced, making the speech more natural and clear, and the emotional transmission more accurate, which significantly improves the overall quality of the speech. S4. Output and feedback: The generated emotional speech is presented to the user through the user interface module. At the same time, feedback information such as the user's feelings and opinions on the speech is collected. The data storage module saves the feedback data. The model training module and the data enhancement module adjust the strategy and parameters based on the feedback, so that the user can directly experience the generated emotional speech. The feedback prompts the system to make targeted improvements, enhance the fit between the system and user needs, improve user satisfaction, and continuously optimize system performance.

[0027] Please see attached Figure 3 -Attached Fig.12 , an embodiment of the present invention provides an emotional speech modeling method based on deep learning, comprising: The emotion feature extraction module is used to extract key features that reflect emotions from the original speech. The key features include pitch and tone. The emotion feature extraction module receives the speech data from the data preprocessing module and passes the extracted feature data to the emotion modeling module. The emotion modeling module is used to receive the feature data from the emotion feature extraction module, generate emotional speech samples through the adversarial network, and use the recurrent neural network model to model the time series of the speech. The generated emotional speech model data is passed to the speech synthesis module; The emotion discrimination module is used to connect with the emotion modeling module, receive the generated emotion speech samples, and pass the feedback data back to the emotion modeling module for optimization after evaluation; The speech synthesis module is used to receive the processed data from the emotion modeling module and generate the final speech output. The generated speech data is passed to the quality assessment module for final evaluation; The quality evaluation and optimization module is used to connect with the speech synthesis module and the user feedback system, receive the output data after speech synthesis for evaluation, and pass it to each module for optimization based on the feedback results; The data enhancement module is used to increase the diversity of the original speech data and improve the generalization ability of the model. It receives the preprocessed data and passes the enhanced data to the emotion feature extraction module. The model training module is used to train and update the parameters of the relevant models, train the model performance through the parameters, connect with the emotion modeling module and the emotion discrimination module, and receive training data and optimization instructions; The data storage module is used to store various system-related data and models, which can be called at any time. It is connected to various modules to receive and provide data that needs to be stored and read. The user interface module is used to realize the communication between users and the system. Users can input data and instructions, and can also see the results generated by the system. It is connected to each module, transmits user instructions and data, and receives the results generated by the system.

[0028] The emotion feature extraction module includes a feature acquisition unit, a feature analysis unit and a feature extraction execution unit. The feature acquisition unit is responsible for collecting key information related to emotion from the original speech data. The feature analysis unit is used to analyze the collected information to analyze which are the key emotion features. The feature extraction execution unit is used to actually perform the feature extraction operation and output the results.

[0029] Specifically, the feature collection unit receives the original voice data, uses voice recognition to convert it into text, and collects acoustic and rhythmic features such as pitch and speaking speed, captures pauses and stresses, etc., to provide various materials for subsequent analysis, and comprehensively screens out various key information related to emotions, covering semantics, acoustics and other levels, laying a rich material foundation for emotional feature analysis; The feature analysis unit uses the sentiment analysis model and algorithm to analyze semantics against the sentiment dictionary, combines acoustic and other feature association rules, comprehensively considers multiple feature relationships, accurately identifies key sentiment features, eliminates interference information, improves the accuracy and effectiveness of sentiment feature extraction, and makes subsequent extraction more focused; The feature extraction execution unit quantitatively extracts and organizes the key emotional features determined by analysis, such as text emotional vocabulary, speech acoustic features, etc., according to established methods and formats, accurately and efficiently performs the extraction task, outputs the emotional features in a standardized and usable form, and provides high-quality input data for subsequent emotion-related applications.

[0030] The emotion modeling module includes a model generation unit, a time series processing unit and a model parameter adjustment unit. The model generation unit is used to generate a preliminary emotion speech model using an adversarial network. The time series processing unit is used to model and optimize the time series of speech using a recurrent neural network. The model parameter adjustment unit is used to adjust the parameters of the model according to the input characteristics and generation effects.

[0031] Specifically, the model generation unit uses adversarial network technology to continuously optimize the generation capability through the adversarial game process between the generator and the discriminator, thereby generating a preliminary emotional speech model and simulating a speech pattern with emotional characteristics. It can quickly produce a basic emotional speech model framework, providing an initial blueprint for further improvement and optimization, and starting the emotional speech modeling process. The time series processing unit uses recurrent neural networks (RNN) and its variants, such as LSTM and GRU, to process speech data in chronological order, capture the dynamic change characteristics of speech in time series, and then achieve modeling and optimization, effectively process speech time series information, and improve the model's ability to understand and express the changes of speech emotions over time, so that the emotional speech model is more in line with the dynamic characteristics of actual speech emotions; The model parameter adjustment unit uses optimization algorithms, such as gradient descent, to fine-tune the various parameters of the model based on the input emotional feature data and the current generation effect of the model to improve model performance. By continuously adjusting parameters, the emotional speech model can better adapt to the input features, improve the model's simulation accuracy and generation quality of emotional speech, and make the model output more in line with expectations.

[0032] The emotion discrimination module includes an evaluation standard setting unit, an evaluation execution unit and a feedback generation unit. The evaluation standard setting unit is used to evaluate the specific standards and indicators of the generated speech. The evaluation execution unit is used to perform actual evaluation on the generated emotional speech. The feedback generation unit is used to generate feedback information based on the evaluation results and pass it to the emotion modeling module.

[0033] Specifically, the evaluation standard setting unit combines the relevant theories and application needs of emotional speech, comprehensively considers multiple factors such as the accuracy of emotional expression, naturalness, and fit with the set context of the speech, to clarify the specific standards and indicator system for evaluating generated speech, and provides a scientific, comprehensive and targeted evaluation scale for the subsequent evaluation execution unit, ensuring that the evaluation of generated speech is reasonable and well-founded, and ensuring the objectivity and accuracy of the evaluation; The evaluation execution unit uses professional speech analysis technology and emotion discrimination algorithms to conduct detailed detection and quantitative analysis of the generated emotional speech from multiple aspects such as emotion intensity, intonation changes, and semantic emotion matching based on the standards and indicators established by the evaluation standard setting unit. It can accurately and objectively evaluate the actual quality of the generated emotional speech, clearly present its performance in various dimensions of emotional expression, and provide a detailed and reliable evaluation data basis for the feedback generation unit; Based on the specific evaluation results obtained by the evaluation execution unit, the feedback generation unit compares and analyzes the gaps with the preset goals, and specifically extracts improvement suggestions and problems in aspects such as speech emotion expression and model parameter adjustment, and then generates system feedback information. It can effectively convert the evaluation situation into feedback content that can guide the improvement of the emotion modeling module, prompting the emotion modeling module to optimize the model according to the feedback, thereby continuously improving the quality and effect of generating emotional speech.

[0034] The speech synthesis module includes a synthesis algorithm application unit, a speech optimization unit and a synthesis result output unit. The synthesis algorithm application unit is used to perform synthesis operations using parameterized and neural network speech synthesis algorithms. The speech optimization unit is used to optimize the sound quality and emotional expression of the synthesized speech. The synthesis result output unit is used to output the final synthesized emotional speech to the quality assessment module.

[0035] Specifically, the synthesis algorithm application unit works with the help of parameterized and neural network speech synthesis algorithms. The parameterized algorithm synthesizes by setting various parameters of the speech, such as pitch and duration; the neural network algorithm generates speech by learning speech patterns using deep neural networks, and continuously optimizes the synthesis effect through a large amount of training data. It can effectively perform speech synthesis operations according to the set algorithm and generate an initial speech version with certain speech characteristics and emotional tendencies, laying the foundation for subsequent optimization and output; The speech optimization unit optimizes the speech generated by the synthesis algorithm application unit from the aspects of sound quality and emotional expression. In terms of sound quality, it may improve the clarity and fullness of the sound by adjusting audio parameters and applying audio processing technology; in terms of emotional expression, it further strengthens the emotional color in the speech based on emotional labels or related parameters to make it more in line with the expected emotional state, significantly improve the overall quality of the synthesized speech, make it clearer and more pleasant in sound quality, and more accurate and natural in emotional expression, making the synthesized speech more appealing and expressive; The synthesis result output unit is responsible for accurately transmitting and outputting the final synthesized emotional speech after processing by the previous unit to the quality assessment module in accordance with the prescribed format and requirements, ensuring the integrity and accuracy of data transmission, and realizing the effective connection between the synthesized emotional speech and the quality assessment module, providing accurate samples for the subsequent evaluation of the quality of the synthesized speech, and ensuring the smooth progress of the entire speech synthesis and evaluation process.

[0036] The quality assessment and optimization module includes a quality assessment index calculation unit, an optimization strategy formulation unit and an optimization execution supervision unit. The quality assessment index calculation unit is used to calculate the specific index value of the generated voice quality, the optimization strategy formulation unit is used to formulate the system optimization strategy according to the assessment results, and the optimization execution supervision unit is used to supervise the execution of the optimization strategy.

[0037] Specifically, the quality assessment index calculation unit uses professional speech analysis technology and related algorithms to evaluate the speech quality from multiple dimensions. For example, in terms of sound quality, it analyzes the clarity, fluency, and presence of noise of the audio. In terms of emotional expression, it detects whether the accuracy and intensity of the emotion meet expectations. By performing quantitative analysis on these specific features, the corresponding index values ​​are calculated. For example, the signal-to-noise ratio of the audio can be used as an indicator of the clarity of the sound quality. The specific index values ​​reflecting various aspects of the generated speech quality can be accurately obtained, providing data support for a comprehensive and objective understanding of the speech quality status, and providing a clear basis for the formulation of subsequent optimization strategies. The optimization strategy formulation unit analyzes the problems existing in voice quality based on the evaluation results obtained by the quality evaluation index calculation unit. If the problem is related to sound quality, it may formulate strategies such as adjusting audio parameters and replacing audio processing algorithms. For situations where emotional expression is poor, it will consider measures such as modifying emotional model parameters and optimizing emotional feature extraction. By comprehensively considering various factors, a systematic and targeted optimization strategy is formulated. The optimization execution supervision unit ensures that the optimization strategy can be effectively executed, ensures that the optimization work proceeds as planned, and promptly discovers and solves problems that arise during the execution process, thereby continuously improving the quality of the generated speech.

[0038] The data enhancement module includes a data expansion strategy formulation unit, an enhancement algorithm execution unit and an enhancement data screening unit. The data expansion strategy formulation unit is used to formulate strategies for expanding and enhancing the original data. The enhancement algorithm execution unit is used to execute specific data enhancement algorithms to generate new data. The enhancement data screening unit is used to screen and filter the enhanced data to ensure its quality and effectiveness.

[0039] Specifically, the data augmentation strategy formulation unit analyzes the characteristics of the original data and application requirements, and formulates augmentation and enhancement strategies such as cropping, splicing, and noise addition based on experience and knowledge, providing a clear direction for data augmentation and effectively improving data richness and availability to meet relevant requirements; The enhanced algorithm execution unit selects the corresponding algorithm according to the formulated strategy, such as generating new data by adding noise and splicing parameters, successfully generating diverse new data, changing the original singleness, and providing rich materials for model training and improving performance; The enhanced data screening unit sets standards such as integrity and identifiability, and uses these standards to check and eliminate new data that does not meet the requirements, ensuring the effectiveness of enhanced data quality, eliminating poor-quality data, and ensuring the smooth progress of subsequent work.

[0040] The model training module includes a training data preparation unit, a training process control unit and a training result evaluation unit. The training data preparation unit is used to collect, organize and preprocess data for model training. The training process control unit is used to control the progress of model training, which includes setting the number of training rounds and adjusting the learning rate. The training result evaluation unit is used to evaluate the progress of model training and determine whether further training or parameter adjustment is needed based on the progress.

[0041] Specifically, the training data preparation unit collects data related to model training from various data sources, which may include databases, file systems, network collection, etc. The collected data will be classified and sorted according to the data type and model requirements. For example, voice data will be divided according to different emotion categories, voice duration and other characteristics. Through comprehensive collection, careful sorting and necessary preprocessing, a high-quality, standardized and targeted data set is provided for model training, ensuring a good start for model training and improving the efficiency and effect of model training. Before model training begins, the training process control unit sets the appropriate number of training rounds based on factors such as the complexity of the model, the size of the data, and the expected training effect. The number of training rounds determines the number of times the model learns the data. Too few rounds may lead to insufficient learning, while too many rounds may lead to overfitting. The training process can be accurately controlled. By reasonably setting the number of training rounds and dynamically adjusting the learning rate, the model can fully learn the laws in the data during the training process and avoid problems such as overfitting, thus ensuring the stability and effectiveness of model training and improving the final performance of the model. The training result evaluation unit uses a variety of evaluation indicators to measure the progress of model training. These indicators may include accuracy, recall, F1 value, mean square error (MSE), etc., depending on the type of model and application scenario. For example, for classification models, accuracy is often used to measure the proportion of correct classifications by the model; for regression models, mean square error is often used to measure the error between the model prediction value and the true value. Through scientific evaluation indicators, the progress of model training can be accurately evaluated, and problems in the model training process can be discovered in time, providing a reliable basis for whether further training or parameter adjustment is needed, thereby helping to continuously optimize the performance of the model so that it can better adapt to actual application needs.

[0042] The data storage module includes a data classification storage unit, a data backup and recovery unit and a storage resource management unit. The data classification storage unit is used to classify and store different types of data, including original data, feature data and model data. The data backup and recovery unit is used to regularly back up important data and restore data when necessary. The storage resource management unit is used to manage storage resources, to detect whether there is enough space to store data, and to optimize storage efficiency and storage space.

[0043] Specifically, the data classification storage unit identifies and classifies different types of data such as original, feature, and model data, determines the storage path according to its source, feature, version, etc., and stores them in corresponding areas respectively, so that the data storage is clear and organized, which is convenient for fast and accurate query, call and management, reduces the time cost of search, and is conducive to long-term storage and maintenance of data; The data backup and recovery unit regularly backs up important data according to preset strategies, records relevant information, determines the backup version and recovers data according to the situation when problems occur, reduces the risk of data loss, ensures data security and integrity, and can restore data in a timely and accurate manner when recovery is needed, so that business processes can return to normal as soon as possible; The storage resource management unit monitors the use of storage resources in real time, determines whether there is sufficient space, optimizes storage efficiency through technology, allocates space reasonably, and promptly resolves the problem of insufficient space to ensure that data can be stored, improve overall storage performance, reduce costs, and make data storage more scientific and efficient.

[0044] The user interface module includes a user input receiving unit, a result display unit and a user interaction response unit. The user input receiving unit is used to receive voice data and related information of instructions input by the user. The result display unit is used to display the results generated by the system to the user in an intuitive manner. The results include voice playback and graphic display. The user interaction response unit is used to promptly respond to the user's operations and feedback during the interaction process.

[0045] Specifically, the user input receiving unit is mainly responsible for receiving various types of information input by the user, whether it is voice data or related instructions, etc. It can accurately capture the content expressed by the user, provide original materials for subsequent system processing, ensure that the information input by the user is accurately received, and enable the system to carry out subsequent work based on this, ensuring that the first step of the user's interaction with the system is completed smoothly without missing any key input; The result display unit presents the results generated by the system, such as audio content in the form of voice playback and various charts in the form of graphic display, to users in an intuitive and easy-to-understand way, so that users can clearly and conveniently understand the results of system processing, enhance user experience, and quickly know the relevant information output by the system, whether listening to voice or looking at graphics; The user interaction response unit responds to user operations, such as clicks, slides, etc., and feedback information in a timely manner during the interaction between the user and the system, maintains the smoothness of the interaction, and enables the user to feel the timeliness and interactivity of the system, as if communicating with an object that can respond in real time, thereby enhancing the user's favorability and trust in the system and ensuring a smooth interaction process.

[0046] Example 1: Emotion modeling based on generative adversarial network (GAN) and recurrent neural network (RNN): This implementation combines the generative adversarial network (GAN) and recurrent neural network (RNN) technologies to achieve accurate modeling of speech emotions. GAN is used to generate speech samples with emotional characteristics, and RNN is used to capture the time series characteristics of speech to ensure the coherence and naturalness of speech. Emotional feature extraction: The system extracts emotion-related features such as pitch, tone, speech rate, and spectrum from the input speech data. These feature data are processed at multiple levels through a convolutional neural network (CNN) to generate feature vectors. Generative adversarial training: The feature vector is input to the generator of the GAN model to generate preliminary emotional speech samples. The discriminator evaluates these samples and continuously optimizes the generator through adversarial training to make the generated speech samples more realistic. Time series modeling: The generated speech samples are modeled through the RNN model to capture the dynamic change characteristics of the speech and ensure the coherence and naturalness of the speech in emotional expression; Emotional speech generation: The final generated emotional speech sample is used to generate high-quality speech output through the speech synthesis module. The speech output has rich emotional expression and high naturalness. This implementation significantly improves the delicacy of emotional expression and the naturalness of speech generation, and is suitable for application scenarios that require a high degree of personalization and emotional expression, such as voice assistants and virtual characters.

[0047] Example 2: Emotion modeling based on multimodal fusion: This implementation introduces multimodal fusion technology into the emotion modeling process, and achieves richer emotion expression by combining multimodal data such as speech, image and text. This method not only considers the emotional features in speech, but also integrates the emotional information in images (such as expressions) and text; Multimodal data collection and processing: The system extracts emotional features from multimodal data such as voice, images (such as facial expressions), and text provided by users. Each modal data is processed through its own neural network model to generate a corresponding emotional feature vector; Multimodal emotion fusion: The emotion features of different modalities are comprehensively processed through the multimodal fusion module to generate a unified emotion representation vector, which can more comprehensively reflect the user's emotional state; Generative adversarial training and time series modeling: The synthesized emotion representation vector is input into the GAN and RNN models for generative adversarial training and time series modeling. The generated speech samples can accurately reflect multimodal emotion characteristics; Emotional speech generation: Through the speech synthesis module, high-quality speech output with multimodal emotional characteristics is generated. This kind of speech can simultaneously reflect the emotions in speech, expression and text, and is suitable for more complex emotional expression scenarios; This implementation significantly improves the accuracy and richness of emotional expression by introducing multimodal fusion, and is particularly suitable for scenarios that require high interactivity and emotional expression, such as virtual characters in virtual reality (VR) and augmented reality (AR) environments.

[0048] Example 3: Real-time emotion modeling based on lightweight model: This implementation optimizes the computational process of emotion modeling and achieves real-time emotional speech generation by using a lightweight deep learning model. This method is mainly aimed at application scenarios that require high real-time requirements, such as online meetings and instant messaging.

[0049] Real-time data collection and preprocessing: The user's real-time voice is input through the microphone, and the system immediately performs data preprocessing, including noise reduction, normalization and other operations to ensure the quality of the input data; Fast emotional feature extraction and modeling: The preprocessed data is processed by a lightweight emotional feature extraction module to generate an emotional feature vector, which is then input into a lightweight GAN model to quickly generate emotional speech samples. Real-time speech synthesis and output: The generated speech samples are directly passed to the speech synthesis module to quickly generate the final emotional speech output and deliver it to the user in real time; This implementation method has excellent real-time performance and computational efficiency, and can achieve instant emotional speech generation during user interaction. It is particularly suitable for applications with high real-time requirements, such as online interaction and voice chat. Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A deep learning-based emotional speech modeling method, characterized in that: The following steps are involved: S1, system startup and initialization: when the system starts, the data preprocessing module is initialized first, the original speech data is input into the emotion feature extraction module, and then after the feature extraction module completes the extraction operation, the extracted feature data is passed to the emotion modeling module. At the same time, the data enhancement module enhances the original data, and then the model training module prepares the data and parameters required for training, and stores the relevant data through the data storage module; S2, emotion modeling and discrimination: the emotion modeling module receives feature data, generates emotion speech samples using adversarial networks, and uses recurrent neural networks to perform time series modeling on the generated samples. The generated speech samples are then fed into the emotion discrimination module, which performs evaluation and feeds the evaluation results back to the modeling module. The modeling module is used for speech optimization, and the data storage module saves intermediate data. S3, speech synthesis and optimization: The optimized emotional speech samples are passed to the speech synthesis module. The synthesis algorithm application unit uses the relevant algorithm to generate the final speech output. The generated speech is then finally evaluated and optimized by the quality assessment and optimization module. The data storage module records the relevant data. The model training module adjusts the training strategy according to the evaluation results. S4. Output and feedback: The generated emotional speech is presented to the user through the user interface module, the user interface or the output system, and user feedback is collected at the same time. The user feedback is used to further optimize the system performance. The data storage module stores the user feedback data. The model training module performs targeted training based on the feedback. The data enhancement module adjusts the enhancement strategy based on the feedback.

2. An emotional speech modeling system based on deep learning, applied to the emotional speech modeling method based on deep learning according to claim 1, characterized in that: include: The emotion feature extraction module is used to extract key features that reflect emotions from the original speech, and the key features include pitch and tone; The emotion modeling module is used to receive the feature data from the emotion feature extraction module, generate an emotion speech sample through an adversarial network, and use a recurrent neural network model to model the time series of the speech, and pass the generated emotion speech model data to the speech synthesis module; The emotion discrimination module is used to connect with the emotion modeling module, receive the generated emotion speech samples, and transmit the feedback data back to the emotion modeling module for optimization after evaluation; The speech synthesis module is used to receive the processed data from the emotion modeling module and generate the final speech output, and the generated speech data is passed to the quality assessment module for final assessment; The quality evaluation and optimization module is used to connect with the speech synthesis module and the user feedback system, receive the output data after speech synthesis for evaluation, and pass it to each module for optimization according to the feedback results; The data enhancement module is used to increase the diversity of the original speech data, improve the generalization ability of the model, receive the preprocessed data, and pass the enhanced data to the emotion feature extraction module; The model training module is used to train and update the parameters of the relevant model, train the model performance through the parameters, connect with the emotion modeling module and the emotion discrimination module, and receive training data and optimization instructions; The data storage module is used to store various system-related data and models, and is used for calling at any time, connected to various modules, and receiving and providing data that needs to be stored and read; The user interface module is used to realize the communication between the user and the system. The user can input data and instructions, and can also see the generated results of the system. It is connected with each module, transmits user instructions and data, and receives the results generated by the system.

3. The emotional speech modeling system based on deep learning according to claim 2, characterized in that: The emotion feature extraction module includes a feature collection unit, a feature analysis unit and a feature extraction execution unit. The feature collection unit is responsible for collecting key information related to emotion from the original speech data. The feature analysis unit is used to analyze the collected information to analyze which are the key emotion features. The feature extraction execution unit is used to actually perform the feature extraction operation and output the results.

4. The emotional speech modeling system based on deep learning according to claim 2, characterized in that: The emotion modeling module includes a model generation unit, a time series processing unit and a model parameter adjustment unit. The model generation unit is used to generate a preliminary emotion speech model using an adversarial network. The time series processing unit is used to model and optimize the time series of speech using a recurrent neural network. The model parameter adjustment unit is used to adjust the parameters of the model according to the input characteristics and generation effects.

5. The emotional speech modeling system based on deep learning according to claim 2, characterized in that: The emotion discrimination module includes an evaluation standard setting unit, an evaluation execution unit and a feedback generation unit. The evaluation standard setting unit is used to evaluate the specific standards and indicators of the generated speech. The evaluation execution unit is used to perform actual evaluation on the generated emotional speech. The feedback generation unit is used to generate feedback information based on the evaluation results and pass it to the emotion modeling module.

6. The deep learning-based emotional speech modeling system according to claim 2, characterized in that: The speech synthesis module includes a synthesis algorithm application unit, a speech optimization unit and a synthesis result output unit. The synthesis algorithm application unit is used to perform synthesis operations using parameterized and neural network speech synthesis algorithms. The speech optimization unit is used to optimize the sound quality and emotional expression of the synthesized speech. The synthesis result output unit is used to output the final synthesized emotional speech to the quality assessment module.

7. The emotional speech modeling system based on deep learning according to claim 2, characterized in that: The quality assessment and optimization module includes a quality assessment index calculation unit, an optimization strategy formulation unit and an optimization execution supervision unit. The quality assessment index calculation unit is used to calculate the specific index value for evaluating the generated voice quality. The optimization strategy formulation unit is used to formulate a system optimization strategy based on the assessment results. The optimization execution supervision unit is used to supervise the execution of the optimization strategy.

8. The deep learning-based emotional speech modeling system according to claim 2, characterized in that: The data enhancement module includes a data expansion strategy formulation unit, an enhancement algorithm execution unit and an enhancement data screening unit. The data expansion strategy formulation unit is used to formulate a strategy for expanding and enhancing the original data, the enhancement algorithm execution unit is used to execute a specific data enhancement algorithm to generate new data, and the enhancement data screening unit is used to screen and filter the enhanced data to ensure its quality and effectiveness.

9. The deep learning-based emotional speech modeling system according to claim 2, characterized in that: The model training module includes a training data preparation unit, a training process control unit and a training result evaluation unit. The training data preparation unit is used to collect, organize and preprocess data for model training. The training process control unit is used to control the progress of model training, which includes setting the number of training rounds and adjusting the learning rate. The training result evaluation unit is used to evaluate the progress of model training and determine whether further training or parameter adjustment is needed based on the progress.

10. The deep learning-based emotional speech modeling system according to claim 2, characterized in that: The data storage module includes a data classification storage unit, a data backup and recovery unit and a storage resource management unit. The data classification storage unit is used to classify and store different types of data, including original data, feature data and model data. The data backup and recovery unit is used to regularly back up important data and restore data when necessary. The storage resource management unit is used to manage storage resources, detect whether there is enough space to store data, and optimize storage efficiency and storage space. The user interface module includes a user input receiving unit, a result display unit and a user interaction response unit. The user input receiving unit is used to receive voice data and related information of instructions input by the user. The result display unit is used to display the results generated by the system to the user in an intuitive manner. The results include voice playback and graphic display. The user interaction response unit is used to promptly respond to the user's operations and feedback during the interaction process.

Images