Voice synthesis method and system for AI accompanying robot based on voiceprint recognition

Abstract

The invention relates to the technical field of voice signal processing, and particularly discloses an AI accompanying robot voice synthesis method and system based on voiceprint recognition, and the method comprises the steps: collecting a user voice signal, carrying out the preprocessing, and obtaining time-frequency distribution data and a system state parameter set; based on a time-frequency analysis result, extracting robust identity features robust to emotion changes; based on a nonlinear dynamic analysis result, extracting dynamic emotional characteristics such as emotional confusion degree, stability, intensity baseline and fluctuation intensity; standardizing the two types of features and carrying out dynamic weighted fusion according to confidence to form a decoupling feature vector for executing identity confirmation and emotion recognition in parallel; when low-confidence identity judgment and high-intensity emotion conflict occur, historical interaction records are introduced for arbitration correction; and finally generating a synthesis instruction of emotional rhythm control, and driving an end-to-end speech synthesis model to output natural speech with both user identity timbre and corresponding emotional color.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech signal processing, in particular to an AI companion robot speech synthesis method and system based on voiceprint recognition. BACKGROUND

[0002] With the in-depth development of artificial intelligence technology, AI companion robots based on voiceprint recognition are gradually applied to emotional care, remote communication and personalized services in daily life. In order to improve the naturalness and emotional temperature of interaction, such systems need to have two core capabilities: one is to accurately identify the identity of the interlocutor and call the corresponding personalized voice for speech synthesis; the second is to perceive the emotional state of the interlocutor and give the corresponding emotional color in the synthesized speech. The prior art usually adopts an independent voiceprint recognition module and an emotion recognition module to process user speech in parallel, and then drives the speech synthesis system to work.

[0003] The prior art has the following disadvantages: The system reliability problem caused by acoustic feature coupling interference in the prior art. When the emotion fluctuates violently, this coupling interference is particularly significant: the voiceprint model will misjudge the strong emotional features as permanent drift of the speaker identity features, resulting in voiceprint verification failure or identity misidentification, which destroys the continuity and specificity of the interaction; at the same time, the emotion recognition model may also be disturbed by the individual-specific voice, affecting its accurate extraction of cross-user general emotion representation. This strong coupling of emotion-identity features leads to a chain of error decisions by the system in critical application scenarios. SUMMARY

[0004] The purpose of the present application is to provide an AI companion robot speech synthesis method and system based on voiceprint recognition to solve the problems in the above background.

[0005] The purpose of the present application can be achieved by the following technical solutions: An AI companion robot speech synthesis method based on voiceprint recognition, comprising the following steps: S1: Collecting user original speech signals, performing synchronous parallel analysis processing thereon, and obtaining first analysis results and second analysis results; wherein the first analysis results are time-frequency distribution data obtained by multi-resolution time-frequency analysis, and the second analysis results are a set of system state parameters obtained by nonlinear dynamics analysis; S2: Based on the first analysis results, applying an adaptive filtering strategy to suppress the transient components in the time-frequency distribution data related to emotional fluctuations, and extracting robust identity features representing the user's inherent acoustic structure; S3: Based on the second analysis results, extracting dynamic emotional features representing the user's current emotional state from the set of system state parameters; S4: fusing the robust identity feature and the dynamic emotion feature to form a decoupling feature vector; performing user identity confirmation and user emotion state recognition in parallel based on the decoupling feature vector, and generating corresponding personalized speech synthesis instructions; S5: calling a target speech synthesis model corresponding to the confirmed identity based on the speech synthesis instruction; the target speech synthesis model takes the text content to be synthesized as input, and regulates the prosody parameters of the synthesized speech under the dynamic emotion feature, and outputs synthesized speech with target identity timbre and corresponding emotional color.

[0006] As a further scheme of the present application: the acquisition process of the first analysis result is: The original speech signal of the user is decomposed by using complex wavelet packet transform, a symmetric complex wavelet with linear phase is used as a mother wavelet, the signal is decomposed to a preset depth, and a complete sub-band tree is formed; The modulus square of the complex wavelet coefficient of each terminal node in the complete sub-band tree is calculated, the energy distribution sequence of each sub-band on the time axis is obtained, and an initial time-frequency distribution is constructed; The initial time-frequency distribution is corrected by weighting, and the weight factor of the weighting correction has a nonlinear relationship with the center frequency of the sub-band, so as to simulate the sensitivity of the human ear to different frequencies; The time-frequency distribution data after the weighting correction is output as the first analysis result.

[0007] As a further scheme of the present application: the acquisition process of the second analysis result is: The original speech signal of the user is reconstructed in phase space, and the phase space trajectory of the speech dynamic system is reconstructed by using the time delay method and the embedding dimension method; On the basis of the reconstructed phase space, the multi-scale permutation entropy representing the complexity of the system and the maximum Lyapunov exponent representing the chaotic characteristics of the system are calculated; The recurrence rate and the determinism rate quantitative parameters in the recurrence plot of the phase space trajectory are calculated, the recurrence rate is used to describe the stability of the system state, and the determinism rate is used to distinguish deterministic processes and random noise; The multi-scale permutation entropy, the maximum Lyapunov exponent, the recurrence rate and the determinism rate jointly constitute the system state parameter set as the second analysis result.

[0008] As a further scheme of the present application: the robust identity feature representing the inherent acoustic structure of the user specifically includes: Based on the time-frequency distribution data after the weighting correction, the vowel core section with energy concentration and duration meeting the stability threshold in the speech section is detected; In the vowel core segment, the trajectories of the first three formant frequencies are tracked, and the frequency variance and energy stationarity of each formant trajectory in the segment are calculated; Taking the frequency variance and energy stationarity as stability indicators, an adaptive filtering weight map corresponding to the time dimension of the time-frequency distribution data is generated; The adaptive filtering weight map is applied to the time-frequency distribution data for point-by-point weighted filtering to suppress transient components, and the filtered data is reduced in dimension and features are extracted to generate the robust identity feature.

[0009] As a further scheme of the present application: the dynamic emotional feature representing the current emotional state of the user specifically includes: Based on the system state parameter set, the entropy value of the multi-scale permutation entropy on the fine scale is calculated as an emotional chaos degree indicator, and the ratio of the recurrence rate to the determinism rate is taken as an emotional stability indicator; In a preset sliding time window, the mean and variance of the maximum Lyapunov exponent in multiple consecutive time windows are calculated as an emotional intensity baseline value and an emotional fluctuation intensity value, respectively; The emotional chaos degree indicator, the emotional stability indicator, the emotional intensity baseline value, and the emotional fluctuation intensity value are normalized and vectorized to form a comprehensive emotional description vector; The comprehensive emotional description vector is taken as the dynamic emotional feature representing the current emotional state of the user.

[0010] As a further scheme of the present application: the robust identity feature and the dynamic emotional feature are fused to form a decoupled feature vector, specifically including: The robust identity feature and the dynamic emotional feature are mapped to a unified feature space and standardized based on Z-score to eliminate the dimensional influence; A real-time confidence score is calculated for the robust identity feature and the dynamic emotional feature, respectively; and a fusion weight is assigned according to the proportion of the real-time confidence score; The robust identity feature and the dynamic emotional feature with the assigned weight are vector-spliced, and the high-dimensional vector after splicing is subjected to principal component analysis for dimension reduction to obtain the low-dimensional and compact decoupled feature vector.

[0011] As a further scheme of the present application: the user identity confirmation and user emotional state recognition are performed, specifically including: The decoupled feature vector is input into a pre-trained identity confirmation classifier and an emotional state classifier to obtain an initial identity and an initial emotional category, respectively; establishing a conflict arbitrator, when the confidence probability of the initial identity identification is lower than a first threshold value, and the emotional intensity corresponding to the initial emotional category is higher than a second threshold value, it is determined that an identity-emotion identification conflict occurs; When the conflict occurs, the conflict arbitrator calls the historical identity record of the last several successful interactions, and adopts the majority voting principle to correct the current initial identity identification, and generates a final identity confirmation result and a final emotional state identification result; According to the final identity confirmation result and the final emotional state identification result, a personalized speech synthesis instruction is formed.

[0012] As a further scheme of the present application: the specific construction process of the speech synthesis model comprises: Collecting the calm state speech data of the target speaker, performing text transcription, audio quality inspection and prosody smoothness screening on the collected data, and constructing a standardized personalized speech data set; An end-to-end neural network speech synthesis model is constructed, which comprises a text encoder, a speaker embedding vector extractor, a prosody controller and an acoustic feature decoder; A large-scale multi-speaker speech data set is used to pre-train the speech synthesis model, so that it has general speech synthesis capability; then the speech synthesis model is fine-tuned using the standardized personalized speech data set, and the speaker embedding vector extractor is optimized to accurately capture the timbre characteristics of the target speaker; The fine-tuned speech synthesis model is tested using a verification set voice that does not participate in the training, and when the acoustic feature similarity between the synthesized voice and the original voice of the target speaker reaches a predetermined threshold, the construction of the speech synthesis model is completed.

[0013] As a further scheme of the present application: the output of the synthesized speech with the target identity timbre and the corresponding emotional color, specifically comprises: The dynamic emotional features are input into a pre-trained emotion-prosody mapping network, and the emotion-prosody mapping network outputs a set of specific and quantifiable target prosody parameters, including pitch offset, speech rate scaling factor and energy intensity factor; The text encoder synchronously predicts the natural prosody baseline parameters corresponding to the text content and the neutral emotion when encoding the text to be synthesized; The target prosody parameters and the natural prosody baseline parameters are weighted and fused; The acoustic feature decoder generates acoustic features containing target emotional color based on the output of the text encoder, the speaker embedding vector and the fused prosody parameters, and finally converts the synthesized speech by the vocoder; The training process of the emotion-prosody mapping network is: constructing a database containing a large number of speech samples and their corresponding texts, speaker identities and accurate emotion labels; initializing the emotion-prosody mapping network and a prosody parameter discriminator, the discriminator being used to distinguish between predicted prosody parameters and prosody parameters extracted in real speech; During the training process, the goal of the emotion-prosody mapping network is to generate prosody parameters conforming to the target emotion which are sufficient to deceive the discriminator, while the goal of the discriminator is to accurately identify the authenticity of the parameters; Through repeated iterations, the emotion-prosody mapping network learns to generate prosody parameters from input emotion features.

[0014] The AI companion robot speech synthesis system based on voiceprint recognition comprises: A signal analysis and feature decoupling module is used to collect original speech signals of a user, perform synchronous parallel analysis processing thereon, and obtain first analysis results and second analysis results; wherein the first analysis results are time-frequency distribution data obtained through multi-resolution time-frequency analysis, and the second analysis results are a set of system state parameters obtained through nonlinear dynamics analysis; A robust identity feature extraction module is used to suppress transient components related to emotional fluctuations in the time-frequency distribution data based on the first analysis results by applying an adaptive filtering strategy, and extract robust identity features representing the inherent acoustic structure of the user; A dynamic emotion feature extraction module is used to extract dynamic emotion features representing the current emotional state of the user from the set of system state parameters based on the second analysis results; A fusion decision and instruction generation module is used to fuse the robust identity features and the dynamic emotion features to form a decoupled feature vector; based on the decoupled feature vector, user identity confirmation and user emotion state recognition are performed in parallel, and corresponding personalized speech synthesis instructions are generated; An emotional personalized speech synthesis module is used to call a target speech synthesis model corresponding to the confirmed identity based on the speech synthesis instructions; the target speech synthesis model takes the text content to be synthesized as input, and regulates the prosody parameters of the synthesized speech under the control of the dynamic emotion features, and outputs synthesized speech with target identity timbre and corresponding emotional color.

[0015] The present application has the following advantages: (1) By combining voiceprint recognition technology and emotion analysis, the present application can extract information reflecting the identity characteristics and current emotional state of the user from the user's voice. This not only makes the generated voice have the user's unique timbre characteristics, but also dynamically adjusts the prosodic parameters (such as intonation, speech rate, intensity, etc.) of the voice according to the user's emotional state, so that the response of the AI companion robot is more in line with the user's emotional state, providing a more natural and personalized communication experience.

[0016] (2) The present application uses advanced signal processing techniques (such as multi-resolution time-frequency analysis and nonlinear dynamics analysis) to deeply analyze the voice signal, effectively suppressing the interference factors caused by emotional fluctuations, ensuring the stability and accuracy of identity feature extraction. At the same time, by constructing an efficient emotion-prosody mapping network, the precise conversion from emotional features to voice prosodic parameters is realized. Such design greatly enhances the authenticity of voice synthesis, making the robot's response closer to human natural communication. BRIEF DESCRIPTION OF DRAWINGS

[0017] The present application will be further described below in conjunction with the accompanying drawings.

[0018] Figure 1 is a flowchart of the method of the present application; Figure 2 is a flowchart of the system of the present application. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0020] Please refer to Figure 1 The present application is an AI companion robot voice synthesis method based on voiceprint recognition, which includes the following steps: S1: Collecting the user's original voice signal and performing synchronous parallel analysis processing to obtain the first analysis result and the second analysis result; wherein the first analysis result is the time-frequency distribution data obtained by multi-resolution time-frequency analysis, and the second analysis result is the system state parameter set obtained by nonlinear dynamics analysis; S2: Based on the first analysis result, apply adaptive filtering strategy to suppress the transient components related to emotional fluctuations in the time-frequency distribution data, and extract the robust identity features representing the user's inherent acoustic structure; S3: Based on the second analysis result, extract the dynamic emotional features representing the user's current emotional state from the system state parameter set; S4: fuse the robust identity feature and the dynamic emotion feature to form a decoupled feature vector; based on the decoupled feature vector, perform user identity confirmation and user emotion state recognition in parallel, and generate corresponding personalized speech synthesis instructions; S5: based on the speech synthesis instructions, call a target speech synthesis model corresponding to the confirmed identity; the target speech synthesis model takes the text content to be synthesized as input, and regulates the prosody parameters of the synthesized speech under the control of the dynamic emotion feature, and outputs synthesized speech with target identity timbre and corresponding emotional color.

[0021] In S1, the user's original speech signal is collected, and a synchronous parallel analysis process is performed to obtain a first analysis result and a second analysis result; wherein the first analysis result is time-frequency distribution data obtained by multi-resolution time-frequency analysis, and the second analysis result is a set of system state parameters obtained by nonlinear dynamics analysis, which specifically includes: The system collects the user's original speech signal through a microphone array installed on the robot body. The microphone array adopts a ring layout, which can effectively receive speech signals from different directions. During the collection process, the system first performs preliminary suppression of environmental noise, and uses a noise reduction algorithm based on spectral subtraction to preprocess the original signal. The preprocessed speech signal is divided into speech segments and non-speech segments by calculating the short-time energy and zero-crossing rate. Only when the signal is confirmed as a valid speech segment, it will enter the subsequent processing flow. The confirmed speech segment is then pre-emphasized, and a first-order high-pass filter is used to enhance high-frequency components to compensate for the high-frequency attenuation of the speech signal during transmission. The preprocessed speech signal is divided into overlapping frames, each with a length of 25 milliseconds and a frame shift of 10 milliseconds, providing standardized input data for subsequent time-frequency analysis and nonlinear analysis.

[0022] The acquisition process of the first analysis result: The first analysis result is time-frequency distribution data obtained by multi-resolution time-frequency analysis. This process uses complex wavelet packet transform as the core analysis method. First, a symmetric complex wavelet with linear phase characteristics is selected as the mother wavelet, which ensures the accuracy of the phase information during signal decomposition. The preprocessed speech signal is input into the wavelet packet transform algorithm, which decomposes the signal completely according to the preset decomposition depth, forming a complete sub-band tree structure. Each terminal node of this sub-band tree corresponds to a specific frequency band range.

[0023] Next, the system calculates the modulus square of the complex wavelet coefficients corresponding to each terminal node. This calculation is performed along the time axis, thus obtaining the energy distribution of each sub-band at different time points. All these sub-band energy distribution sequences together constitute the initial time-frequency distribution map. This distribution map reflects the change in energy of the speech signal at different frequency components over time.

[0024] The system then performs weighted correction on the initial time-frequency distribution. The weighting factors used in the correction process have a non-linear relationship with the center frequency values ​​of each sub-band. This relationship is designed to simulate the sensitivity characteristics of the human auditory system to different frequencies, resulting in a higher weight allocation for the mid-frequency region compared to the low- and high-frequency regions. The weighted-corrected time-frequency distribution data is then output as the first analytical result, providing a time-frequency representation that better reflects human auditory perception for subsequent identity feature extraction.

[0025] The process of obtaining the second analysis result: The second analytical result is the set of system state parameters obtained through nonlinear dynamics analysis. This process begins with phase space reconstruction of the preprocessed speech signal. Phase space reconstruction is accomplished using a combination of time delay and embedding dimension methods. By selecting the time delay parameter and embedding dimension, the one-dimensional speech time series is reconstructed into a high-dimensional phase space trajectory. This trajectory reflects the dynamic characteristics of the speech generation system.

[0026] Based on the reconstructed phase space, the system calculates several nonlinear dynamic characteristic parameters. Among them, the multi-scale permutation entropy is used to characterize the complexity of the system at different time scales. To calculate this parameter, the phase space trajectories are first coarsened at multiple scales, and then the permutation entropy value is calculated at each scale. The maximum Lyapunov exponent is used to quantify the chaotic characteristics of the system; its value is obtained by calculating the divergence rate of adjacent trajectories in the phase space.

[0027] The system also calculates two important quantitative parameters in the phase space trajectory recurrence graph: the recurrence rate and the deterministic rate. The recurrence rate is obtained by statistically analyzing the proportion of recursive points to the total number of points in the recurrence graph, and is used to characterize the stability of the system state. The deterministic rate is obtained by calculating the proportion of diagonal structures formed in the recurrence graph to all recursive points, and is used to distinguish the influence of deterministic processes from random noise. Finally, the four parameters—multi-scale permutation entropy, maximum Lyapunov exponent, recurrence rate, and deterministic rate—are combined into a system state parameter set, which is output as the second analytical result, providing a feature representation based on nonlinear dynamics theory for subsequent emotion feature extraction.

[0028] In S2, based on the first analysis result, an adaptive filtering strategy is applied to suppress transient components related to emotional fluctuations in the time-frequency distribution data, and robust identity features characterizing the user's inherent acoustic structure are extracted, specifically including: The process involves detecting vowel core segments. This process takes weighted and corrected time-frequency distribution data as input, which contains complete information on the energy changes of the speech signal across different frequency sub-bands over time. The system first calculates the sum of the energy of all sub-bands at each moment in the time dimension, obtaining a short-time energy profile of the speech signal. Based on this energy profile, the system identifies active speech segments with energy exceeding a preset threshold. Within each active speech segment, the system further analyzes the energy distribution characteristics in the frequency dimension, searching for time periods where energy is concentrated in the mid-to-low frequency region (usually corresponding to the vowel formant frequency range) and where the energy distribution is relatively concentrated. For these candidate segments, the system calculates their duration; only segments whose duration exceeds a set stability threshold (e.g., 100 milliseconds) are confirmed as vowel core segments. This stability threshold ensures that the selected segments are long enough to contain a stable formant structure, providing a reliable data foundation for subsequent analysis.

[0029] After identifying the vowel core region, the system enters the formant trajectory tracking and stability analysis phase. For each identified vowel core region, the system first accurately estimates the frequency values ​​of the first three formants using linear predictive coding analysis or cepstral analysis. These formant frequencies are directly related to the speaker's vocal tract shape and are key features characterizing the speaker. The system continuously tracks the frequency changes of these three formants within the entire vowel core region, forming formant trajectory curves, with each analysis frame lasting 10 milliseconds. The system calculates the frequency variance of each trajectory within the entire region, reflecting the degree of fluctuation in the formant frequencies. The system also calculates the energy stability within the corresponding frequency band for each formant. These two indicators—frequency variance and energy stability—together constitute the quantitative basis for evaluating the stability of the formants.

[0030] Based on the formant stability analysis, the system generates an adaptive filter weight map. This process transforms the stability index into a specific filter. For each time point in the time-frequency distribution data, the system assigns a weight value based on the stability assessment of the vowel core region to which that time point belongs. Specifically, in time periods with smaller formant frequency variance and higher energy stability, the system assigns a higher weight value (close to 1), indicating that the sound characteristics of these time periods are more likely to reflect the speaker's stable vocal tract characteristics; while in areas with larger formant fluctuations or non-vowel regions, a lower weight value (close to 0) is assigned, indicating that the sound characteristics of these time periods may be more affected by factors such as emotions. The weight value allocation uses a continuous function mapping to ensure a smooth transition of the weight map in the time dimension, avoiding distortion caused by abrupt changes. The final generated weight map has the same time dimension as the original time-frequency distribution data, and assigns a corresponding weight to each time point.

[0031] The final stage involves weighted filtering and robust identity feature extraction. The system performs point-by-point multiplication between the adaptive filtering weight map and the weighted and adjusted time-frequency distribution data to achieve adaptive filtering in the time-frequency domain. This filtering process effectively suppresses transient components related to emotional fluctuations in the time-frequency distribution while preserving stable components representing the user's inherent acoustic structure. The filtered time-frequency data undergoes dimensionality reduction in the frequency dimension, typically using methods such as principal component analysis or linear discriminant analysis to convert the high-dimensional time-frequency representation into a low-dimensional feature vector. This feature vector mainly contains information closely related to the speaker's vocal tract characteristics, such as formant parameters after stability weighting and the shape of the spectral envelope, ultimately forming the robust identity features required by the system. This feature exhibits good robustness to emotional changes, providing reliable input data for subsequent voiceprint recognition.

[0032] In S3, based on the second parsing result, dynamic emotional features representing the user's current emotional state are extracted from the system state parameter set; The extraction process of dynamic emotion features is based on a system state parameter set obtained through nonlinear dynamics analysis. This system state parameter set includes quantitative parameters such as multi-scale permutation entropy, maximum Lyapunov exponent, recursion rate, and determinism rate. These parameters describe the characteristics of the dynamic system behind the speech signal from different perspectives. The extraction process first requires calculating four secondary indicators with clear emotion indications from these basic parameters.

[0033] The emotional confusion index is obtained by calculating the entropy value of multi-scale permutation entropy at a fine scale. The analysis of multi-scale permutation entropy requires first performing multi-scale coarse-grained processing on the original time series. At the fine scale, the detailed features of the time series are preserved to the greatest extent. The calculation process of permutation entropy includes steps such as phase space reconstruction, state sequence symbolization, and calculation of permutation pattern probability distribution. The permutation entropy value obtained at the fine scale can effectively reflect the irregularity and complexity of the speech signal at a short time scale. This complexity is directly related to the physiological state changes when the speaker is emotionally agitated. When emotionally agitated, the autonomic nervous system leads to a decrease in the control precision of the vocal organs, resulting in higher randomness and complexity in the speech signal. Therefore, the permutation entropy value at the fine scale can serve as an effective indicator of emotional confusion.

[0034] The Emotional Stability Index is calculated by dividing the recurrence rate by the deterministic rate. The recurrence rate represents the probability that a dynamic system's state will revert to a neighboring region in phase space; a higher recurrence rate indicates greater system stability. The deterministic rate quantifies the proportion of diagonal structures in the recurrence graph, reflecting the degree of determinism in the system's dynamics. The Emotional Stability Index comprehensively assesses the stability of emotional states through the ratio of these two factors. A high ratio indicates a stable emotional state, while a significantly decreased ratio reflects substantial fluctuations in the emotional state. This index is particularly suitable for detecting transitional changes in emotional states.

[0035] The calculation of emotion intensity-related indicators needs to be performed within a pre-defined sliding time window. The length of the sliding time window is typically set to 2 seconds to balance the temporal resolution and stability requirements of the feature. At each window position, the mean of the maximum Lyapunov exponent within that window is calculated. This mean reflects the average intensity of the system's chaotic characteristics within that time period and can be used as a baseline value for emotion intensity. Simultaneously, the variance of the maximum Lyapunov exponent within the same window is calculated. This variance reflects the severity of emotion fluctuations and is called the emotion fluctuation intensity value. The maximum Lyapunov exponent characterizes the system's sensitivity to initial conditions, i.e., the magnitude of chaotic characteristics. As emotion intensity increases, the instability of the vocal system increases, leading to an increase in the maximum Lyapunov exponent value.

[0036] After obtaining the four basic sentiment indicators, data standardization is required. Since the dimensions and numerical ranges of each indicator differ, directly concatenating them would lead to some indicators dominating the overall vector. Standardization employs a min-max normalization method, linearly transforming the value of each indicator to the range of 0 to 1. Specifically, for each indicator, its minimum and maximum values ​​are calculated across the entire training dataset. Then, each original value is subtracted from the minimum value and divided by the difference between the maximum and minimum values. This process ensures that different indicators are numerically comparable.

[0037] Finally, the four standardized emotion indicators are vectorized and concatenated in a fixed order to form a four-dimensional comprehensive emotion description vector. Each dimension of this vector corresponds to the degree of emotional confusion, emotional stability, baseline emotional intensity, and intensity of emotional fluctuation, respectively. This comprehensive vector, as the final emotion feature, can comprehensively describe the speaker's current emotional state from multiple complementary perspectives. In practical applications, this feature vector will be input into an emotion classifier for the identification and classification of emotional states.

[0038] The four emotion indicators include: emotion confusion index, emotion stability index, emotion intensity baseline value, and emotion fluctuation intensity value.

[0039] In S4, robust identity features and dynamic emotion features are fused to form a decoupled feature vector. Based on the decoupled feature vector, user identity verification and user emotion state recognition are performed in parallel, and corresponding personalized speech synthesis instructions are generated, specifically including: The process of fusing robust identity features with dynamic emotion features first requires feature space alignment and standardization. Robust identity features typically include spectral envelope parameters reflecting the speaker's vocal tract structure, while dynamic emotion features include nonlinear dynamic parameters reflecting emotional state. Since these two types of features have different physical meanings and dimensions, direct fusion would lead to features with larger numerical ranges dominating subsequent classification. Therefore, it is necessary to first map these two types of features into a unified feature space, which is usually a higher-dimensional vector space capable of simultaneously accommodating the expression of both identity and emotion information.

[0040] Next, a Z-score-based standardization method is used to process the two types of features separately. For robust identity features, the mean and standard deviation on the training dataset are calculated. Then, the mean is subtracted from each feature value and divided by the standard deviation, resulting in a processed feature with a mean of 0 and a standard deviation of 1. The same operation is applied to dynamic sentiment features. This processing ensures that different features are numerically comparable and eliminates the influence of differences in units of measurement.

[0041] Based on standardization, the system calculates a real-time confidence score for each feature type. For robust identity features, the confidence score is primarily based on the signal-to-noise ratio (SNR) of the current speech frame and the stability of the vowel core segment. A higher SNR and more stable vowel segment formant trajectories result in a higher confidence score. For dynamic emotion features, the confidence score is based on the inherent consistency of nonlinear dynamic parameters, such as whether the parameters exhibit a pattern consistent with a specific emotional state. The confidence score is calculated using a sliding window approach, combining feature values ​​from the current frame and several preceding and following frames for comprehensive evaluation.

[0042] Based on the calculated confidence scores of the two types of features, the system allocates fusion weights proportionally. Specifically, the confidence scores of robust identity features and dynamic emotion features are added together to obtain the total score, and the fusion weight of each feature is equal to the proportion of its confidence score in the total score. This dynamic weight allocation mechanism ensures that identity features occupy a larger proportion of the fused features when the acoustic environment is ideal and the speech quality is high; while when emotional expression is obvious and nonlinear features are significant, the proportion of emotion features increases accordingly.

[0043] The two types of features, with pre-assigned weights, are concatenated to form a high-dimensional fused feature vector. The dimension of this vector equals the sum of the dimensions of the robust identity features and the dynamic sentiment features. To avoid the curse of dimensionality and extract the most effective information, principal component analysis (PCA) is performed to reduce the dimensionality of this high-dimensional vector. PCA first calculates the covariance matrix of the feature vectors on a large training dataset, then obtains the eigenvalues ​​and eigenvectors of this matrix. After sorting the eigenvalues ​​from largest to smallest, the eigenvectors corresponding to the top k eigenvalues ​​are selected as the projection matrix, projecting the original high-dimensional vector into a low-dimensional space. This results in a decoupled feature vector with reduced dimensionality and highly concentrated information.

[0044] User identification and emotion state recognition: The decoupled feature vectors are simultaneously input into two pre-trained classifiers for parallel recognition. The identity verification classifier typically employs a support vector machine or deep neural network structure, which has already learned the feature space distribution patterns corresponding to different speaker identities during the training phase. The emotion state classifier uses a similar classification algorithm, but its training objective is to classify different emotion categories. The two classifiers output initial identity identifiers and initial emotion categories, respectively, along with corresponding confidence probabilities. These probabilities reflect the classifier's degree of certainty regarding its current judgment.

[0045] The system establishes a conflict arbitration mechanism to handle potential inconsistencies in recognition results. The conflict arbitrator monitors the outputs of the two classifiers in real time. When the confidence probability output by the identity verification classifier is lower than a preset first threshold (e.g., 0.6), and the emotion intensity corresponding to the emotion category output by the emotion state classifier is higher than a second threshold (e.g., 0.7), an identity-emotion recognition conflict is determined to have occurred. This type of conflict typically occurs in scenarios where drastic fluctuations in the user's emotions cause significant changes in acoustic characteristics.

[0046] Once a conflict is identified, the conflict arbitrator initiates a correction procedure. The system maintains a buffer of historical interaction records, storing the identity verification results of the most recent number of successful interactions (e.g., 5). A successful interaction is defined as an identity verification confidence probability higher than a high threshold (e.g., 0.9). The arbitrator retrieves these historical identity records from the buffer and uses a majority voting principle to determine the most likely correct identity. If a particular identity appears more than half the time, that identity is used as the final identity verification result, overriding the current low-confidence initial identity. Simultaneously, the emotion state recognition result remains unchanged and is used as the final emotion state recognition result.

[0047] Finally, the system generates personalized speech synthesis instructions based on the corrected identity verification and emotion state recognition results. These instructions consist of two key parts: a target speech synthesis model selection instruction and a prosodic parameter control instruction. The target speech synthesis model selection is based on the final identity verification result; the system chooses the model corresponding to the user's identity from a pre-stored pool of personalized speech synthesis models. The prosodic parameter control instruction, based on the final emotion state recognition result, maps the emotion category to specific control parameters such as fundamental frequency curve, speech rate, and energy intensity. These two parts together constitute the complete personalized speech synthesis instruction, which is transmitted to the speech synthesis engine to execute subsequent speech generation operations.

[0048] The entire processing adopts a pipelined architecture to ensure real-time performance from feature fusion to instruction generation. The system employs an appropriate buffering mechanism to handle continuous input of the speech stream, ensuring that each speech frame is processed promptly while maintaining contextual coherence. The historical interaction record buffer uses a first-in, first-out (FIFO) strategy for updates, ensuring that the most recent interaction data is saved, thus improving the accuracy of conflict arbitration.

[0049] In S5, based on the speech synthesis instruction, the target speech synthesis model corresponding to the confirmed identity is invoked. The target speech synthesis model takes the text content to be synthesized as input and its prosodic parameters are adjusted by dynamic emotional features to output synthesized speech with the target identity's timbre and corresponding emotional coloring. Specifically, this includes: The construction of the speech synthesis model begins with the acquisition of speech data from the target speaker. Speech data recorded by the target speaker in a calm state should be collected, with a total duration recommended to be no less than 30 minutes, to ensure sufficient coverage of the speaker's vocal characteristics. Recording should be conducted in an acoustically treated environment using professional recording equipment to ensure a signal-to-noise ratio higher than 35 dB. The acquired raw speech data is first transcribed into text to ensure a complete correspondence between the speech content and the text script. Then, rigorous audio quality checks are performed, including removing speech segments containing obvious coughs, pauses, or background noise, and checking for overloaded or underloaded audio amplitudes. Prosodic stationarity screening is achieved by analyzing the fundamental frequency profile and energy changes of speech segments, retaining only stable speech segments with fundamental frequency and energy fluctuations within a preset range for subsequent training. Through these steps, a high-quality, standardized, personalized speech dataset is constructed.

[0050] The constructed speech synthesis model employs an end-to-end neural network architecture comprising four core components. The text encoder is responsible for converting the input text sequence into a high-dimensional semantic feature representation, typically using an attention-based encoder structure to capture contextual information within the text. The speaker embedding vector extractor is a crucial component, extracting a fixed-dimensional vector representation from the input speech. This vector should uniquely characterize the timbre of the target speaker, unaffected by text content or prosodic style. The prosodic controller receives external prosodic control parameters and converts them into feature representations influencing the prosodic nature of the synthesized speech. The acoustic feature decoder is the final core component, generating corresponding acoustic features, typically Mel spectrograms, conditioned on the text encoder's output, the speaker embedding vector, and the prosodic control features.

[0051] The model training employs a two-stage strategy. The first stage involves pre-training using a large-scale multi-speaker speech dataset containing thousands of hours of speech data from hundreds of different speakers. The goal of pre-training is to enable the model to master the general ability to convert text to speech, including learning text-to-phoneme mappings and basic prosodic patterns. In the second stage, the pre-trained model is fine-tuned using a prepared standardized personalized speech dataset. During fine-tuning, most model parameters are updated with a small learning rate, while parameters related to the speaker embedding vector extractor are optimized with a larger learning rate. This training strategy allows the model to accurately capture the unique vocal characteristics of the target speaker while maintaining its general speech synthesis capabilities.

[0052] Model validation is the final step in the construction process. The fine-tuned model is tested using a validation set of speech samples (approximately 20% of the total data) that were not used in the training process. During testing, the same text is input into the model for speech synthesis, and the similarity in acoustic features between the synthesized speech and the original speech of the target speaker is calculated. Several similarity metrics are used, including Mel-frequency cepstral distortion, fundamental frequency contour correlation coefficient, and spectral envelope similarity. When the weighted composite score of all metrics reaches a predetermined threshold (e.g., above 0.85), the model is considered usable, and the speech synthesis model construction is complete.

[0053] Emotional speech generation and prosodic control: The generation of emotional speech begins with the calculation of prosodic parameters. Dynamic emotional features are input into a pre-trained emotion-prosodic mapping network, which maps these features to a set of quantifiable target prosodic parameters through a multi-layered neural network structure. The fundamental frequency offset represents the proportion of fundamental frequency change relative to neutral speech; a positive value indicates a higher fundamental frequency, and a negative value indicates a lower fundamental frequency. The speech rate scaling factor controls the playback speed of the speech; a value greater than 1 indicates a faster speech rate, and a value less than 1 indicates a slower speech rate. The energy intensity factor adjusts the dynamic range of speech amplitude, affecting the perceived emphasis of the speech. These parameters collectively define the specific acoustic representation of the target emotion.

[0054] During the text encoding stage, the text encoder not only converts the input text into semantic features but also simultaneously predicts sentiment-neutral prosodic baseline parameters corresponding to the text content. This process is based on linguistic prosodic rules, such as interrogative sentences typically having rising intonation, declarative sentences having falling intonation, and key words requiring stress. The predicted baseline parameters include fundamental frequency curves, phoneme durations, and energy distributions, which constitute a default prosodic pattern that conforms to linguistic rules.

[0055] The target prosodic parameters and the natural prosodic baseline parameters need to be weighted and fused. The fusion weights are dynamically adjusted according to the intensity of emotional features. When the emotional intensity is high, the target prosodic parameters have a larger weight; when the emotional intensity is low, more neutral prosodic baseline parameters are retained. The fused prosodic parameters maintain the prosodic rules of the language itself while incorporating the prosodic features of emotional expression, thus avoiding the problem of unnatural prosody in synthesized speech.

[0056] The acoustic feature decoder uses the text encoder's output as its content base, the speaker embedding vector as its timbre condition, and the fused prosodic parameters as its prosodic control condition to synthesize corresponding acoustic features through a conditional generation mechanism. This process typically employs an autoregressive generation approach, where the result of each generation step influences subsequent steps. The generated acoustic features are ultimately fed into a neural vocoder to convert them into a time-domain waveform, i.e., the final playable speech signal. The vocoder uses a pre-trained general model capable of reconstructing the acoustic features into a high-quality, natural, and fluent speech waveform.

[0057] Training methods for emotion-prosody mapping networks: Training an emotion-prosody mapping network requires building a large-scale emotional speech database. This database contains tens of thousands of speech samples with precise emotion labels. Each sample corresponds to a text transcription, speaker identification information, and manually annotated emotion labels. The emotion labels are annotated using a combination of discrete emotion categories and continuous emotion dimensions, including basic categories such as anger, happiness, sadness, and calmness, as well as valence, arousal, and dominance scores.

[0058] The network training employs an adversarial training framework. During the initialization phase, both the emotion-prosodic mapping network and the prosodic parameter discriminator need to be initialized simultaneously. The emotion-prosodic mapping network uses a feedforward neural network structure, taking emotion feature vectors as input and outputting prosodic parameter vectors. The prosodic parameter discriminator uses a convolutional neural network structure to determine whether the input prosodic parameters are true parameters or network-generated parameters.

[0059] During training, the two networks engage in adversarial learning. The goal of the emotion-prosodic mapping network is to generate prosodic parameters that are as similar as possible to those extracted from real emotional speech, making it impossible for the discriminator to distinguish between them. The discriminator, on the other hand, strives to accurately differentiate between real prosodic parameters and those generated by the network. This adversarial training prompts the generative network to continuously improve its generation quality, ultimately producing highly realistic emotional prosodic parameters.

[0060] The training process involves iterative optimization. Each iteration comprises two phases: first, the discriminator is trained using a set of real prosodic parameters and network-generated parameters, updating the discriminator parameters to improve its discrimination ability; then, the generative network is trained, fixing the discriminator parameters and updating the generative network parameters by using generated parameters to deceive the discriminator's target. After thousands of iterations, when the discriminator can no longer effectively distinguish between real and generated parameters, the training is considered converged, and the emotion-prosodic mapping network has learned the accurate mapping relationship from emotion features to prosodic parameters.

[0061] Please see Figure 2 As shown, the AI ​​companion robot voice synthesis system based on voiceprint recognition includes: The signal analysis and feature decoupling module is used to acquire the user's original speech signal, perform synchronous parallel analysis processing on it, and obtain a first analysis result and a second analysis result; wherein, the first analysis result is time-frequency distribution data obtained through multi-resolution time-frequency analysis, and the second analysis result is a set of system state parameters obtained through nonlinear dynamic analysis; A robust identity feature extraction module, based on the first parsing result, applies an adaptive filtering strategy to suppress transient components related to emotional fluctuations in the time-frequency distribution data, and extracts robust identity features that characterize the user's inherent acoustic structure. A dynamic emotion feature extraction module, which extracts dynamic emotion features representing the user's current emotional state from the system state parameter set based on the second parsing result; The fusion decision and instruction generation module fuses robust identity features and dynamic emotion features to form a decoupled feature vector; based on the decoupled feature vector, user identity verification and user emotion state recognition are performed in parallel, and corresponding personalized speech synthesis instructions are generated. An emotionally personalized speech synthesis module is provided. Based on speech synthesis instructions, the module calls a target speech synthesis model corresponding to the confirmed identity. The target speech synthesis model takes the text content to be synthesized as input and is subject to dynamic emotional features to regulate the prosodic parameters of its synthesized speech, outputting synthesized speech with the target identity's timbre and corresponding emotional coloring.

[0062] The working principle of this invention is as follows: By acquiring the user's original speech signal and performing multi-resolution time-frequency analysis and nonlinear dynamics analysis, identity features reflecting the user's inherent acoustic structure and emotional features reflecting the current emotional state are obtained, respectively. First, an adaptive filtering strategy is used to suppress transient components related to emotional fluctuations, extracting robust identity features. Second, dynamic emotional features are calculated using nonlinear dynamics parameters. Next, these two types of features are fused to form a decoupled feature vector, and based on this, user identity verification and emotional state recognition are performed in parallel to generate personalized speech synthesis instructions. Finally, based on the verified identity and recognized emotional state, the corresponding speech synthesis model and prosody control parameters are invoked to output synthesized speech with the target identity timbre and corresponding emotional coloring. This technical solution achieves accurate capture of user identity and emotional state, thereby generating a more natural and personalized voice interaction experience.

[0063] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A voice synthesis method for AI companion robots based on voiceprint recognition, characterized in that, Includes the following steps: S1: Collect the user's original voice signal, perform synchronous parallel parsing processing on it, and obtain a first parsing result and a second parsing result; wherein, the first parsing result is the time-frequency distribution data obtained through multi-resolution time-frequency analysis, and the second parsing result is the system state parameter set obtained through nonlinear dynamic analysis; S2: Based on the first analysis result, an adaptive filtering strategy is applied to suppress transient components related to emotional fluctuations in the time-frequency distribution data, and robust identity features representing the user's inherent acoustic structure are extracted. S3: Based on the second analysis result, extract dynamic emotional features representing the user's current emotional state from the system state parameter set; S4: Fuse robust identity features with dynamic emotion features to form a decoupled feature vector; based on the decoupled feature vector, perform user identity verification and user emotion state recognition in parallel, and generate corresponding personalized speech synthesis instructions; S5: Based on the speech synthesis instruction, call the target speech synthesis model corresponding to the confirmed identity; the target speech synthesis model takes the text content to be synthesized as input and is controlled by dynamic emotional features to regulate the prosodic parameters of its synthesized speech, and outputs synthesized speech with the timbre of the target identity and containing the corresponding emotional color.

2. The AI ​​companion robot speech synthesis method based on voiceprint recognition according to claim 1, characterized in that, The process of obtaining the first parsing result is as follows: Complex-valued wavelet packet transform is used to decompose the user's original speech signal. A symmetric complex wavelet with linear phase is used as the mother wavelet to decompose the signal to a preset depth, forming a complete sub-band tree. Calculate the modulus square of the complex wavelet coefficients of each terminal node in the complete subband tree, obtain the energy distribution sequence of each subband on the time axis, and form the initial time-frequency distribution; The initial time-frequency distribution is weighted and corrected, and the weighting factor of the weighting correction has a nonlinear relationship with the center frequency of the sub-band, so as to simulate the sensitivity of the human ear to different frequencies. The weighted and corrected time-frequency distribution data is output as the first analytical result.

3. The AI ​​companion robot speech synthesis method based on voiceprint recognition according to claim 1, characterized in that, The process of obtaining the second parsing result is as follows: The phase space of the user's original speech signal is reconstructed, and the phase space trajectory of the speech dynamics system is reconstructed by the time delay method and the embedding dimension method. Based on the reconstructed phase space, the multi-scale permutation entropy characterizing the system complexity and the maximum Lyapunov exponent characterizing the chaotic properties of the system are calculated. The recurrence rate and determinism rate are quantized in the phase space trajectory recurrence graph. The recurrence rate is used to characterize the stability of the system state, and the determinism rate is used to distinguish between deterministic processes and random noise. The system state parameter set is composed of the multi-scale permutation entropy, the maximum Lyapunov exponent, the recursion rate, and the determinism rate, and is output as the second analytical result.

4. The AI ​​companion robot speech synthesis method based on voiceprint recognition according to claim 1, characterized in that, The extraction of robust identity features representing the user's inherent acoustic structure specifically includes: Based on the weighted and corrected time-frequency distribution data, the vowel core segments in the speech segment with concentrated energy and duration meeting the stable threshold are detected. Within the vowel core region, the trajectories of the first three formant frequencies are tracked, and the frequency variance and energy stability of each formant trajectory within the region are calculated. Using frequency variance and energy stability as stability indicators, an adaptive filter weight map corresponding to the time dimension of the time-frequency distribution data is generated. The adaptive filtering weight map is applied to the time-frequency distribution data to perform point-by-point weighted filtering, suppress transient components, and then the filtered data is subjected to dimensionality reduction and feature extraction to generate the robust identity features.

5. The AI ​​companion robot speech synthesis method based on voiceprint recognition according to claim 1, characterized in that, The extraction of dynamic emotional features representing the user's current emotional state specifically includes: Based on the system state parameter set, the entropy value of the multi-scale permutation entropy at the fine scale is calculated as an indicator of emotional disorder, while the ratio of recursion rate to determinism rate is used as an indicator of emotional stability. Within a preset sliding time window, the mean and variance of the maximum Lyapunov index are calculated over multiple consecutive time windows, and used as the baseline value of emotional intensity and the intensity value of emotional fluctuation, respectively. The emotional disorder index, emotional stability index, baseline value of emotional intensity, and value of emotional fluctuation intensity are normalized and vectorized and concatenated to form a comprehensive emotional description vector. The comprehensive emotion description vector is used as a dynamic emotion feature to represent the user's current emotional state.

6. The AI ​​companion robot speech synthesis method based on voiceprint recognition according to claim 1, characterized in that, The process of fusing robust identity features with dynamic emotion features to form a decoupled feature vector specifically includes: Robust identity features and dynamic emotion features are mapped to a unified feature space and then standardized based on Z-score to eliminate the influence of dimensionality. Calculate a real-time confidence score for both the robust identity feature and the dynamic emotion feature; assign fusion weights based on the proportion of the real-time confidence scores; The robust identity features and dynamic sentiment features with assigned weights are concatenated into vectors, and the high-dimensional vector after concatenation is reduced in dimensionality by principal component analysis to obtain the low-dimensional and compact decoupled feature vector.

7. The AI ​​companion robot speech synthesis method based on voiceprint recognition according to claim 1, characterized in that, The process of performing user identity verification and user emotional state recognition specifically includes: The decoupled feature vectors are simultaneously input into the pre-trained identity verification classifier and emotion state classifier to obtain the initial identity identifier and initial emotion category, respectively. A conflict arbitrator is established. When the confidence probability of the initial identity identifier is lower than a first threshold and the emotion intensity corresponding to the initial emotion category is higher than a second threshold, an identity-emotion recognition conflict is determined to have occurred. When a conflict occurs, the conflict arbitrator calls upon the historical identity records of the most recent successful interactions and uses the majority voting principle to overwrite and correct the current initial identity, generating the final identity confirmation result and the final emotion state recognition result. Personalized voice synthesis commands are formed based on the final identity verification result and the final emotion state recognition result.

8. The AI ​​companion robot speech synthesis method based on voiceprint recognition according to claim 1, characterized in that, The specific construction process of the speech synthesis model includes: Collect calm speech data of the target speaker, perform text transcription, audio quality checks and prosodic stability screening, and construct a standardized personalized speech dataset. Construct an end-to-end neural network speech synthesis model, which includes a text encoder, a speaker embedding vector extractor, a prosody controller, and an acoustic feature decoder. The speech synthesis model is pre-trained using a large-scale multi-speaker speech dataset to enable it to master general speech synthesis capabilities. Then, the speech synthesis model is fine-tuned using a standardized personalized speech dataset, with a focus on optimizing the speaker embedding vector extractor to accurately capture the timbre features of the target speaker. The fine-tuned speech synthesis model is tested using a validation set of speech that was not used in the training. When the acoustic feature similarity between the synthesized speech and the original speech of the target speaker reaches a predetermined threshold, the construction of the speech synthesis model is completed.

9. The AI ​​companion robot speech synthesis method based on voiceprint recognition according to claim 1, characterized in that, The output is a synthesized speech with the target identity timbre and corresponding emotional coloring, specifically including: Dynamic emotional features are input into a pre-trained emotion-prosodic mapping network, which outputs a set of specific, quantifiable target prosodic parameters, including fundamental frequency offset, speech rate scaling factor, and energy intensity factor. When encoding the text to be synthesized, the text encoder simultaneously predicts the emotion-neutral natural prosodic baseline parameters corresponding to the text content; The target prosodic parameters are weighted and fused with the natural prosodic baseline parameters; The acoustic feature decoder generates acoustic features containing the target's emotional coloring based on the output of the text encoder, the speaker embedding vector, and the fused prosodic parameters. Finally, the vocoder converts these features into synthesized speech. The training process of the emotion-prosody mapping network is as follows: Construct a database containing a large number of speech samples and their corresponding texts, speaker identities, and precise emotion tags; The emotion-prosodic mapping network and a prosodic parameter discriminator are initialized, the discriminator being used to distinguish between predicted prosodic parameters and prosodic parameters extracted from real speech; During training, the goal of the emotion-prosody mapping network is to generate prosodic parameters that are sufficient to deceive the discriminator and conform to the target emotion, while the goal of the discriminator is to accurately identify the authenticity of the parameters. Through repeated iterations, the emotion-prosody mapping network learns to generate prosodic parameters from the input emotion features.

10. A voice synthesis system for an AI companion robot based on voiceprint recognition, characterized in that, The AI ​​companion robot speech synthesis method based on voiceprint recognition as described in any one of claims 1-9 includes: The signal analysis and feature decoupling module is used to acquire the user's original speech signal, perform synchronous parallel analysis processing on it, and obtain a first analysis result and a second analysis result; wherein, the first analysis result is time-frequency distribution data obtained through multi-resolution time-frequency analysis, and the second analysis result is a set of system state parameters obtained through nonlinear dynamic analysis; A robust identity feature extraction module, based on the first parsing result, applies an adaptive filtering strategy to suppress transient components related to emotional fluctuations in the time-frequency distribution data, and extracts robust identity features that characterize the user's inherent acoustic structure. A dynamic emotion feature extraction module, which extracts dynamic emotion features representing the user's current emotional state from the system state parameter set based on the second parsing result; The fusion decision and instruction generation module fuses robust identity features and dynamic emotion features to form a decoupled feature vector; based on the decoupled feature vector, user identity verification and user emotion state recognition are performed in parallel, and corresponding personalized speech synthesis instructions are generated. An emotionally personalized speech synthesis module is provided. Based on speech synthesis instructions, the module calls a target speech synthesis model corresponding to the confirmed identity. The target speech synthesis model takes the text content to be synthesized as input and is subject to dynamic emotional features to regulate the prosodic parameters of its synthesized speech, outputting synthesized speech with the target identity's timbre and corresponding emotional coloring.

Images