Real-time voice synthesis and conversion method in intelligent voice call system

Abstract

The invention relates to the technical field of voice processing, in particular to a real-time voice synthesis and conversion method in an intelligent voice call system, which comprises the following steps of: acquiring voice data input by a user through a voice recognition module, preprocessing the voice data, performing semantic analysis through a bidirectional recurrent neural network and an attention mechanism, and obtaining a semantic analysis result; the method comprises the following steps: extracting a key semantic unit, selecting a proper voice template based on user historical interaction data and an adaptive template matching algorithm, performing voice synthesis by using a generative adversarial network, realizing cross-language or dialect real-time voice conversion through a multi-task learning algorithm, and combining phoneme-level optimization processing to ensure the naturalness and continuity of the voice, so as to improve the voice conversion efficiency. And through real-time analysis and a self-adaptive adjustment mechanism fed back by the user, the voice output quality is optimized, and the user experience is improved. According to the invention, efficient and natural speech synthesis and conversion services can be provided in various intelligent devices and application scenarios, and complex requirements in different language environments can be met.

Description

Technical Field

[0001] The present invention relates to the field of speech processing technology, and in particular to a real-time speech synthesis and conversion method in an intelligent voice call system. Background Art

[0002] With the rapid development of artificial intelligence technology, intelligent voice call systems have been widely used in various application scenarios, such as smart assistants, customer service, and telemedicine. These systems achieve natural language interaction with users through speech synthesis and conversion technology. However, with the diversification and complexity of user needs, higher requirements are placed on the naturalness of speech synthesis, the accuracy of emotional expression, and the real-time conversion capabilities of multiple languages ​​and dialects. In order to meet these requirements, modern voice call systems must have efficient semantic understanding capabilities, flexible speech generation strategies, and highly adaptive optimization mechanisms to improve user experience and the wide application of the system.

[0003] However, the existing technology still has significant deficiencies in speech synthesis and conversion. First, most traditional speech synthesis methods rely on preset templates or simple rules, which results in the generated speech being relatively stiff in emotional expression and semantic consistency, and lacking natural and fluent expression. In addition, in the process of multi-language and multi-dialect conversion, the existing technology is difficult to ensure the coherence and accuracy of speech signals, and the naturalness of cross-language speech synthesis is low. Secondly, the existing system has deficiencies in the dynamic adjustment and adaptive optimization of user feedback, and it is difficult to adjust the quality of speech output in real time according to the user's personalized needs, thus affecting the overall user experience. Summary of the invention

[0004] Based on the above objectives, the present invention provides a real-time speech synthesis and conversion method in an intelligent voice call system.

[0005] A real-time speech synthesis and conversion method in an intelligent voice call system comprises the following steps:

[0006] S1, voice data collection: the voice data input by the user is collected in real time through a preset voice recognition module, and preprocessed, the preprocessing includes noise elimination, voice signal enhancement and voice feature extraction to ensure the accuracy of subsequent processing;

[0007] S2, semantic analysis and understanding: the pre-processed speech data is input into the preset semantic analysis module, which performs semantic analysis and understanding on the speech content based on the natural language processing algorithm, extracts key semantic units, which include subject words, sentiment tendency and contextual information, and feeds the results back to the preset speech synthesis module;

[0008] S3, speech template selection: according to the output result of the semantic analysis module, the preset speech synthesis module selects the speech template that best matches the semantics from the pre-built multi-language and multi-dialect speech template library, wherein the speech template includes speech feature parameters, pitch, speech speed and speech style;

[0009] S4, speech synthesis and generation: the selected speech template is matched with the semantic unit, and the speech synthesis module generates a real-time speech signal according to the matching result, and performs refinement processing on the speech signal, including speech smoothing, pitch adjustment and speech speed adjustment;

[0010] S5, real-time speech conversion: after speech synthesis is completed, the generated speech signal is received and processed by the preset real-time speech conversion module, and the generated speech signal is converted into another language or dialect according to user settings or system requirements, while maintaining the emotional consistency and semantic accuracy of the speech;

[0011] S6, speech output and feedback: The final generated and converted speech signal is played to the user in real time through the preset speech output module. According to user feedback and interaction, the speech synthesis and conversion strategy is dynamically adjusted to continuously optimize the speech synthesis quality and user experience.

[0012] Optionally, the speech signal preprocessing in step S1 adopts a noise elimination algorithm based on a convolutional neural network to enhance the clarity of the speech signal, specifically including:

[0013] S11, voice signal framing: the collected voice signal is divided into multiple frames, each frame length is L, and the overlapping length between frames is M;

[0014] S12, feature extraction: perform short-time Fourier transform on each frame of speech signal to obtain frequency domain feature representation;

[0015] S13, noise estimation: The frequency domain features are processed using a convolutional neural network model to estimate the noise spectrum. The convolutional neural network model structure includes multiple convolutional layers and pooling layers. The convolution kernel size is K×K, where K is the convolution kernel size, and the activation function uses ReLU.

[0016] S14, signal enhancement: using the estimated noise spectrum N(d, t) to process the speech signal spectrum S(f, t), using the spectrum subtraction method to remove the noise, and obtaining an enhanced speech signal spectrum;

[0017] S15, signal reconstruction: perform inverse short-time Fourier transform on the enhanced speech signal spectrum and reconstruct it into a time domain speech signal.

[0018] Optionally, the semantic analysis module preset in the step S2 adopts an algorithm based on a bidirectional recurrent neural network and an attention mechanism to extract key semantic units in the speech data, specifically including:

[0019] S21, speech-to-text serialization: convert the preprocessed speech signal into a text sequence, vectorize the text sequence, and use word embedding technology to map the text sequence into a high-dimensional space to generate a word vector matrix X;

[0020] S22, semantic analysis model construction: construct a semantic analysis model based on a bidirectional recurrent neural network. The model includes an input layer, a bidirectional recurrent neural network layer, an attention mechanism layer, and a fully connected layer, wherein;

[0021] Input layer: receives word vector matrix X as input;

[0022] Bidirectional recurrent neural network layer: processes the input word embedding matrix and generates forward and backward hidden state sequences;

[0023] Attention mechanism layer: Apply the attention mechanism to the hidden state sequence generated by the bidirectional recurrent neural network layer to generate the context vector c t , to enhance the model’s attention to important semantic information;

[0024] Fully connected layer: The context vector c t Input to the fully connected layer and further converted into semantic unit representation;

[0025] S23, semantic unit extraction: extract key semantic units from the output of the fully connected layer, including topic words, sentiment tendency and contextual information, and feed the results back to the speech synthesis module for further processing.

[0026] Optionally, the voice template selection in step S3 is performed based on an adaptive template matching algorithm, and the adaptive template matching algorithm dynamically adjusts the voice template selection strategy according to the user's historical voice interaction data, specifically including:

[0027] S31, user history data analysis: extract the user's historical voice interaction data from the database, including voice content, voice style and emotional tendency, and generate a user feature vector U, expressed as: U = [u1, u2, ..., u m ], where u is the eigenvalue, u m represents the mth eigenvalue;

[0028] S32, template feature matching: using cosine similarity to calculate the similarity between the current speech semantic unit and each template in the speech template library;

[0029] S33, dynamically adjust the strategy: according to the similarity calculation result, dynamically adjust the template selection strategy, and give priority to the template with the highest matching degree with the user's historical interaction record.

[0030] Optionally, the speech synthesis and generation in step S4 adopts a speech synthesis algorithm based on a generative adversarial network to improve the naturalness and emotional expression effect of the generated speech, specifically including:

[0031] S41, Generator network design: Design the generator network structure, the input is a random noise vector z, and the output is a synthesized speech signal The generator network includes multiple fully connected layers and convolutional layers;

[0032] S42, discriminator network design: Design the discriminator network structure, with the input being the real speech signal S and the synthetic speech signal generated by the generator. The output is the discrimination result (real or synthetic), and the discriminator network includes multiple convolutional layers and fully connected layers;

[0033] S43, Generative Adversarial Training: The generator and discriminator optimize their respective parameters through adversarial training, so that the generator can generate more natural speech signals.

[0034] Optionally, the real-time speech conversion in step S5 adopts a multi-task learning algorithm to synchronously learn multiple language models when performing language or dialect conversion to improve the accuracy and naturalness of cross-language conversion, specifically including:

[0035] S51, Task definition: The speech conversion task is defined as multiple subtasks, each of which corresponds to the speech conversion of a target language or dialect. For each subtask, an independent language conversion model M is constructed. i , where i represents the i-th target language or dialect, and each model M i The goal is to convert an input source speech signal into an output speech signal in a target language or dialect;

[0036] S52, Model architecture design: Design a multi-task learning model architecture. The multi-task learning model includes a shared layer and a task-specific layer. The shared layer extracts common speech features, and the task-specific layer performs customized feature conversion and speech generation according to the requirements of each subtask.

[0037] S53, multi-task joint training: The multi-task learning model is jointly trained through a multi-task learning strategy, so that the shared layer learns the common features of each task, while the task-specific layers optimize their respective language conversion effects;

[0038] S54, language or dialect conversion: In actual use, according to user settings or system requirements, the corresponding task-specific layer is selected from the multi-task learning model to convert the source speech signal in real time to generate the speech signal of the target language or dialect, and ensure that the emotional expression and semantic accuracy are consistent;

[0039] S55, conversion result optimization: post-processing the converted speech signal, including speech smoothing, pitch adjustment, and speech rate control, to ensure that the generated speech signal has high-quality natural expression in the target language or dialect.

[0040] Optionally, the language or dialect conversion in step S54 further includes phoneme-level optimization processing, specifically including:

[0041] S541, Phoneme-level annotation: In the source speech signal The phoneme-level annotation is performed in the process of decomposing the speech signal into a phoneme sequence P, which can be expressed as: P = [p1, p2, ..., p m ], where p is the phoneme, p m Represents the mth phoneme. The labeling process uses a pre-trained phoneme recognition model to ensure the accuracy of the phoneme sequence;

[0042] S542, phoneme mapping rule establishment: according to the phoneme system of the target language or dialect, a phoneme mapping rule from the source language to the target language is established, the rule including direct mapping between phonemes and conversion processing of phoneme combinations to adapt to the pronunciation habits of the target language or dialect;

[0043] S543, phoneme conversion and adjustment: using the phoneme mapping rule, convert the phoneme sequence P of the source language into the phoneme sequence P' of the target language, expressed as: P' = [p1', p2', ..., p n ′];

[0044] S544, speech signal reconstruction: reconstructing a speech signal S′ of the target language or dialect according to the converted phoneme sequence P′;

[0045] S545, Phoneme-level optimization: The reconstructed speech signal is optimized at the phoneme level, including speech smoothing, pitch refinement, and pronunciation coherence enhancement, to ensure that the final speech output has natural speech fluency and clarity in the target language or dialect.

[0046] Optionally, in the speech output and feedback process in step S6, the output quality of speech synthesis is further optimized through real-time analysis of user feedback data and an adaptive adjustment mechanism, specifically including:

[0047] S61, user feedback data collection: collect user feedback data on voice output in real time through a preset user interaction module. The feedback data includes user voice comments, text feedback and interaction records, which serve as basic data for subsequent optimization of the system.

[0048] S62, output quality evaluation: Use preset evaluation indicators (such as speech clarity, naturalness, emotional expression, etc.) to evaluate the speech output in real time, and generate an evaluation result Q based on the collected feedback data, expressed as: Q = [q1, q2, ..., q n ], where q is the score of the evaluation index, q n Represents the score of the nth evaluation indicator;

[0049] S63, dynamic adjustment mechanism: adjust the speech synthesis parameter θ according to the evaluation result Q q ,Parameters include the pitch, speaking speed, voice style and emotional expression of speech synthesis;

[0050] S64, Adaptive Optimization: Combining historical feedback data and current evaluation results, the overall strategy of speech synthesis is continuously optimized through an adaptive learning mechanism;

[0051] S65, Feedback loop optimization: When the user starts a new round of interaction, the speech synthesis parameters will be further optimized based on the new feedback data. Through continuous feedback and adjustment cycles, the voice output quality will be continuously optimized to meet the dynamic needs of users.

[0052] Optionally, in the speech output and feedback process in step S6, the user's emotional feedback is analyzed in real time, and the emotional parameters of speech synthesis are dynamically adjusted according to the analysis results, specifically including:

[0053] S66, emotional feedback collection: collecting the user's emotional feedback data on the voice output in real time through a preset user interaction module, including user voice and text comments;

[0054] S67, sentiment analysis model: Build a model based on sentiment classification, perform sentiment analysis on user feedback data, and output sentiment label E, which is expressed as: E = [e1, e2, ..., e k ], where e is the emotion category, e k represents the kth sentiment category;

[0055] S68, emotional parameter adjustment: dynamically adjust the emotional parameter θ of speech synthesis according to the results of emotional analysis e ,Parameters include pitch, speaking speed and voice style to enhance the emotional expression of speech.

[0056] Beneficial effects of the present invention:

[0057] The present invention greatly improves the accuracy and naturalness of speech synthesis by combining a bidirectional recurrent neural network, an attention mechanism, and a generative adversarial network. In particular, through the optimization processing of the fully connected layer and the phoneme level, it ensures that the generated speech signal is not only semantically consistent but also emotionally rich, thereby making the speech output closer to the effect of natural human speech. The multi-task learning model and phoneme-level conversion further ensure high-quality conversion across languages ​​and dialects, and achieve fluent speech communication in a multilingual environment.

[0058] The present invention collects and analyzes user feedback data in real time, dynamically adjusts various parameters of speech synthesis and conversion, and adaptively optimizes the output effect. By introducing an adaptive learning mechanism, the tone, speech speed, and emotional expression of the speech can be adjusted according to the personalized needs and interaction scenarios of different users, thereby significantly improving the user experience. At the same time, the optimization mechanism of the feedback loop ensures that the system maintains efficient and high-quality speech output during long-term use, meeting the continuous needs of users.

[0059] The present invention is applicable to a variety of smart devices and application scenarios, including smart phones, smart speakers, car systems and robots. Through flexible template selection, adaptive adjustment and multi-task learning, the system can autonomously optimize speech synthesis and conversion strategies according to the characteristics and application environments of different devices, ensuring the provision of high-quality voice services under various complex conditions. This diverse applicability gives the present invention broad application prospects and market value. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0061] Figure 1 A schematic diagram of a method flow of an embodiment of the present invention;

[0062] Figure 2 Schematic diagram of the process flow of step S5 according to an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. At the same time, it is explained here that in order to make the embodiments more detailed, the following embodiments are the best and preferred embodiments, and those skilled in the art may also adopt other alternatives to implement some known technologies; and the accompanying drawings are only for more specific description of the embodiments, and are not intended to specifically limit the present invention.

[0064] It should be noted that the references to "one embodiment", "an embodiment", "an exemplary embodiment", "some embodiments" and the like in the specification indicate that the embodiments described may include specific features, structures or characteristics, but not every embodiment may include the specific features, structures or characteristics. In addition, when a specific feature, structure or characteristic is described in conjunction with an embodiment, it should be within the knowledge of a person skilled in the art to implement such feature, structure or characteristic in conjunction with other embodiments (whether or not explicitly described).

[0065] In general, a term can be understood, at least in part, from its use in context. For example, depending, at least in part, on the context, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in the singular sense, or can be used to describe a combination of features, structures, or characteristics in the plural sense. Additionally, the term "based on" can be understood as not necessarily intended to convey an exclusive set of factors, but can instead, depending, at least in part, on the context, allow for the presence of other factors that are not necessarily explicitly described.

[0066] like Figure 1-Figure 2 As shown, a real-time speech synthesis and conversion method in an intelligent voice call system comprises the following steps:

[0067] S1, voice data collection: The voice data input by the user is collected in real time through the preset voice recognition module, and preprocessed. The preprocessing includes noise elimination, voice signal enhancement and voice feature extraction to ensure the accuracy of subsequent processing;

[0068] S2, semantic analysis and understanding: the pre-processed speech data is input into the preset semantic analysis module, which performs semantic analysis and understanding on the speech content based on the natural language processing algorithm, extracts key semantic units, which include subject words, sentiment tendency and contextual information, and feeds the results back to the preset speech synthesis module;

[0069] S3, speech template selection: Based on the output of the semantic analysis module, the preset speech synthesis module selects the most semantically consistent speech template from the pre-built multi-language and multi-dialect speech template library. The speech template includes speech feature parameters, pitch, speaking speed, and speech style to ensure that the generated speech is consistent with the emotion and semantics of the original speech;

[0070] S4, speech synthesis and generation: Match the selected speech template with the semantic unit. The speech synthesis module generates a real-time speech signal based on the matching result and performs detailed processing on the speech signal, including speech smoothing, pitch adjustment, and speech speed adjustment. The generated speech signal must remain natural and fluent, and have a high ability to express emotions.

[0071] S5, real-time speech conversion: after speech synthesis is completed, the generated speech signal is received and processed by the preset real-time speech conversion module, and the generated speech signal is converted into another language or dialect according to user settings or system requirements, while maintaining the emotional consistency and semantic accuracy of the speech;

[0072] S6, speech output and feedback: The final generated and converted speech signal is played to the user in real time through the preset speech output module. According to user feedback and interaction, the speech synthesis and conversion strategy is dynamically adjusted to continuously optimize the speech synthesis quality and user experience.

[0073] The speech signal preprocessing in step S1 uses a noise elimination algorithm based on a convolutional neural network to enhance the clarity of the speech signal, including:

[0074] S11, voice signal framing: the collected voice signal is divided into multiple frames, each frame length is L, and the overlapping length between frames is M;

[0075] S12, feature extraction: perform short-time Fourier transform on each frame of speech signal to obtain frequency domain feature representation;

[0076] S13, noise estimation: The frequency domain features are processed using a convolutional neural network model to estimate the noise spectrum. The convolutional neural network model structure includes multiple convolutional layers and pooling layers. The convolution kernel size is K×K, where K is the convolution kernel size, and the activation function uses ReLU.

[0077] S14, signal enhancement: The estimated noise spectrum N(f, t) is used to process the speech signal spectrum S(f, t), and the spectrum subtraction method is used to remove the noise to obtain the enhanced speech signal spectrum, which is expressed as:

[0078]

[0079] in, is the enhanced speech signal spectrum, f is the frequency, t is the time;

[0080] S15, signal reconstruction: performing inverse short-time Fourier transform on the enhanced speech signal spectrum to reconstruct it into a time-domain speech signal;

[0081] By using the convolutional neural network model for noise estimation and spectrum subtraction methods, the background noise can be effectively removed, the clarity of the speech signal can be improved, and high-quality speech input can be provided for subsequent processing.

[0082] The semantic analysis module preset in step S2 uses an algorithm based on a bidirectional recurrent neural network and an attention mechanism to extract key semantic units from speech data, including:

[0083] S21, speech-to-text serialization: convert the preprocessed speech signal into a text sequence, and vectorize the text sequence. Use word embedding technology to map the text sequence into a high-dimensional space and generate a word vector matrix X, which is expressed as: X = [x1, x2, ... x n ], where x is a vector representation, x n is the vector representation of the nth word;

[0084] S22, semantic analysis model construction: construct a semantic analysis model based on a bidirectional recurrent neural network. The model includes an input layer, a bidirectional recurrent neural network layer, an attention mechanism layer, and a fully connected layer, wherein;

[0085] Input layer: receives word vector matrix X as input;

[0086] Bidirectional recurrent neural network layer: processes the input word vector matrix and generates forward and backward hidden state sequences, expressed as:

[0087]

[0088] Where f is the activation function, and are the forward and backward hidden states, W and U are weight matrices, and b is the bias term;

[0089] Attention mechanism layer: Apply the attention mechanism to the hidden state sequence generated by the bidirectional recurrent neural network layer to generate the context vector c t , to enhance the model's attention to important semantic information, expressed as:

[0090]

[0091]

[0092] Among them, h i is the hidden state, α ti is the attention weight, e ti is the calculation result of the scoring function;

[0093] Fully connected layer: The context vector c t The input is sent to the fully connected layer and further converted into a semantic unit representation. The output of the fully connected layer is represented as:

[0094] y=W c ·c t +b c ;

[0095] Among them, W c is the weight matrix of the fully connected layer, b c is the bias term, y is the output semantic unit representation;

[0096] S23, semantic unit extraction: extract key semantic units from the output of the fully connected layer, including topic words, sentiment tendency, and contextual information, and feed the results back to the speech synthesis module for further processing;

[0097] By building a semantic analysis model based on a bidirectional recurrent neural network, key semantic units can be extracted more accurately, improving the semantic understanding and expression capabilities of the speech synthesis system.

[0098] The voice template selection in step S3 is based on an adaptive template matching algorithm. The adaptive template matching algorithm dynamically adjusts the voice template selection strategy according to the user's historical voice interaction data, specifically including:

[0099] S31, user history data analysis: extract the user's historical voice interaction data from the database in the system, including voice content, voice style and emotional tendency, and generate a user feature vector U, which is expressed as: U = [u1, u2, ..., u m ], where u is the eigenvalue, u m represents the mth eigenvalue;

[0100] S32, template feature matching: the cosine similarity is used to calculate the similarity between the current speech semantic unit and each template in the speech template library. The cosine similarity is calculated as:

[0101]

[0102] Among them, T j is the speech template feature vector, U is the user feature vector;

[0103] S33, dynamically adjust the strategy: dynamically adjust the template selection strategy according to the similarity calculation result, and give priority to the template with the highest matching degree with the user's historical interaction record;

[0104] Through the adaptive template matching algorithm, it is possible to combine the user's historical data and dynamically adjust the voice template selection strategy to achieve more personalized speech synthesis and improve user satisfaction.

[0105] The speech synthesis and generation in step S4 adopts a speech synthesis algorithm based on a generative adversarial network to improve the naturalness and emotional expression of the generated speech, including:

[0106] S41, Generator network design: Design the generator network structure, the input is a random noise vector z, and the output is a synthesized speech signal The generator network includes multiple fully connected layers and convolutional layers;

[0107] S42, discriminator network design: Design the discriminator network structure, with the input being the real speech signal S and the synthetic speech signal generated by the generator. The output is the discrimination result (real or synthetic), and the discriminator network includes multiple convolutional layers and fully connected layers;

[0108] S43, Generative Adversarial Training: The generator and discriminator optimize their respective parameters through adversarial training, so that the generator generates more natural speech signals, expressed as:

[0109]

[0110] Among them, G is the generator, D is the discriminator, and p data (S) is the real data distribution, p z (z) is the noise distribution;

[0111] By introducing generative adversarial networks, the naturalness and emotional expression ability of speech synthesis can be effectively improved, making the generated speech more in line with user expectations.

[0112] The real-time speech conversion in step S5 uses a multi-task learning algorithm to simultaneously learn multiple language models when converting languages ​​or dialects to improve the accuracy and naturalness of cross-language conversion, including:

[0113] S51, Task definition: The speech conversion task is defined as multiple subtasks, each of which corresponds to the speech conversion of a target language or dialect. For each subtask, an independent language conversion model M is constructed. i , where i represents the i-th target language or dialect, and each model M i The goal is to convert an input source speech signal into an output speech signal in a target language or dialect;

[0114] S52, Model architecture design: Design a multi-task learning model architecture. The multi-task learning model includes a shared layer and a task-specific layer. The shared layer extracts common speech features, and the task-specific layer performs customized feature conversion and speech generation according to the requirements of each subtask.

[0115] Shared layer: Input is the generated source speech signal The general feature F of speech is extracted through a series of convolutional layers and recurrent layers, expressed as:

[0116] Among them, θ shared is the parameter of the shared layer, f shared is the mapping function of the shared layer, and F is the extracted general speech feature;

[0117] Task-specific layer: For each task i, using shared features F and task-specific parameters θ iGenerate a speech signal S in the target language or dialect i , expressed as: S i =f task (F;θ i );

[0118] Among them, θ i is the specific parameter of task i, f task is the mapping function of the task-specific layer;

[0119] S53, multi-task joint training: The multi-task learning model is jointly trained through a multi-task learning strategy, so that the shared layer learns the common features of each task, and the task-specific layers optimize their respective language conversion effects. The specific training goal is to minimize the weighted sum of the loss functions of each task, expressed as:

[0120]

[0121] Among them, J i is the loss function of the i-th task, λ i is the weight coefficient of the i-th task, and n is the total number of tasks;

[0122] S54, language or dialect conversion: In actual use, according to user settings or system requirements, the corresponding task-specific layer is selected from the multi-task learning model to convert the source speech signal in real time to generate the speech signal of the target language or dialect, and ensure that the emotional expression and semantic accuracy are consistent;

[0123] S55, conversion result optimization: post-processing the converted speech signal, including speech smoothing, pitch adjustment and speech rate control, to ensure that the generated speech signal has high-quality natural expression in the target language or dialect;

[0124] By building a multi-task learning model, efficient and accurate speech conversion between different languages ​​or dialects is achieved. The multi-task learning strategy not only improves the accuracy and naturalness of speech conversion, but also enables the model to adapt to real-time conversion needs in multiple language environments.

[0125] The language or dialect conversion in step S54 further includes phoneme-level optimization processing, specifically including:

[0126] S541, Phoneme-level annotation: In the source speech signal The phoneme-level annotation is performed in the process of decomposing the speech signal into a phoneme sequence P, which can be expressed as: P = [p1, p2, ..., p m ], where p is the phoneme, p m Represents the mth phoneme. The labeling process uses a pre-trained phoneme recognition model to ensure the accuracy of the phoneme sequence;

[0127] S542, phoneme mapping rule establishment: according to the phoneme system of the target language or dialect, a phoneme mapping rule from the source language to the target language is established, the rule including direct mapping between phonemes and conversion processing of phoneme combinations to adapt to the pronunciation habits of the target language or dialect;

[0128] S543, phoneme conversion and adjustment: using the phoneme mapping rule, convert the phoneme sequence P of the source language into the phoneme sequence P' of the target language, expressed as: P' = [p1', p2', ..., p n ′], during the conversion process, necessary phoneme adjustments are made according to the pronunciation habits and speech coherence of different languages ​​to maintain the natural fluency of speech;

[0129] S544, speech signal reconstruction: reconstructing the speech signal S′ of the target language or dialect according to the converted phoneme sequence P′. During the reconstruction process, speech synthesis is performed in combination with speech feature parameters (such as pitch, speech speed, etc.) of the target language or dialect to ensure that the generated speech signal has high-quality natural expression;

[0130] S545, Phoneme-level optimization: Perform phoneme-level optimization processing on the reconstructed speech signal, including speech smoothing, pitch refinement, and pronunciation coherence enhancement, to ensure that the final speech output has natural speech fluency and clarity in the target language or dialect;

[0131] Through the fine conversion and optimization of phonemes, the generated speech signal is ensured to have high-quality natural expression and speech coherence in the target language or dialect, thereby improving the overall quality of speech conversion.

[0132] In the speech output and feedback process in step S6, the output quality of speech synthesis is further optimized through real-time analysis of user feedback data and an adaptive adjustment mechanism, specifically including:

[0133] S61, user feedback data collection: collect user feedback data on voice output in real time through a preset user interaction module. The feedback data includes user voice comments, text feedback and interaction records, which serve as basic data for subsequent optimization of the system.

[0134] S62, output quality evaluation: Use preset evaluation indicators (such as speech clarity, naturalness, emotional expression, etc.) to evaluate the speech output in real time, and generate an evaluation result Q based on the collected feedback data, expressed as: Q = [q1, q2, ..., q n ], where q is the score of the evaluation index, q n Represents the score of the nth evaluation indicator;

[0135] S63, dynamic adjustment mechanism: adjust the speech synthesis parameter θ according to the evaluation result Qq ,The parameters include the pitch of speech synthesis, speaking speed, voice style and emotional expression, and the goal of adjustment is to optimize the ,quality of speech output based on user feedback and actual needs;

[0136] S64, Adaptive Optimization: Combining historical feedback data and current evaluation results, the overall strategy of speech synthesis is continuously optimized through an adaptive learning mechanism. The system can adaptively adjust parameters in different environments and user preferences to ensure high quality of speech output and user satisfaction.

[0137] S65, Feedback loop optimization: When the user conducts a new round of interaction, the speech synthesis parameters will be further optimized based on the new feedback data. Through continuous feedback and adjustment cycles, the speech output quality is continuously optimized to meet the dynamic needs of users.

[0138] By introducing real-time analysis and dynamic adjustment mechanisms for user feedback data, the output quality of speech synthesis is further optimized. Through adaptive optimization and loop feedback, the system can continuously improve the clarity, naturalness and user satisfaction of speech output.

[0139] In the speech output and feedback process in step S6, the user's emotional feedback is analyzed in real time, and the emotional parameters of speech synthesis are dynamically adjusted according to the analysis results, specifically including:

[0140] S66, emotional feedback collection: collecting the user's emotional feedback data on the voice output in real time through a preset user interaction module, including user voice and text comments;

[0141] S67, sentiment analysis model: Build a model based on sentiment classification, perform sentiment analysis on user feedback data, and output sentiment label E, which is expressed as: E = [e1, e2, ..., e k ], where e is the emotion category, e k represents the kth sentiment category;

[0142] S68, emotional parameter adjustment: dynamically adjust the emotional parameter θ of speech synthesis according to the results of emotional analysis e ,Parameters include pitch, speaking speed and voice style to enhance the emotional expression of speech;

[0143] By introducing emotional feedback analysis and dynamic adjustment mechanisms, the emotional parameters of speech synthesis can be optimized in real time to improve user experience satisfaction.

[0144] The present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention. In order to make the public have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, but those skilled in the art can fully understand the present invention without the description of these details. In addition, in order to avoid unnecessary confusion about the essence of the present invention, well-known methods, processes, procedures, components and circuits are not described in detail.

[0145] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A real-time speech synthesis and conversion method in an intelligent voice call system, characterized in that: The following steps are involved: S1, voice data collection: collect the voice data input by the user in real time through the preset voice recognition module, and perform preprocessing, the preprocessing includes noise elimination, voice signal enhancement and voice feature extraction; S2, semantic analysis and understanding: the pre-processed speech data is input into the preset semantic analysis module, which performs semantic analysis and understanding on the speech content based on the natural language processing algorithm, extracts key semantic units, which include subject words, sentiment tendency and contextual information, and feeds the results back to the preset speech synthesis module; S3, speech template selection: according to the output result of the semantic analysis module, the preset speech synthesis module selects the speech template that best matches the semantics from the pre-built multi-language and multi-dialect speech template library, wherein the speech template includes speech feature parameters, pitch, speech speed and speech style; S4, speech synthesis and generation: the selected speech template is matched with the semantic unit, and the speech synthesis module generates a real-time speech signal according to the matching result, and performs refinement processing on the speech signal, including speech smoothing, pitch adjustment and speech speed adjustment; S5, real-time speech conversion: after speech synthesis is completed, the generated speech signal is received and processed by the preset real-time speech conversion module, and the generated speech signal is converted into another language or dialect according to user settings or system requirements, while maintaining the emotional consistency and semantic accuracy of the speech; S6, speech output and feedback: The final generated and converted speech signal is played to the user in real time through the preset speech output module. According to user feedback and interaction, the speech synthesis and conversion strategy is dynamically adjusted to continuously optimize the speech synthesis quality and user experience.

2. The real-time speech synthesis and conversion method in an intelligent voice call system according to claim 1, characterized in that: The speech signal preprocessing in step S1 adopts a noise elimination algorithm based on a convolutional neural network to enhance the clarity of the speech signal, specifically including: S11, voice signal framing: dividing the collected voice signal into multiple frames; S12, feature extraction: perform short-time Fourier transform on each frame of speech signal to obtain frequency domain feature representation; S13, noise estimation: The frequency domain features are processed using a convolutional neural network model to estimate the noise spectrum. The convolutional neural network model structure includes multiple convolutional layers and pooling layers. The convolution kernel size is K×K, where K is the convolution kernel size, and the activation function uses ReLU. S14, signal enhancement: using the estimated noise spectrum to process the speech signal spectrum, using a spectrum subtraction method to remove noise, and obtaining an enhanced speech signal spectrum; S15, signal reconstruction: perform inverse short-time Fourier transform on the enhanced speech signal spectrum and reconstruct it into a time domain speech signal.

3. The real-time speech synthesis and conversion method in an intelligent voice call system according to claim 2, characterized in that: The semantic analysis module preset in the step S2 uses an algorithm based on a bidirectional recurrent neural network and an attention mechanism to extract key semantic units in the speech data, specifically including: S21, speech-to-text serialization: convert the preprocessed speech signal into a text sequence, vectorize the text sequence, and use word embedding technology to map the text sequence into a high-dimensional space to generate a word vector matrix X; S22, semantic analysis model construction: construct a semantic analysis model based on a bidirectional recurrent neural network. The model includes an input layer, a bidirectional recurrent neural network layer, an attention mechanism layer, and a fully connected layer, wherein; Input layer: receives word vector matrix X as input; Bidirectional recurrent neural network layer: processes the input word embedding matrix and generates forward and backward hidden state sequences; Attention mechanism layer: Apply the attention mechanism to the hidden state sequence generated by the bidirectional recurrent neural network layer to generate the context vector c t , to enhance the model’s attention to important semantic information; Fully connected layer: The context vector c t Input to the fully connected layer and further converted into semantic unit representation; S23, semantic unit extraction: extract key semantic units from the output of the fully connected layer, including topic words, sentiment tendency and contextual information, and feed the results back to the speech synthesis module for further processing.

4. The real-time speech synthesis and conversion method in an intelligent voice call system according to claim 3, characterized in that: The voice template selection in step S3 is performed based on an adaptive template matching algorithm, and the adaptive template matching algorithm dynamically adjusts the voice template selection strategy according to the user's historical voice interaction data, specifically including: S31, user history data analysis: extract the user's historical voice interaction data from the database, including voice content, voice style and emotional tendency, and generate a user feature vector U, expressed as: U = [u1, u2, ..., u m ], where u is the eigenvalue, u m represents the mth eigenvalue; S32, template feature matching: using cosine similarity to calculate the similarity between the current speech semantic unit and each template in the speech template library; S33, dynamically adjust the strategy: according to the similarity calculation result, dynamically adjust the template selection strategy, and give priority to the template with the highest matching degree with the user's historical interaction record.

5. The real-time speech synthesis and conversion method in an intelligent voice call system according to claim 4, characterized in that: The speech synthesis and generation in step S4 adopts a speech synthesis algorithm based on a generative adversarial network to improve the naturalness and emotional expression effect of the generated speech, specifically including: S41, Generator network design: Design the generator network structure, the input is a random noise vector z, and the output is a synthesized speech signal The generator network includes multiple fully connected layers and convolutional layers; S42, discriminator network design: Design the discriminator network structure, with the input being the real speech signal S and the synthetic speech signal generated by the generator. The output is the discrimination result. The discriminator network includes multiple convolutional layers and fully connected layers. S43, Generative Adversarial Training: The generator and discriminator optimize their respective parameters through adversarial training, so that the generator can generate natural speech signals.

6. The real-time speech synthesis and conversion method in an intelligent voice call system according to claim 5, characterized in that: The real-time speech conversion in step S5 adopts a multi-task learning algorithm to synchronously learn multiple language models when performing language or dialect conversion to improve the accuracy and naturalness of cross-language conversion, specifically including: S51, Task definition: The speech conversion task is defined as multiple subtasks, each of which corresponds to the speech conversion of a target language or dialect. For each subtask, an independent language conversion model M is constructed. i , where i represents the i-th target language or dialect, and each model M i The goal is to convert an input source speech signal into an output speech signal in a target language or dialect; S52, Model architecture design: Design a multi-task learning model architecture. The multi-task learning model includes a shared layer and a task-specific layer. The shared layer extracts common speech features, and the task-specific layer performs customized feature conversion and speech generation according to the requirements of each subtask. S53, multi-task joint training: The multi-task learning model is jointly trained through a multi-task learning strategy, so that the shared layer learns the common features of each task, while the task-specific layers optimize their respective language conversion effects; S54, language or dialect conversion: In actual use, according to user settings or system requirements, a corresponding task-specific layer is selected from the multi-task learning model to perform real-time conversion on the source speech signal to generate a speech signal of the target language or dialect; S55, conversion result optimization: post-processing the converted speech signal, including speech smoothing, pitch adjustment and speech speed control.

7. The real-time speech synthesis and conversion method in an intelligent voice call system according to claim 6, characterized in that: The language or dialect conversion in step S54 further includes phoneme-level optimization processing, specifically including: S541, Phoneme-level annotation: In the source speech signal The phoneme-level annotation is performed in the process of decomposing the speech signal into a phoneme sequence P, which can be expressed as: P = [p1, p2, ..., p m ], where p is the phoneme, p m represents the mth phoneme; S542, phoneme mapping rule establishment: according to the phoneme system of the target language or dialect, a phoneme mapping rule from the source language to the target language is established, the rule including direct mapping between phonemes and conversion processing of phoneme combinations to adapt to the pronunciation habits of the target language or dialect; S543, phoneme conversion and adjustment: using the phoneme mapping rule, convert the phoneme sequence P of the source language into the phoneme sequence P' of the target language, expressed as: P' = [p1', p2', ..., p n ′]; S544, speech signal reconstruction: reconstructing a speech signal S′ of the target language or dialect according to the converted phoneme sequence P′; S545, phoneme-level optimization: performing phoneme-level optimization processing on the reconstructed speech signal, including speech smoothing, pitch refinement, and pronunciation coherence enhancement.

8. The real-time speech synthesis and conversion method in an intelligent voice call system according to claim 7, characterized in that: In the speech output and feedback process in step S6, the output quality of speech synthesis is further optimized through real-time analysis of user feedback data and an adaptive adjustment mechanism, specifically including: S61, user feedback data collection: collecting user feedback data on voice output in real time through a preset user interaction module, the feedback data including user voice comments, text feedback and interaction records; S62, output quality evaluation: Use the preset evaluation index to evaluate the speech output in real time, and generate the evaluation result Q according to the collected feedback data, which is expressed as: Q = [q1, q2, ..., q n ], where q is the score of the evaluation index, q n Represents the score of the nth evaluation indicator; S63, dynamic adjustment mechanism: adjust the speech synthesis parameter θ according to the evaluation result Q q ,Parameters include the pitch, speaking speed, voice style and emotional expression of speech synthesis; S64, Adaptive Optimization: Combining historical feedback data and current evaluation results, the overall strategy of speech synthesis is continuously optimized through an adaptive learning mechanism; S65, Feedback loop optimization: When the user starts a new round of interaction, the speech synthesis parameters will be further optimized based on the new feedback data. Through continuous feedback and adjustment cycles, the voice output quality will be continuously optimized to meet the dynamic needs of users.

9. The real-time speech synthesis and conversion method in an intelligent voice call system according to claim 8, characterized in that: In the speech output and feedback process in step S6, the emotional feedback of the user is analyzed in real time, and the emotional parameters of speech synthesis are dynamically adjusted according to the analysis results, which specifically includes: S66, emotional feedback collection: collecting the user's emotional feedback data on the voice output in real time through a preset user interaction module, including user voice and text comments; S67, sentiment analysis model: Build a model based on sentiment classification, perform sentiment analysis on user feedback data, and output sentiment label E, which is expressed as: E = [e1, e2, ..., e k ], where e is the emotion category, e k represents the kth sentiment category; S68, emotional parameter adjustment: dynamically adjust the emotional parameter θ of speech synthesis according to the results of emotional analysis e , parameters include pitch, speaking speed, and voice style.

Images