Method, system, medium, and product for interactive speech synthesis based on affect regulation

Abstract

The application discloses an interactive speech synthesis method and system based on emotion regulation, a medium and a product, and relates to the field of speech synthesis, which comprises the following steps: selecting a speaker role; acquiring the audio of the speaker role in real time; performing speech-to-text transcription on the audio of the speaker role to obtain a first text; performing audio emotion and intention recognition on the audio of the speaker role to obtain a second text; splicing the first text and the second text to obtain a dialogue text; performing emotion regulation on the dialogue text to obtain an emotion-regulated dialogue text; and outputting target speech according to the emotion-regulated dialogue text and the speaker role. The application can capture the emotional information input by a user in real time, dynamically adjust the dialogue text in real time according to the emotional information input by the user, so that the target speech output is always matched with the current emotional state, and the naturalness and real-time performance of human-computer interaction are greatly improved.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, and in particular to an interactive speech synthesis method, system, medium, and product based on emotion modulation. Background Technology

[0002] Currently, text-to-speech (TPS) technology is widely used, and its core process typically includes text preprocessing, phoneme conversion, prosody annotation, feature extraction, and vocoder generation. However, most current methods lack real-time emotion recognition and modulation mechanisms, resulting in relatively uniform speech in terms of emotion, style, and personalized expression. Furthermore, traditional methods involve complex preprocessing and are data-dependent; the phoneme conversion and prosody annotation processes not only increase algorithm complexity but also heavily rely on large-scale, high-quality labeled data, making them prone to introducing errors. Summary of the Invention

[0003] To address the technical problems existing in the background art, this invention proposes an interactive speech synthesis method, system, medium, and product based on emotion modulation.

[0004] In a first aspect, the present invention proposes an interactive speech synthesis method based on emotion modulation, comprising:

[0005] Real-time acquisition of speaker's audio;

[0006] The speaker's audio is transcribed into text to obtain the first text.

[0007] Audio sentiment and intent recognition is performed on the speaker's audio to obtain the second text;

[0008] By concatenating the first and second texts, the dialogue text is obtained.

[0009] Emotional modulation is applied to the dialogue text to obtain emotionally modulated dialogue text;

[0010] Choose the speaker role;

[0011] Based on the emotional modulation dialogue text and the speaker's role, the target speech is output.

[0012] Preferably, the dialogue text is subjected to sentiment modulation to obtain sentiment-modified dialogue text, specifically including:

[0013] Using a pre-defined large language model and combining historical dialogue records, sentiment analysis and text generation processing are performed on the dialogue text to obtain sentiment-modulated dialogue text.

[0014] The expression for the sentiment-modified dialogue text is t7 = LLM(History(t6)); where t6 represents the dialogue text, LLM is the large language model, History is the operation of wrapping the dialogue text t6 into the historical dialogue record, and t7 is the sentiment-modified dialogue text.

[0015] Preferably, before selecting the speaker role, the following steps are also included:

[0016] Add a speaker role;

[0017] Get the speaker's video for the speaker role;

[0018] Feature extraction is performed on the speaker's video to obtain the speaker's role features;

[0019] The speaker's role characteristics are combined to form the speaker's overall characteristics.

[0020] Preferably, after combining the speaker role characteristics into the overall speaker characteristics, the method further includes:

[0021] Save the speaker's role and the corresponding overall speaker characteristics.

[0022] Preferably, the speaker role characteristics include: speaker ID, speaker facial feature vector, speaker age information, speaker video material audio data content, character style description, and speaker video material text content.

[0023] Preferably, feature extraction is performed on the speaker's video to obtain speaker role features, specifically including:

[0024] The speaker videos are automatically sorted according to the order in which they were recorded, and the order of the speaker videos is used as the speaker ID.

[0025] The speaker's facial features are extracted from the video using a pre-defined facial recognition model to obtain the speaker's facial feature vector.

[0026] The speaker's age information is obtained by using a pre-set age prediction model to predict the speaker's age in the video.

[0027] Audio-video separation tools are used to separate the audio from the speaker's video to obtain the audio data content of the video material;

[0028] Using a pre-defined large-scale image-text dialogue model, the speaker's style is extracted from the video to obtain a style description.

[0029] The speech-to-text transcription of the speaker's video is performed using a pre-defined text transcription model to obtain the text content of the speaker's video material.

[0030] Preferably, the expression of the dialogue text is:

[0031] t6 = Concat(ASR(input) audio ),ALM(input audio ));

[0032] In the formula, ASR is the text transcription model, ALM is the large-scale audio-text dialogue model, and input... audio t6 represents the speaker's audio, t6 represents the dialogue text, and Concat indicates concatenation.

[0033] Preferably, the target speech is output based on the emotion-modulated dialogue text and the speaker's role, specifically including:

[0034] Based on the speaker's role, the speaker's overall characteristics are obtained;

[0035] The speaker's overall characteristics, speaker ID t1, and the emotion modulation dialogue text are combined to form input features;

[0036] Input features are fed into a pre-defined VoiceChat model to obtain the target speech;

[0037] Stream the target speech.

[0038] Secondly, this invention also proposes an interactive speech synthesis system based on emotion modulation, comprising:

[0039] The speaker role module is used to select the speaker role;

[0040] The dialogue text acquisition module is used to acquire speaker audio in real time; perform speech-to-text transcription on the speaker audio to obtain the first text; perform audio emotion and intent recognition on the speaker audio to obtain the second text; and concatenate the first text and the second text to obtain the dialogue text.

[0041] The interactive emotion control module is used to perform emotion control on the dialogue text to obtain the emotion-controlled dialogue text.

[0042] The speech synthesis module is used to output the target speech based on the emotional adjustment of the dialogue text and the speaker's role.

[0043] Thirdly, the present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the emotion-modulation-based interactive speech synthesis method as described in any one of the first aspects.

[0044] Fourthly, the present invention also proposes a computer program product, comprising a computer program that, when executed by a processor, implements the steps of the emotion-modulation-based interactive speech synthesis method as described in any one of the first aspects.

[0045] The proposed interactive speech synthesis method, system, medium, and product based on emotion modulation in this invention can capture the emotional information input by the user in real time by first selecting the speaker role and then acquiring the speaker role's audio in real time. This allows for real-time dynamic emotional adjustment of the dialogue text based on the user's emotional input, thereby adjusting parameters such as volume, speech rate, and tone of the target speech in the speech generation process in real time. This ensures that the output target speech always matches the current emotional state, significantly improving the naturalness and real-time performance of human-computer interaction. It effectively makes up for the shortcomings of existing technologies in terms of emotional expression and personalization, and can achieve multi-role, multi-emotion interactive speech generation capabilities without the need for model fine-tuning for specific speakers. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating an interactive speech synthesis method based on emotion modulation in one embodiment of the present invention.

[0047] Figure 2 This is a schematic diagram illustrating the extraction of overall speaker features in one embodiment of the present invention.

[0048] Figure 3 This is a schematic diagram of emotion regulation of dialogue text in one embodiment of the present invention.

[0049] Figure 4 This is a schematic diagram of the large language model structure in one embodiment of the present invention. Detailed Implementation

[0050] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0051] Reference Figure 1 The present invention proposes an interactive speech synthesis method based on emotion modulation, comprising:

[0052] Choose the speaker role;

[0053] Get the dialogue text;

[0054] Emotional modulation is applied to the dialogue text to obtain emotionally modulated dialogue text;

[0055] Based on the emotional modulation dialogue text and the speaker's role, the target speech is output.

[0056] This invention first selects the speaker role, then obtains the dialogue text, and performs emotion modulation on the dialogue text to adjust the volume, speech rate, tone, and other characteristics of the output target speech. This effectively makes up for the shortcomings of existing technologies in terms of emotional expression and personalization. Moreover, it can achieve interactive speech generation capabilities with multiple roles and emotions without the need for model fine-tuning for specific speakers.

[0057] In this embodiment, the obtained dialogue text t6 includes two parts: the first part is the first text generated by recording with a microphone and then using a text transcription model to analyze the speaker's audio; the second part is a descriptive text obtained by using a large audio-text dialogue model to identify the speaker's audio emotional intent, i.e., the second text; and finally, the first text and the second text are concatenated as the input dialogue text t6.

[0058] The expression for the dialogue text t6 is t6 = Concat(ASR(input)). audio ),ALM(input audio In the formula, ASR is the text transcription model, ALM is the large-scale audio-text dialogue model, and input... audio t6 is the audio of the speaker; t6 is the final dialogue text; Concat indicates concatenation.

[0059] Therefore, in this embodiment, obtaining the dialogue text specifically includes:

[0060] Real-time acquisition of speaker's audio;

[0061] The speaker's audio is transcribed into text to obtain the first text.

[0062] Audio sentiment and intent recognition is performed on the speaker's audio to obtain the second text;

[0063] By concatenating the first and second texts, the dialogue text is obtained.

[0064] This embodiment can capture the emotional information input by the user in real time by acquiring the speaker's audio. Based on the emotional information input by the user, the dialogue text can be dynamically adjusted in real time. This allows for real-time adjustment of parameters such as volume, speed, and tone of the target speech during the speech generation process, ensuring that the output target speech always matches the current emotional state. This significantly improves the naturalness and real-time performance of human-computer interaction.

[0065] Specifically, this embodiment uses a microphone to record audio of the speaker in real time.

[0066] In this embodiment, sentiment modulation is performed on the dialogue text to obtain sentiment-modified dialogue text t7, specifically including:

[0067] Using a pre-defined Large Language Model (LLM) and historical dialogue records, sentiment analysis and text generation are performed on t6 to obtain sentiment-modified dialogue text t7, thus embedding sentiment-modified dialogue text t7 with... <break> 、 <long-break> 、 <breath> 、 <laugh>Special characters, such as voice parameters (volume, speech rate, and tone), are controlled in the emotionally modulated dialogue text t7.

[0068] The expression for the sentiment-modified dialogue text t7 is t7 = LLM(History(t6)); where LLM is the large language model, History is the operation of wrapping the dialogue text t6 into the historical dialogue record, and t7 is the sentiment-modified dialogue text after sentiment modulation.

[0069] The emotion-modulation dialogue text generated in this embodiment integrates the current dialogue text and the historically processed dialogue text, which can provide more accurate dialogue text that is more in line with the current emotional state.

[0070] like Figure 4 As shown, the large language model in this embodiment is based on the Transformer architecture and adopts a decoder-only structure, achieving sequence modeling through stacked self-attention layers and feedforward networks. This embodiment also introduces Group Query Attention (GQA) to optimize the efficiency of traditional multi-head attention through group computation. Furthermore, the positional encoding method used in this embodiment employs YaRN, a variant of traditional rotated positional encoding.

[0071] The model in this embodiment supports a context length of 128K tokens and can generate coherent text of 8K tokens, making it suitable for long text tasks.

[0072] In one specific embodiment, prior to selecting the speaker role, the following is also included:

[0073] Add a speaker role;

[0074] Get the speaker's video for the speaker role;

[0075] Feature extraction is performed on the speaker's video to obtain the speaker's role features;

[0076] The speaker's role characteristics are combined to form the speaker's overall characteristics.

[0077] Specifically, firstly, speaker roles are added by recording speaker video footage through a camera; secondly, speaker role feature information is extracted, including speaker ID, speaker facial features, speaker age information, speaker style description, text content of speaker video footage, and audio data content of speaker video footage. The above feature information is encoded as the overall features of the speaker, so that it can be used as a speaker role in subsequent emotion-modulated dialogue text, thereby realizing personalized speech synthesis function.

[0078] In this embodiment, acquiring the speaker's video specifically includes:

[0079] The speaker's video is obtained by using a camera to capture real-time video footage of the speaker.

[0080] In this embodiment, the speaker's role characteristics include: speaker ID t1, speaker's facial feature vector f0, speaker's age information t2, speaker's audio data content from video material t3, character style description t4, and speaker's text content from video material t5.

[0081] This embodiment is configured in such a way that it integrates speaker images, text descriptions, and voice data, thereby achieving the extraction and fusion of multi-dimensional features. This results in the generated voice being richer and more natural in terms of emotional expression and personalized features, meeting diverse application needs.

[0082] It should be understood that in this embodiment, the speaker ID t1 is automatically sorted according to the input order, and the value type is an integer. The expression for t1 is t1 = n (n = 1, 2, 3, ...); where n is a positive integer.

[0083] In this embodiment, the age information t2 is obtained from a preset age prediction model, and the result is an integer value. The formula for calculating t2 is t2 = AgePredModel(image); where AgePredModel is the age prediction model, image is a frame containing the speaker, and t2 is the predicted age information.

[0084] In this embodiment, the audio data content t3 of the speaker's video material is obtained by separating the speaker's video and saving the audio data in tensor format. The formula for calculating t3 is t3 = ToTensor(Tools(video)); where Tools is an audio-video separation tool used to obtain pure audio information; and ToTensor is an audio-tensor conversion function.

[0085] In this embodiment, the character style description t4 is obtained from a preset large-scale image-text dialogue model. By setting prompt word templates, the model describes the speaker's style in the image, and the result is a string containing keywords for style description. The formula for calculating t4 is t4 = VLM(image); where VLM is the large-scale image-text dialogue model, image is a frame containing the speaker, and t4 is the keyword for style description.

[0086] In this embodiment, the text content t5 of the speaker's video footage is obtained from a preset text transcription model, and the result is a string containing the speaker's speech content in the audio. The formula for calculating the text content t5 of the speaker's video footage is t5 = ASR(audio); where ASR is the text transcription model; audio is the audio containing the speaker's clear voice, extracted from the video footage; and t5 is the speaker's speech content in the audio.

[0087] Therefore, in this embodiment, feature extraction is performed on the speaker's video to obtain speaker role features, specifically including:

[0088] The speaker videos are automatically sorted according to the order in which they were recorded, and the order of the speaker videos is used as the speaker ID t1;

[0089] The speaker's facial features are extracted from the video using a pre-defined facial recognition model to obtain the speaker's facial feature vector f0.

[0090] The speaker's age information t2 is obtained by using a pre-set age prediction model to predict the speaker's age in the video.

[0091] Audio and video separation tools are used to separate the audio from the speaker's video to obtain the audio data content t3 of the video material;

[0092] Using a pre-defined large-scale image-text dialogue model, the speaker's style is extracted from the video, resulting in a style description t4.

[0093] Using a pre-defined text transcription model, the speaker's video is transcribed into text, resulting in the text content t5 of the speaker's video material.

[0094] It should be noted that all models in this embodiment have been trained.

[0095] In this embodiment, the speaker's overall feature F is a combination of the above features, separated by special characters, and is expressed as: F = Concat(<speaker t1> , <face>,f0,<\face>, <age>,t2,<\age>, <audio>,t3,<\audio>,<style>,t4,<\style>,<text>,t5,<\text>,<\speaker t1>); where Concat is the concatenation operation.

[0096] like Figure 2 As shown, in one specific embodiment, a speaker's video material is acquired; features are extracted from the speaker's video material to obtain speaker role features; wherein, the speaker role features include speaker ID (001), speaker facial features, speaker age information (23 years old), speaker style description, text content of speaker video material, and audio data content of speaker video material, and the above feature information is encoded as the overall features of the speaker.

[0097] The speaker's style description depicts a chibi-style girl with an exaggerated head-to-body ratio. Her curly hair, sneakers, and shopping bag details blend Japanese Harajuku style with street fashion, showcasing youthful vitality while subtly incorporating consumer culture symbols. Warm light and shadow and dynamic postures convey a sense of healing.

[0098] In a further embodiment, after combining the speaker role features as the overall speaker features, the method further includes:

[0099] Save the speaker's role and the corresponding overall speaker characteristics.

[0100] Specifically, in this embodiment, the speaker's overall characteristics F are saved on the local disk in pickle file format.

[0101] This embodiment is configured in such a way that when using an added speaker role in the future, the relevant feature information of the speaker role can be directly read and loaded from the local storage file.

[0102] Therefore, in another specific embodiment, selecting a speaker role specifically includes: selecting a speaker role from the saved speaker roles based on the dialogue text.

[0103] In this embodiment, the target speech is output based on the emotion-modulated dialogue text and the speaker's role, specifically including:

[0104] Based on the speaker's role, the speaker's overall characteristics are obtained;

[0105] The speaker's overall characteristics, speaker ID t1, and the emotion modulation dialogue text are combined to form input features;

[0106] Input features are fed into a pre-defined VoiceChat model to obtain the target speech;

[0107] Stream the target speech.

[0108] This embodiment combines the overall speaker characteristics of the selected speaker role with the emotionally modulated dialogue text as input features for the VoiceChat model. The VoiceChat model then obtains the target speech and outputs it in a streaming manner, achieving personalized speech synthesis. This enables interactive speech generation capabilities for multiple roles and emotions without the need for fine-tuning the model for specific speakers.

[0109] It's important to note that the VoiceChat model in this embodiment was pre-trained on a large-scale multilingual dataset and follows the aforementioned input data format. Therefore, it exhibits good generalization performance and can adapt to new samples without fine-tuning. Furthermore, the VoiceChat model is based on a general large-scale model architecture, does not rely on phonemes for text-to-speech conversion, and directly maps characters to Mel spectrograms. This allows it to handle any language represented by a text, reducing the complexity of the method.

[0110] In one specific embodiment, when the role settings include speaker A and speaker B, the user selects the speaker role, such as speaker A (the other party's role) and speaker B (the user's own role);

[0111] Next, as Figure 3 As shown, the user records in real time through the microphone to obtain the real-time audio of speaker B (the user's own role); speaker A (the other party's role) uses the system's preset feature data and does not need to obtain real-time audio.

[0112] The dialogue text was obtained using a text transcription model based on speaker B's real-time audio.

[0113] By performing sentiment modulation on the dialogue text using a large audio-text dialogue model, we obtained the sentiment-modified dialogue text t7.

[0114] Load the overall features F of speaker A respectively A Overall characteristics F of speaker B B ;

[0115] The features of speaker A, the features of speaker B, the ID t1 of speaker A, and the emotion modulation dialogue text t7 are concatenated with the following formula to form the model input I: where I = Concat(F A ,F B ,t1,t7); where Concat is the concatenation operation; after encoding, each type of data is adjusted to 1024 dimensions;

[0116] After receiving the input feature I, the VoiceChat model directly maps the character to the Mel spectrum, generating the target speech without the need for traditional phoneme conversion and prosody annotation.

[0117] Ultimately, the generated target speech is played in real time via streaming output, enabling highly natural and personalized interaction between the user and the method.

[0118] In this embodiment, the text transcribed from the user's real-time audio is "I didn't do well on today's exam, I'm in a bad mood, what can I do to relieve it?"

[0119] In this embodiment, the audio-text model analyzes the text and concludes that "this audio expresses feelings of disappointment and helplessness due to failing an exam, while also revealing a desire to find a solution. The tone and emotion may be low and helpless with a hint of longing."

[0120] In this embodiment, the model input t6 is "I didn't do well on today's exam, and I feel terrible. What can I do to relieve this feeling?" This audio expresses the disappointment and helplessness caused by failing the exam, while also revealing a desire to find a solution. The tone and emotion may be low and helpless, with a hint of longing.

[0121] In this embodiment, the text content t7 of the target speech is " <break>I completely understand how you feel right now. The frustration of failing the exam feels like a heavy stone pressing on your chest, doesn't it? First, gently hug the part of yourself that's trying so hard. Allow yourself half an hour to space out, listen to a melancholic song, or sigh to the sky. <breath>These emotions are all permitted. <laugh>Give yourself an encouraging funny face in the mirror, or run three laps around the track and let the sweat wash away your frustration—when your body is moving, your brain releases dopamine, which makes you happy! <break> 。 <long-break>Remember? Exams are just a stop on the long race of life, not the cutoff score. Last year, a senior scored 47 in physics. He turned his error notebook into a journal, recording every knowledge point with comics. Later, he turned his life around and ranked in the top ten of his grade in the college entrance exam. You see, your current confusion might just be the beginning of your transformation. Believe that there are always stars hidden behind the clouds.

[0122] Secondly, this invention also proposes an interactive speech synthesis system based on emotion modulation, comprising:

[0123] The speaker role module is used to select the speaker role;

[0124] The dialogue text acquisition module is used to acquire speaker audio in real time; perform speech-to-text transcription on the speaker audio to obtain the first text; perform audio emotion and intent recognition on the speaker audio to obtain the second text; and concatenate the first text and the second text to obtain the dialogue text.

[0125] The interactive emotion control module is used to perform emotion control on the dialogue text to obtain the emotion-controlled dialogue text.

[0126] The speech synthesis module is used to output the target speech based on the emotional adjustment of the dialogue text and the speaker's role.

[0127] In this embodiment, the dialogue text acquisition module is pre-set with a text transcription model and an audio-text dialogue model. The text transcription model is used to transcribe the speaker's audio into text to obtain the first text; the audio-text dialogue model is used to identify the speaker's audio emotional intent to obtain the second text.

[0128] The expression for the dialogue text is t6 = Concat(ASR(input)). audio ),ALM(input audio ));

[0129] In the formula, ASR is the text transcription model, ALM is the large-scale audio-text dialogue model, and input... audio t6 represents the speaker's audio, t6 represents the dialogue text, and Concat indicates concatenation.

[0130] The expression for the sentiment-modified dialogue text is t7 = LLM(History(t6)); where t6 represents the dialogue text, LLM is the large language model, History is the operation of wrapping the dialogue text t6 into the historical dialogue record, and t7 is the sentiment-modified dialogue text.

[0131] In this embodiment, the speaker role module is also used to add speaker roles; obtain speaker videos of speaker roles; extract features from speaker videos to obtain speaker role features; and combine speaker role features as overall speaker features.

[0132] In a further embodiment, a storage module is also included, which is used to save the speaker role and the corresponding speaker general characteristics.

[0133] The speaker role characteristics in this embodiment include: speaker ID, speaker facial feature vector, speaker age information, speaker video material audio data content, character style description, and speaker video material text content.

[0134] The process of obtaining speaker role characteristics is similar to that in the first aspect.

[0135] Thirdly, the present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the emotion-modulation-based interactive speech synthesis method as described in any one of the first aspects.

[0136] Fourthly, the present invention also proposes a computer program product, comprising a computer program that, when executed by a processor, implements the steps of the emotion-modulation-based interactive speech synthesis method as described in any one of the first aspects.

[0137] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention. < / break> < / laugh> < / breath> < / break> < / audio> < / age> < / face> < / laugh> < / breath> < / long-break> < / break>

Claims

1. An emotion-regulation based interactive speech synthesis method, characterized by, Comprise: selecting a speaker role; real-time obtaining speaker role audio; performing speech-to-text transcription on the speaker role audio to obtain a first text; performing audio sentiment and intent recognition on the speaker role audio to obtain a second text; The first text and the second text are spliced to obtain the dialogue text; wherein, the expression of the dialogue text is ; in the formula, is a text transcription model, is an audio-text dialogue large model, is a speaker role audio, is the dialogue text, represents splicing; The preset large language model is used in combination with historical dialogue records to perform sentiment analysis and text generation processing on the dialogue text to obtain a sentiment regulation dialogue text; wherein, an expression of the sentiment regulation dialogue text is ; in the expression, represents the dialogue text, is a large language model, is an operation of obtaining the dialogue text into the historical dialogue records, is a sentiment regulation dialogue text that has been subjected to sentiment regulation; According to the speaker role, a speaker overall feature is obtained; the speaker overall feature is formed into an input feature with the emotion regulation dialogue text; the input feature is input into a preset model, a target voice is obtained; and the target voice is output in a streaming manner.

2. The affect-regulated interactive speech synthesis method of claim 1, wherein, Before selecting the speaker role, further comprising: adding a speaker role; obtaining a speaker video of the speaker role; performing feature extraction on the speaker video to obtain speaker role features; combining the speaker role features as speaker overall features; Preferably, after combining the speaker role features as speaker overall features, further comprising: saving the speaker role and the corresponding speaker overall features.

3. The affect-regulated interactive speech synthesis method of claim 2, wherein, The speaker role features include: speaker ID, speaker face feature vector, speaker age information, video material audio data content, character style description, and speaker video material text content.

4. The affect-regulated interactive speech synthesis method of claim 3, wherein, Performing feature extraction on the speaker video to obtain speaker role features, specifically comprising: automatically sorting the speaker video according to the entry order of the speaker video, and taking the order of the speaker video as the speaker ID; using a pre-set face recognition model to perform face feature extraction on the speaker video to obtain a speaker face feature vector; using a pre-set age prediction model to perform age prediction on the speaker video to obtain speaker age information; using an audio-video separation tool to separate the audio of the speaker video to obtain video material audio data content; using a pre-set graphic-text dialogue large model to extract character style from the speaker video to obtain character style description; using a pre-set text transcription model to perform speech-to-text transcription on the speaker video to obtain speaker video material text content.

5. An affect-regulated interactive speech synthesis system, characterized by Comprise: a speaker role module for selecting a speaker role; a dialogue text acquisition module for real-time obtaining speaker role audio; The speech-to-text transcription is performed on the speaker role audio to obtain a first text; audio emotion and intent recognition is performed on the speaker role audio to obtain a second text; the first text and the second text are spliced to obtain a dialogue text; wherein the expression of the dialogue text is ; in the formula, is a text transcription model, is an audio-text dialogue large model, is the speaker role audio, is the dialogue text, represents splicing; An interactive emotion regulation module is configured to utilize a preset large language model to perform emotion analysis and text generation processing on the dialogue text in combination with historical dialogue records to obtain emotion regulation dialogue text; wherein the expression of the emotion regulation dialogue text is ; in the expression, represents the dialogue text, is the large language model, is an operation of obtaining the dialogue text wrapped in the historical dialogue records, is the emotion regulation dialogue text after emotion regulation; The voice synthesis module is configured to obtain a speaker general feature according to a speaker role; form an input feature by combining the speaker general feature and the emotion-regulated dialogue text; input the input feature into a preset model to obtain a target voice; and output the target voice in a streaming manner.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the interactive speech synthesis method based on emotion regulation in any one of claims 1-4.

7. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the interactive speech synthesis method based on emotion regulation in any one of claims 1-4.

Images