A sound cloning method based on emotional response

Abstract

The application discloses a voice cloning method based on emotional response, comprising: obtaining input text and generating an emotional vector; obtaining a reference voice segment according to the emotional vector to calculate acoustic emotional intensity; fusing acoustic emotional intensity, text information and emotional keyword syntax structure weight based on dependency syntax analysis to generate refined emotional intensity indicators; adjusting acoustic parameters based on the indicators and synthesizing a preliminary cloned voice segment; performing feedback optimization based on context comparison on the preliminary cloned voice segment to obtain an optimized voice parameter set; and performing iterative synthesis using the optimized parameter set to generate a final cloned voice. The method solves the problem of rigid emotional expression of cloned voices and low matching degree of emotional intensity and voice parameters in the prior art, and improves the emotional authenticity and appeal of cloned voices.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a voice cloning method based on emotional response. Background Technology

[0002] Voice cloning technology, as an important branch of speech synthesis, has broad application prospects in scenarios such as virtual assistants, audiobooks, film and television dubbing, and personalized human-computer interaction. Current voice cloning technology can, to a certain extent, replicate the basic voice features of a specific speaker, such as timbre and pitch, thereby generating speech with the speaker's identity characteristics. However, existing voice cloning methods still have shortcomings in the dimension of emotional expression.

[0003] Existing methods typically focus on mimicking the physical properties of sound, with limited ability to capture the complex emotional information embedded in text content. This results in cloned voices, while possessing similar timbre, often sounding monotonous, stiff, and lacking realism and emotional impact when expressing emotions such as joy, anger, sorrow, and happiness, making them unsuitable for scenarios requiring rich emotional interaction. When faced with texts of varying emotional nuances, cloned voices cannot adjust their emotional expression accordingly, leading to inconsistencies between the sound and the content's emotional tone and negatively impacting the user experience.

[0004] Furthermore, one of the technical challenges in achieving emotional voice cloning lies in how to accurately quantify the intensity of emotion and establish a stable mapping relationship between this quantification and acoustic parameters (such as pitch, speech rate, and volume). Emotional intensity is a subjective and dynamically changing attribute, related not only to emotional words in the text but also influenced by context, syntactic structure, and other factors. Existing technologies typically employ simplistic methods to assess emotional intensity, such as relying solely on the frequency of emotional words. This approach ignores subtle differences and dynamic variations in emotional expression, resulting in an inability to accurately control the emotional intensity of synthesized voices. For example, for texts expressing the same emotion, "I am happy" and "I am so happy!" exhibit significantly different emotional intensities, and existing technologies struggle to reflect these differences in cloned voices, thus limiting the effectiveness of voice cloning technology in real-world scenarios. Therefore, accurately quantifying emotional intensity and effectively mapping it to the dynamic adjustment of sound parameters is a pressing technical problem in the field of voice cloning. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a voice cloning method based on emotional response, so as to solve the problems of inaccurate emotional expression and low matching degree between emotional intensity and voice parameters in the background art.

[0006] To address the aforementioned technical problems, one aspect of the present invention provides a voice cloning method based on emotional response, the method comprising the following steps:

[0007] Step one involves acquiring the input text and performing natural language processing to extract its semantic features and contextual information. A pre-defined sentiment recognition model is then used to analyze these semantic features and contextual information, generating a sentiment vector representing the sentiment type and intensity of the input text. Specifically, the sentiment recognition model can be a deep learning-based neural network model, trained on massive amounts of sentiment-annotated text data. This allows the model to map the input text features to a multi-dimensional sentiment space, obtaining sentiment vectors containing multiple sentiment categories and their corresponding probability distributions. Optionally, after generating the sentiment vector, it can be preliminarily adjusted by incorporating historical interaction data or feedback information from specific business scenarios to improve its accuracy in specific application scenarios.

[0008] Step two: Based on the emotion type represented in the emotion vector, retrieve and obtain one or more reference sound segments matching the emotion type from a preset audio sample database; calculate the matching degree between the emotion intensity contained in the reference sound segment and the emotion vector; when the matching degree is higher than a preset first threshold, extract acoustic features such as pitch parameters and speech rate parameters of the reference sound segment, and calculate the acoustic emotion intensity based on the acoustic features. The audio sample database pre-stores a large amount of audio data labeled with emotion type and emotion intensity. By comparing the emotion vector with the samples in the database, a true acoustic representation of text emotion can be found as a reference, thus making the quantification of emotion intensity more reliable.

[0009] Step three involves fusing the acoustic sentiment intensity with text-related information extracted from the input text to generate a refined sentiment intensity index. This invention achieves precise quantification of sentiment intensity through a phased, iterative sentiment intensity index generation mechanism. Specifically, the acoustic sentiment intensity is first used as a basic component, combined with information such as the text length and sentiment keyword density of the input text, and calculated using a preset weighted fusion algorithm to obtain an intermediate intensity value. Furthermore, to refine the calculation of the sentiment intensity index, this method also introduces sentiment keyword syntactic structure weight analysis based on dependency parsing. By performing dependency parsing on the input text, the role of sentiment keywords in the syntactic structure is identified, and different weights are assigned accordingly. For example, degree adverbs modifying sentiment keywords (such as "very" or "slightly") are identified, and their weights are adjusted based on the intensity of the degree adverbs; or, different weights are assigned based on the syntactic position of sentiment keywords in the sentence (such as main components or modifiers), with keywords in main structures such as subject-verb-object typically having higher weights. By comprehensively considering this syntactic structural information, the intermediate intensity value is further weighted and adjusted to obtain a refined emotional intensity index that can more accurately reflect the intensity of the true emotions in the text.

[0010] Step four: Based on the refined emotional intensity index, the acoustic parameters of the target sound are adjusted using a parameter mapping algorithm to generate an adjusted pitch parameter sequence. Specifically, it is determined whether the refined emotional intensity index exceeds a preset intensity index range. If it does, a dynamic scaling mechanism is activated, such as using linear interpolation to map the index value to a scaling ratio of the acoustic parameters. Simultaneously, the changing trend of the refined emotional intensity index with the text content is analyzed, and a correspondence between it and the pitch change trend is established, thereby generating an adjusted pitch parameter sequence that reflects the dynamic changes in emotion. Optionally, when generating the pitch parameter sequence, speech rhythm information extracted from a reference sound segment, such as pauses and stresses, can be integrated to ensure that the generated pitch parameter sequence also matches the target emotion in rhythm.

[0011] Step five involves inputting the adjusted pitch parameter sequence and the preset original timbre template into the sound synthesis model, and using waveform generation technology to synthesize a preliminary cloned sound segment. The original timbre template is a neutral, emotionless timbre sound sample of the target cloned speaker. The sound synthesis model, such as a vocoder based on a deep neural network, can generate sound waveform data containing the target emotion and target timbre, i.e., the preliminary cloned sound segment, based on the input pitch parameter sequence and timbre template.

[0012] Step six involves feedback optimization of the initially cloned sound segments to generate an optimized set of sound parameters. This invention also employs a context-based feedback loop optimization mechanism to further refine the synthesized sound. Specifically, the generated initially cloned sound segments are compared in a real-time interactive context, or emotional response information is obtained by simulating interactive scenarios. It is determined whether the emotional response information deviates from the target emotion. If a deviation exists (e.g., the synthesized soothing voice fails to alleviate the user's negative emotions), it indicates that the current sound parameters do not fully match the emotional intensity. At this point, at least one of the speech rate and volume parameters is adjusted to better match the refined emotional intensity index, thereby obtaining the optimized set of sound parameters. This step constitutes a feedback loop after audio generation, improving the method's adaptability to complex interactive scenarios.

[0013] Step seven involves using the optimized sound parameter set to iteratively synthesize the intermediate representation of the initial cloned sound segment, generating the final complete audio sequence. Specifically, the optimized sound parameter set is applied again to the parameter mapping algorithm to fuse and adjust the spectral features of the initial audio data. During the iterative synthesis process, dynamic change features and rhythmic features defined by the parameter set can be superimposed, and the spectral data can be processed to ensure that the final generated audio achieves accurate mapping in terms of emotional intensity, emotional dynamics, and rhythmic expression. Finally, through processing such as time-domain transformation, a complete audio sequence that achieves accurate mapping of emotional intensity is generated, i.e., the final cloned sound.

[0014] The technical solution provided by this invention achieves precise cloning from text to emotional voice by constructing a multi-stage, iterative method for quantifying emotional intensity and combining it with a dual-feedback loop optimization mechanism. First, an initial emotional vector is generated through an emotional recognition model. Second, acoustic emotional intensity is calculated using an audio sample database, and a refined emotional intensity index is generated by integrating text length, keyword density, and innovative syntactic structure weight analysis, improving the accuracy of emotional intensity quantification. Third, a refined mapping and synthesis process is established from the refined emotional intensity index to acoustic parameters such as pitch and speech rate, ensuring that emotional information is accurately reflected in the physical properties of the sound. Finally, by introducing a feedback optimization stage of real-time context comparison after audio generation, the sound parameters are adjusted and iteratively synthesized, further improving the emotional adaptability and naturalness of the cloned voice in real-world interactive scenarios. This invention solves the problems of stiff emotional expression and low matching degree between emotional intensity and sound parameters in existing technologies for cloned voices, improving the emotional realism and appeal of cloned voices, thereby improving the user experience in personalized human-computer interaction and other application scenarios. Attached Figure Description

[0015] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0016] Figure 1 This is a flowchart illustrating a voice cloning method based on emotional response provided in an embodiment of the present invention.

[0017] Figure 2 This is a schematic diagram illustrating the process of generating a refined emotional intensity index in an embodiment of the present invention.

[0018] Figure 3 This is a schematic diagram of the process for feedback optimization of the initially cloned sound segment in an embodiment of the present invention.

[0019] Figure 4 This is a schematic diagram of the structure of a voice cloning device based on emotional response provided in an embodiment of the present invention.

[0020] Figure 5 This is a schematic diagram illustrating the mapping relationship between the refined emotional intensity index and the tone parameter sequence in an embodiment of the present invention.

[0021] Figure 6 This is a schematic diagram comparing the time domain of audio waveforms under different emotional intensity indicators in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Example 1

[0024] This embodiment provides a voice cloning method based on emotional response, referring to... Figure 1 This method is primarily applied to smart devices or software systems that require personalized and emotional voice interaction. It generates cloned voices with rich emotional expression by precisely quantifying the emotional intensity of text and mapping it to acoustic parameters.

[0025] Step S100: Obtain the input text and perform natural language processing on the input text to extract its semantic features and context information; use a preset sentiment recognition model to analyze the semantic features and context information to generate a sentiment vector representing the sentiment type and sentiment intensity of the input text.

[0026] Specifically, in one application scenario, a user inputs a text via voice or text, such as "I finally finished my work today, it's so great!" The system first acquires this input text. Then, it performs natural language processing preprocessing operations on the text, including word segmentation, part-of-speech tagging, and stop word removal. The processed text is then input into a pre-defined sentiment recognition model. Preferably, the sentiment recognition model is a pre-trained language model based on the Transformer architecture, such as BERT (Bidirectional Encoder Representations from Transformers). This model, trained on massive amounts of text corpora labeled with sentiment information, is able to understand the deep semantics and contextual relationships of the text. After analysis, the model outputs a multi-dimensional sentiment vector. This vector can be a numerical array containing multiple sentiment dimensions and their corresponding confidence scores, for example, [joy: 0.9, surprise: 0.2, neutral: 0.1], and also includes a text sentiment intensity value, for example, 7.5 (assuming the intensity range is 0-10). Optionally, to improve accuracy in specific scenarios, the system can also be adjusted based on historical interaction data. For example, if the system records that a user's subsequent interactive behavior is more positive when expressing strong positive emotions, it can fine-tune the confidence level of joy or the intensity value of text emotion in the currently generated emotion vector based on this feedback information, so that it is more in line with the user's personalized expression habits.

[0027] Step S200: Based on the emotion type represented in the emotion vector, retrieve and obtain one or more reference sound segments that match the emotion type from a preset audio sample database; calculate the matching degree between the emotion intensity contained in the reference sound segment and the emotion vector; when the matching degree is higher than a preset first threshold, extract the acoustic features such as pitch parameters and speech rate parameters of the reference sound segment, and calculate the acoustic emotion intensity based on the acoustic features.

[0028] Specifically, the system uses the emotion vector (primarily joy) generated in step S100 to search a pre-set audio sample database. This database pre-stores a large amount of audio data recorded by different speakers, with detailed emotion annotations (such as emotion type and emotion intensity score). The search process can be performed by calculating the cosine similarity between emotion vectors to find the reference sound segment with the closest emotional expression. For example, the system retrieves three audio segments that all express "high joy". The system calculates the matching degree between the emotion intensity of these three segments and the emotion vector corresponding to the input text, and selects the segment with a matching degree higher than a preset first threshold (e.g., 0.85). Assuming a segment is finally selected, the system uses acoustic analysis tools (such as Praat software) to extract the acoustic features of the segment, such as the fundamental frequency (F0) curve, average speech rate, and energy distribution. Based on these objective acoustic features, an acoustic emotion intensity that can represent the true emotional expression of the sound is calculated through a preset conversion model, for example, 8.0. This step anchors the abstract emotional intensity of the text to acoustically based numerical values ​​using real audio samples, thus avoiding the subjectivity of pure text analysis.

[0029] Step S300: Integrate the acoustic emotion intensity and the text-related information extracted from the input text to generate a refined emotion intensity index.

[0030] First, the acoustic sentiment intensity obtained in step S200 is used as the base component. Simultaneously, other text-related information, such as text length and sentiment keyword density, is extracted from the input text. For the text "My work is finally finished today, it's great!", the text length is relatively long and contains multiple sentiment-enhancing words such as "finally" and "it's great!", resulting in a high sentiment keyword density. This information is fused with the acoustic sentiment intensity using a pre-defined weighted fusion algorithm to obtain an intermediate intensity value. This algorithm can be expressed as: I_mid = w1 * I_prelim + w2 * f_len(L) + w3 * f_den(D). Where I_mid represents the intermediate intensity value; I_prelim represents the acoustic sentiment intensity; L represents the text length; D represents the sentiment keyword density; w1, w2, and w3 are pre-defined weighting coefficients; f_len() and f_den() are pre-defined functions that map the text length and keyword density to intensity components; wherein, the pre-defined functions are used to map the longer text length and higher keyword density to intensity components that have a positive gain on the sentiment intensity.

[0031] Furthermore, to more precisely characterize sentiment intensity, this method introduces sentiment keyword syntactic structure weighting analysis based on dependency parsing. The system performs dependency parsing on the input text and constructs a syntactic tree. In the clause "It's really great," the analysis results show that "really" and "too" are degree adverbs modifying "great." Based on a pre-defined dictionary of degree adverbs, the system assigns higher sentiment enhancement weights to "really" and "too." Simultaneously, the sentiment keyword "great," as the predicate of the sentence and located in a core syntactic position, is also assigned a high weight. By integrating this syntactic structure information, the intermediate intensity value is further weighted and adjusted to ultimately generate a refined sentiment intensity index. For example, after this refined adjustment step, the final sentiment intensity index may increase from 8.0 to 9.2, thus more accurately reflecting the strong emotions contained in the original text.

[0032] Step S400: Based on the refined emotional intensity index, the acoustic parameters of the target sound are adjusted using a parameter mapping algorithm to generate an adjusted pitch parameter sequence.

[0033] Specifically, the system obtains the refined sentiment intensity index (9.2) generated in step S300. (Refer to...) Figure 5 This diagram visually illustrates how different emotional intensity indices (e.g., neutral, low intensity 3.5, high intensity 9.2) are mapped to pitch parameter sequences with varying amplitudes. First, it determines whether the index exceeds a preset intensity range (e.g., [1, 10]). If not, it is directly used for parameter mapping. If it does exceed the range (e.g., a calculated value of 11 might occur under extreme emotions), a dynamic scaling mechanism is activated, such as through a sigmoid function or linear interpolation, to smoothly map it back to the range, generating a global scaling ratio for the acoustic parameters. Subsequently, the system analyzes the dynamic changing trend of the refined emotional intensity index within the text (e.g., from a flat narration at the beginning of a sentence to an emotional outburst at the end) and establishes its correspondence with pitch change trends. For example, an intensity index of 9.2 corresponds to a wider pitch range, a higher average fundamental frequency, and faster pitch fluctuations. Based on this, the system generates an adjusted pitch parameter sequence that reflects this strong emotional dynamic. Optionally, to make the rhythm of the cloned voice more natural, speech rhythm information such as pauses and stresses extracted from the reference voice segment in step S200 can be integrated when generating the pitch parameter sequence, so that the generated pitch parameter sequence also matches the target emotion in terms of rhythm and rhyme.

[0034] Step S500: Input the adjusted pitch parameter sequence and the preset original timbre template into the sound synthesis model, and use waveform generation technology to synthesize a preliminary cloned sound segment.

[0035] In this step, the system invokes a voice synthesis model, preferably a neural network vocoder, such as a high-fidelity generative adversarial network vocoder. The model's input consists of two parts: one is the adjusted pitch parameter sequence generated in step S400, which contains the target's emotional information; the other is a preset original timbre template. This timbre template is a voice feature vector (e.g., an x-vector, a deep learning embedding vector used for speaker recognition) extracted from a small amount of neutral, emotionless speech recorded by the target clone speaker, uniquely representing the speaker's timbre. Based on the timbre template and guided by the pitch parameter sequence, the voice synthesis model generates sound waveform data frame by frame, ultimately synthesizing a preliminary cloned voice segment. This voice is timbre-consistent with the target speaker and expresses the strong joy inherent in the input text.

[0036] Step S600: Perform feedback optimization on the initially cloned sound segment to generate an optimized set of sound parameters.

[0037] Reference Figure 3 This step introduces a feedback loop to improve the adaptability of the sound in real-world interactions. Specifically, the preliminary cloned sound segment generated in step S500 is placed in a simulated interactive context. For example, the system simulates a dialogue scenario, and after playing the cloned sound, it analyzes a pre-defined emotional response model of a virtual audience. This model can be trained based on a large-scale emotional speech database, learning the correspondence between different acoustic features and human emotional perception. Its input is the acoustic features of the preliminary cloned sound segment, and its output is the classification and confidence level of the emotion expressed by the sound, which serves as the emotional response information. If the emotional response information deviates from the target emotion, the system considers that the current sound parameters do not fully match the emotional intensity. At this time, the system initiates an optimization program to adjust the speech rate and volume parameters. For example, the system may determine that for intense joy, in addition to a high pitch, there should also be a faster speech rate and a larger volume. Therefore, the system will adjust the speech rate and volume parameters accordingly, forming an optimized set of sound parameters, which includes adjusted pitch, speech rate, volume, and other parameters across multiple dimensions.

[0038] Step S700: Using the optimized set of sound parameters, perform iterative synthesis on the intermediate representation of the initially cloned sound segment to generate the final complete audio sequence.

[0039] Finally, the system applies the optimized set of sound parameters obtained in step S600 back to the sound synthesis process. Specifically, instead of resynthesizing from scratch, it uses the optimized set of parameters (such as adjusted speech rate and volume information) to fuse and adjust the preliminary audio data generated in step S500, based on the intermediate representation (e.g., Mel spectrogram). During iterative synthesis, the system superimposes dynamic features (e.g., volume increases with emotional intensity) and rhythmic features (e.g., brief pauses before emotional climaxes) defined by the parameter set onto the spectral data. After processing, a time-domain transformation technique is used to generate the final, complete audio sequence that accurately maps emotional intensity. This final cloned voice not only has a realistic timbre, but its emotional expression, dynamic changes, and rhythm are also highly consistent with the original text. (See reference...) Figure 6 The figure shows the final audio waveforms generated under different emotional intensity indices. As can be seen from the figure, the waveform generated by the refined emotional intensity index (9.2) exhibits more dramatic dynamic changes in both amplitude and frequency, thus achieving more expressive emotional speech.

[0040] Example 2

[0041] This embodiment provides a voice cloning device based on emotional response. This device can be a computing device such as a server, personal computer, or mobile terminal. (Refer to...) Figure 4 The device includes:

[0042] The text processing module 10 is used to acquire input text, perform natural language processing on the input text, extract its semantic features and context information, and analyze the semantic features and context information using a preset sentiment recognition model to generate a sentiment vector that represents the sentiment type and sentiment intensity of the input text.

[0043] The preliminary quantization module 20 is used to retrieve and obtain one or more reference sound segments that match the emotion type represented in the emotion vector from a preset audio sample database; calculate the matching degree between the emotion intensity contained in the reference sound segment and the emotion vector; when the matching degree is higher than a preset first threshold, extract the acoustic features such as pitch parameters and speech rate parameters of the reference sound segment, and calculate the acoustic emotion intensity based on the acoustic features.

[0044] The intensity refinement module 30 is used to fuse the acoustic sentiment intensity with text-related information extracted from the input text to generate a refined sentiment intensity index. This module further includes a functional unit for performing sentiment keyword syntactic structure weight analysis based on dependency parsing to fine-tune the sentiment intensity.

[0045] The parameter adjustment module 40 is used to adjust the acoustic parameters of the target sound based on the refined emotional intensity index through a parameter mapping algorithm, and generate an adjusted pitch parameter sequence.

[0046] The preliminary synthesis module 50 is used to input the adjusted pitch parameter sequence and the preset original timbre template into the sound synthesis model, and use waveform generation technology to synthesize a preliminary cloned sound segment.

[0047] The feedback optimization module 60 is used to perform feedback optimization on the initially cloned sound segment, generating an optimized set of sound parameters. This module compares the initially cloned sound segment in a real-time interactive context and determines whether there is a deviation based on emotional response information. If a deviation exists, at least one of the speech rate parameter and volume parameter is adjusted.

[0048] The final synthesis module 70 is used to perform iterative synthesis on the intermediate representation of the preliminary cloned sound segment using the optimized set of sound parameters to generate the final complete audio sequence.

[0049] Example 3

[0050] This embodiment is applied to an intelligent vehicle assistance system, aiming to solve the technical problem that existing technologies cannot effectively convey a sense of urgency through cloned voices when broadcasting emergency safety warning information. Conventional voice cloning methods, when processing warning text, can usually only generate voices with consistent timbre but bland emotions, which are difficult to attract the driver's full attention. However, the method of this invention can generate cloned voices with a high degree of alertness through precise quantification and feedback optimization of emotional intensity.

[0051] Specifically, when the system receives input text, such as "An emergency has occurred ahead, and the vehicle needs to leave immediately at the exit ahead," it first executes step one, analyzing the text through an emotion recognition model to generate an emotion vector representing the emotion type of "anxiety" or "emergency." Then, it executes step two, matching and retrieving reference sound segments with similar emergency emotions from an audio sample database based on this emotion vector, extracting acoustic features from these segments, and calculating a preliminary quantification value of the emotion intensity.

[0052] In this scenario, the refinement process in step three demonstrates the necessity of this invention. For the aforementioned warning text, conventional methods may underestimate its urgency due to a lack of in-depth understanding of sentence structure. The method of this invention, through weighted analysis based on dependency parsing, can identify "sudden situation" and "emergency departure" as the core events and instructions of the sentence, and identify the adverbial relationship between "emergency" and "departure." The system will therefore assign higher weights to these keywords and their syntactic structures, thereby effectively weighting and adjusting the initial quantification value to generate a refined sentiment intensity index that accurately reflects the high danger level of the warning information. The value of this index will be higher than that obtained by conventional methods without using syntactic analysis.

[0053] Subsequently, in steps four and five, this high-intensity index is used to adjust the acoustic parameters, generate a sequence of pitch parameters with a higher average fundamental frequency and a wider pitch range, and synthesize preliminary cloned sound segments.

[0054] The key lies in the feedback optimization mechanism in step six, which solves the closed-loop verification problem of the actual sound effect. The system inputs the initial cloned sound segment into a simulated driving environment for performance evaluation. The goal of this evaluation is to detect whether the simulated driver's response time is within a safe threshold. If the system detects that the initial sound, with only a change in pitch, fails to trigger a sufficiently rapid response, it determines that there is an emotional expression deviation. At this point, the feedback optimization module is activated, adjusting other acoustic parameters. For example, without affecting clarity, it moderately increases the overall speech rate and shortens the pause time between "the vehicle needs to be at the exit ahead" and "emergency departure." Through this adjustment, an optimized set of sound parameters is generated.

[0055] Finally, in step seven, the optimized parameter set is used to iteratively synthesize the audio, generating the final cloned sound. This sound not only has a high-pitched and rapid tone, but its speech rate and rhythm have also been optimized to more effectively convey the urgency of instructions and ensure that warning information is effectively received in complex driving environments. This demonstrates the improvement of the present invention's method compared to existing technologies in a specific application.

[0056] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0057] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A voice cloning method based on emotional response, characterized in that, Includes the following steps: Step 1: Obtain the input text, analyze the input text using an emotion recognition model, and generate an emotion vector representing the emotion type and intensity of the input text. Step 2: Based on the emotion vector, retrieve reference sound segments that match the emotion type from the audio sample database, and calculate the acoustic emotion intensity based on the acoustic features of the reference sound segments; Step 3: Integrate the acoustic sentiment intensity with the text-related information extracted from the input text to generate a refined sentiment intensity index. The process of generating the refined sentiment intensity index includes: weighting and fusing the acoustic sentiment intensity with the text length and sentiment keyword density information of the input text to obtain an intermediate intensity value; and performing dependency parsing on the input text to identify degree adverbs modifying sentiment keywords and the position of the sentiment keywords in the syntactic structure, thereby determining the syntactic structure weights, and then using the syntactic structure weights to further weight and adjust the intermediate intensity value to obtain the refined sentiment intensity index. Step four: Based on the refined emotional intensity index, adjust the acoustic parameters of the target sound using a parameter mapping algorithm to generate an adjusted pitch parameter sequence; Step 5: Input the adjusted pitch parameter sequence and the preset original timbre template into the sound synthesis model to synthesize a preliminary cloned sound segment; Step six: Perform feedback optimization on the preliminary cloned sound segment. When the emotional response information of the preliminary cloned sound segment in the interactive context deviates from the emotional type represented by the emotional vector, adjust at least one of the speech rate parameter and volume parameter to generate an optimized set of sound parameters. Step 7: Using the optimized set of sound parameters, perform iterative synthesis on the intermediate representation of the preliminary cloned sound segment to generate the final complete audio sequence; The emotion recognition model is a neural network model based on deep learning, and after generating the emotion vector, it further includes: adjusting the emotion vector by combining historical interaction data or feedback information from preset types of business scenarios.

2. The method according to claim 1, characterized in that, The step of determining the syntactic structure weight based on the position of the sentiment keyword in the syntactic structure specifically includes: assigning different weights based on whether the sentiment keyword is in the main component of the sentence, wherein the sentiment keyword in the subject-verb-object structure is assigned a higher weight.

3. The method according to claim 1, characterized in that, Step four further includes: determining whether the refined emotional intensity index exceeds a preset intensity index range; if it does, then activating a dynamic scaling mechanism to map the refined emotional intensity index value to a scaling ratio of acoustic parameters.

4. The method according to claim 1, characterized in that, In step six, the method of obtaining the emotional response information includes: placing the initially cloned sound segment in a simulated interactive scenario and analyzing it through a preset emotional response model to obtain the emotional response information; the adjusted acoustic parameters include speech rate parameters and volume parameters.

5. The method according to claim 1, characterized in that, The sound synthesis model is a vocoder based on a deep neural network.

6. The method according to claim 1, characterized in that, In step two, after retrieving the reference sound segment, the method further includes: calculating the matching degree between the emotional intensity contained in the reference sound segment and the emotional vector; and extracting the acoustic features of the reference sound segment to calculate the acoustic emotional intensity only when the matching degree is higher than a preset first threshold.

7. The method according to claim 1, characterized in that, In step seven, the intermediate representation of the preliminary cloned sound segment is a Mel spectrogram; the iterative synthesis includes: superimposing the dynamic change features and rhythmic features defined by the optimized sound parameter set onto the Mel spectrogram, and performing a time-domain transformation on the processed Mel spectrogram to generate the final complete audio sequence.

8. The method according to claim 1, characterized in that, Step four further includes: when generating the adjusted pitch parameter sequence, incorporating speech rhythm information extracted from the reference sound segment.

9. A voice cloning device based on emotion response, used to execute the voice cloning method based on emotion response according to any one of claims 1-8, characterized in that, include: The text processing module is used to perform step one as described in claim 1; A preliminary quantization module is used to perform step two as described in claim 1; A strength refining module, used to perform step three as described in claim 1; The parameter adjustment module is used to perform step four as described in claim 1; A preliminary synthesis module is used to perform step five as described in claim 1; The feedback optimization module is used to perform step six as described in claim 1; The final synthesis module is used to perform step seven as described in claim 1.

Images