Method, device and equipment for personalized speech synthesis based on scene perception and natural language description

Abstract

The application provides a personalized speech synthesis method and device based on scene perception and natural language description, and relates to the technical field of speech synthesis, aiming to generate speech with personalization, expressiveness and context compliance. The method comprises: performing semantic analysis on a first text to obtain a first semantic feature, wherein the first text comprises text content and scene description; performing scene classification generation according to the first semantic feature to obtain a scene vector, wherein the scene vector represents a scene category; generating a scene audio feature according to the scene vector; performing semantic analysis on a second text to obtain a second semantic feature, wherein the second text at least comprises a natural language description of speech style; performing emotion category prediction and style feature extraction according to the second semantic feature to obtain a style control parameter; and adjusting the synthesized speech parameters of the text content according to the scene audio feature and the style control parameter to obtain synthesized speech.

Description

Technical Field

[0001] This application relates to the field of speech synthesis technology, and in particular to a personalized speech synthesis method, apparatus and device based on scene awareness and natural language description. Background Technology

[0002] Text-to-Speech (TTS) technology converts text information into speech signals and is widely used in fields such as intelligent assistants, navigation systems, audio reading devices, and customer service robots, providing an efficient and natural solution for human-computer interaction. In recent years, deep learning-based speech synthesis models have made significant improvements in speech quality, fluency, and timbre fidelity, basically meeting the basic speech generation needs in general scenarios.

[0003] However, most speech synthesis methods in related technologies rely on a single speech model, failing to dynamically adjust the speech background according to different application scenarios, resulting in a mismatch between synthesized speech and actual application scenarios. Furthermore, emotion recognition methods typically rely on label-based conditional speech generation, unable to fully understand and utilize users' personalized speech style requirements, such as "cheerful tone" or "deep male voice." Therefore, these technologies lack the ability to dynamically adapt to specific application scenarios, user-specific needs, and interactive contexts. Summary of the Invention

[0004] In view of the above problems, embodiments of this application provide a personalized speech synthesis method, apparatus and device based on scene awareness and natural language description, so as to overcome the above problems or at least partially solve the above problems.

[0005] A first aspect of this application discloses a personalized speech synthesis method based on scene awareness and natural language description, the method comprising:

[0006] Perform semantic analysis on the first text to obtain the first semantic feature, wherein the first text includes text content and scene description;

[0007] Based on the first semantic feature, a scene classification is generated to obtain a scene vector, which represents the scene category.

[0008] Scene audio features are generated based on the scene vector, and the scene audio features include detailed features of the application scene and scene-related audio features;

[0009] Semantic analysis is performed on the second text to obtain second semantic features, wherein the second text includes at least a natural language description of the speech style;

[0010] Based on the second semantic feature, emotion category prediction and style feature extraction are performed to obtain style control parameters, which include: pitch, speech rate, tone and volume;

[0011] Based on the scene sound features and the style control parameters, the synthesized speech parameters of the text content are adjusted to obtain synthesized speech.

[0012] Optionally, the synthesized speech parameters of the text content are adjusted according to the scene sound features and the style control parameters to obtain synthesized speech, including:

[0013] The scene sound features and the style control parameters are fused to obtain comprehensive style adjustment parameters;

[0014] Based on the comprehensive style adjustment parameters, the synthesized speech parameters of the text content are dynamically adjusted to obtain synthesized speech.

[0015] Optionally, generating scene sound features based on the scene vector includes:

[0016] The scene vector is matched with each scene sound template in the scene template library to obtain the target scene sound template that matches the scene vector;

[0017] Based on the target scene sound template and the parameters of the speech synthesis model, scene sound features are generated.

[0018] Optionally, semantic analysis is performed on the first text to obtain first semantic features, including:

[0019] The first text is segmented and annotated using a natural language processing model to obtain the segmented and annotated first text.

[0020] The first semantic feature is obtained by semantically encoding the first text that has been segmented and labeled by a text encoder.

[0021] Optionally, sentiment category prediction and style feature extraction are performed based on the second semantic feature to obtain style control parameters, including:

[0022] The sentiment analysis model is used to predict the sentiment category of the second semantic feature to obtain the first voice control parameters, which include the first pitch, the first speech rate and the volume.

[0023] The second semantic feature is subjected to tone recognition by a tone recognition model to obtain the second speech control parameters, which include tone.

[0024] The second semantic feature is style-transformed using a style-transformation model to obtain the third speech control parameters, which include the second pitch and the second speech rate.

[0025] The first voice control parameter, the second voice control parameter, and the third voice control parameter are integrated to obtain the style control parameter.

[0026] Optionally, the sentiment category of the second semantic feature is predicted using a sentiment analysis model to obtain the first voice control parameters, including:

[0027] The sentiment analysis model is used to predict the sentiment category of the second semantic feature to obtain the target sentiment category label.

[0028] Based on the mapping relationship between emotion category labels and voice parameters, the target emotion category label is mapped to the first voice parameter.

[0029] Optionally, the second semantic feature is style-transferred using a style transfer model to obtain the third speech control parameters, including:

[0030] The second semantic feature is processed by the style transfer model to obtain a style vector;

[0031] The style vector is matched with each style template in the style library to obtain the target style template that matches the style vector;

[0032] The voice control parameters corresponding to the target style template are used as the third voice control parameters.

[0033] Optionally, the second text further includes contextual information about the text content; semantic analysis is performed on the second text to obtain second semantic features, including:

[0034] The second text is parsed and encoded using a language parsing model to obtain the second semantic features.

[0035] A second aspect of this application discloses a personalized speech synthesis device based on scene awareness and natural language description, the device comprising:

[0036] The first analysis module is used to perform semantic analysis on the first text to obtain the first semantic features. The first text includes text content and scene description.

[0037] The first generation module is used to perform scene classification and generation based on the first semantic feature to obtain a scene vector, wherein the scene vector represents the scene category.

[0038] The second generation module is used to generate scene audio features based on the scene vector, wherein the scene audio features include detailed features of the application scene and scene-related audio features;

[0039] The second analysis module is used to perform semantic analysis on the second text to obtain second semantic features, wherein the second text includes at least a natural language description of the speech style;

[0040] The first extraction module is used to predict the sentiment category and extract style features based on the second semantic features to obtain style control parameters, which include: pitch, speech rate, tone and volume.

[0041] The first adjustment module is used to adjust the synthesized speech parameters of the text content according to the scene sound features and the style control parameters to obtain synthesized speech.

[0042] A third aspect of this application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the personalized speech synthesis method based on scene awareness and natural language description described in the first aspect of this application.

[0043] A fourth aspect of this application discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the personalized speech synthesis method based on scene awareness and natural language description described in the first aspect of this application.

[0044] A fifth aspect of this application discloses a computer program product, including a computer program that, when executed by a processor, implements the steps of the personalized speech synthesis method based on scene awareness and natural language description described in the first aspect of this application.

[0045] The embodiments of this application have the following advantages:

[0046] In this embodiment, the first text includes text content and scene description. By processing the first text, scene sound features containing detailed features of the application scenario and scene-related audio features are generated, enabling the identification and adaptation to various application scenarios (e.g., news, train stations, dialogues, etc.). Therefore, this method adjusts the style and atmosphere of the synthesized speech based on the scene description of the first text, not only improving the adaptability of the synthesized speech and generating speech that better meets actual needs in various scenarios, but also enhancing the immersion and realism of the synthesized speech. The second text includes at least a natural language description of the speech style. By processing the second text, style control parameters are obtained, enabling accurate parsing and extraction of natural language descriptions of the speech style, thereby adjusting parameters such as the emotional category, intonation features, pitch, and speech rate of the synthesized speech. Therefore, this method can not only understand rich language descriptions but also flexibly adjust the speech output to meet the personalized speech needs of users, significantly improving the flexibility and controllability of speech synthesis. Thus, by combining scene information and natural language descriptions, this method can generate personalized, expressive, and context-appropriate synthesized speech. Attached Figure Description

[0047] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a flowchart illustrating the steps of a personalized speech synthesis method based on scene awareness and natural language description provided in an embodiment of this application.

[0049] Figure 2 This is a schematic diagram of a scene perception module provided in an embodiment of this application;

[0050] Figure 3 This is a schematic diagram of a natural language description parsing module provided in an embodiment of this application;

[0051] Figure 4 This is a schematic diagram of an adaptive control module provided in an embodiment of this application;

[0052] Figure 5 This is a schematic diagram of the structure of a personalized speech synthesis device based on scene awareness and natural language description provided in an embodiment of this application;

[0053] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0054] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0055] Deep learning-based speech synthesis models have achieved significant improvements in speech quality, fluency, and timbre fidelity. For example, the speech synthesis model WaveNet generates natural speech waveforms through deep convolutional neural networks; while the speech synthesis models Tacotron and FastSpeech directly generate Mel spectrograms from text through an end-to-end neural network architecture and combine them with models such as WaveNet to generate high-quality speech.

[0056] To enhance the expressive power of speech, emotion-specific text-to-speech (TTS) has emerged. Emotion-based speech synthesis methods control the emotional features of speech (e.g., joy, anger, sorrow, and happiness) by analyzing the emotional information in the text. While these methods can adjust features such as pitch, speech rate, and stress to express emotion to some extent, they largely rely on emotion tags or manual annotation. Furthermore, in practical applications, the accuracy and diversity of emotional expression remain significantly limited.

[0057] Therefore, the speech synthesis methods in related technologies have the following limitations: 1) Insufficient scene adaptability: Existing speech synthesis methods usually use a single speech model, making it difficult to adjust the scene sound according to different application scenarios, resulting in synthesized speech that does not match the scene atmosphere and lacks a sense of immersion. 2) Lack of natural language description ability: Existing speech synthesis methods mainly rely on label conditions to generate speech, making it difficult to understand and utilize natural language description information (such as "cheerful tone", "deep male voice", etc.) to guide the speech synthesis process, thus limiting the flexibility and controllability of speech synthesis.

[0058] To overcome the limitations of related technologies, this application provides a personalized speech synthesis method based on scene awareness and natural language description. This method can generate scene sounds according to the generated content, solve the problem of synthesized speech not matching the scene atmosphere, and can understand and utilize natural language description information to adjust the style of synthesized speech. Therefore, it can generate personalized, expressive and context-appropriate synthesized speech.

[0059] The personalized speech synthesis method based on scene awareness and natural language description provided in this application will be described below with reference to the accompanying drawings, through Section 1.1 Personalized Speech Synthesis Method, Section 1.2 Scene Awareness, Section 1.3 Natural Language Description Parsing, and Section 1.4 Adaptive Adjustment.

[0060] 1.1 Personalized Speech Synthesis Methods:

[0061] Reference Figure 1 As shown, Figure 1 This is a flowchart illustrating the steps of a personalized speech synthesis method based on scene awareness and natural language description provided in an embodiment of this application. Figure 1 As shown, the personalized speech synthesis method based on scene awareness and natural language description may include steps S110 to S160:

[0062] Step S110: Perform semantic analysis on the first text to obtain the first semantic features. The first text includes text content and scene description.

[0063] The text content in the first text refers to the content that needs to be synthesized into speech, which can be news, stories, dialogues, etc.; the scene description in the first text refers to the description of the emotional tone and occasion requirements, for example, the scene description can be a news broadcast scene description or a train station scene description.

[0064] The first semantic feature is used to characterize the semantic information of the first text. It is obtained by performing semantic analysis on the first text. In some embodiments, the first semantic feature can be obtained by performing semantic analysis on the first text based on a deep learning model.

[0065] Step S120: Based on the first semantic feature, a scene classification is generated to obtain a scene vector, wherein the scene vector represents the scene category.

[0066] Specifically, based on the semantic information represented by the first semantic feature, the corresponding scene category is identified, and a scene vector is generated according to the scene category. For example, if the first semantic feature identifies the scene as a news broadcast, a scene vector representing the news broadcast scene is generated; if the first semantic feature identifies the scene as a train station, a scene vector representing the train station scene is generated.

[0067] In some embodiments, the first semantic feature can be input into a pre-trained scene classification model, and the pre-trained scene classification model generates scene vectors based on the input first semantic feature.

[0068] Step S130: Generate scene audio features based on the scene vector, wherein the scene audio features include detailed features of the application scene and audio features related to the scene.

[0069] In this embodiment of the application, the scene sound feature is a scene sound feature that is adapted to the current target scene (i.e., the scene corresponding to the scene description in the first text); wherein, the detailed features of the application scene can be news broadcasts, emotional dialogue details, etc.; the audio features related to the scene can be voice features such as volume, timbre, and background noise.

[0070] Different scene vectors correspond to different scene sound features. For example, for a news broadcast scene, a more formal and quiet scene sound feature is generated; while for scenes such as train stations, a noisier scene sound feature is selected.

[0071] Step S140: Perform semantic analysis on the second text to obtain second semantic features. The second text includes at least a natural language description of the speech style.

[0072] In this embodiment, the desired speech synthesis style is represented by natural language description. For example, the natural language description of the speech style could be "cheerful tone" or "deep male voice," etc. The second semantic feature is used to represent the semantic information of the second text. The second semantic feature is obtained by performing semantic analysis on the second text.

[0073] In some embodiments, semantic analysis of the second text can be performed based on a deep learning model to obtain the second semantic features corresponding to the second text.

[0074] Step S150: Based on the second semantic feature, perform sentiment category prediction and style feature extraction to obtain style control parameters, which include: pitch, speech rate, tone and volume.

[0075] Among them, emotion category refers to emotions such as joy, sadness, and anger; style features refer to features such as tone, speech rate, and intonation. Style control parameters are generated based on personalized speech adjustment needs such as emotion category and intonation features (e.g., joy, low voice, fast speech rate).

[0076] In some embodiments, the sentiment category prediction and style feature extraction of the second semantic features can be performed using models such as sentiment analysis models, tone recognition models, and style transfer models to obtain style control parameters.

[0077] Step S160: Adjust the synthesized speech parameters of the text content according to the scene sound features and the style control parameters to obtain synthesized speech.

[0078] In this embodiment, the scene sound features include detailed features of the application scene and audio features related to the scene. The style control parameters include parameters such as pitch, speech rate, tone and volume. Therefore, by using scene sound features and style control parameters, the style and speech characteristics of speech synthesis can be dynamically adjusted to achieve dynamic adjustment of pitch, speech rate, tone and other features of speech synthesis according to user needs, scene changes and emotional feedback, so as to ensure the personalization and context adaptability of speech output.

[0079] Through the above implementation process, the first text includes text content and scene description. By processing the first text, scene sound features containing detailed features of the application scenario and scene-related audio features are generated, which can recognize and adapt to various application scenarios (e.g., news, train stations, dialogues, etc.). Therefore, this method adjusts the style and atmosphere of the synthesized speech according to the scene description of the first text, which not only improves the adaptability of the synthesized speech and generates speech that is more in line with actual needs in various scenarios, but also enhances the immersion and realism of the synthesized speech. The second text includes at least a natural language description of the speech style. By processing the second text, style control parameters are obtained, which can accurately parse and extract the natural language description of the speech style, and then adjust the emotional category, intonation features, pitch, speech rate and other parameters of the synthesized speech. Therefore, this method can not only understand rich language descriptions, but also flexibly adjust the speech output to meet the personalized speech needs of users, significantly improving the flexibility and controllability of speech synthesis. Thus, by combining scene information and natural language description, this method can generate personalized, expressive and context-appropriate synthesized speech.

[0080] 1.2 Scene Awareness:

[0081] In conjunction with the above embodiments, in one embodiment, this application also provides a personalized speech synthesis method based on scene awareness and natural language description. In this method, the "semantic analysis of the first text to obtain the first semantic feature" in step S110 above may specifically include sub-steps S110-1 to S110-2:

[0082] Step S110-1: Perform word segmentation and annotation on the first text using a natural language processing model to obtain the first text after word segmentation and annotation.

[0083] Step S120-2: Semantically encode the first text that has been segmented and annotated by a text encoder to obtain the first semantic feature.

[0084] In this embodiment, segmenting and labeling the first text means dividing the first text into the smallest semantic units (e.g., words, subwords, or characters) and labeling each smallest semantic unit (e.g., part-of-speech tagging, named entity recognition, etc.). The natural language processing model can be a model such as BERT (Bidirectional Encoder Representations from Transformers) or BiLSTM-CRF (Bidirectional Long Short-Term Memory-Conditional Random Field), and the text encoder refers to a deep learning-based BERT or Transformer model.

[0085] In this way, semantic analysis of the first text is performed through a natural language processing model and a text encoder to obtain the first semantic features corresponding to the first text. This allows for subsequent scene perception based on the first semantic features, and the generation of scene sound features that are adapted to the scene, thereby improving the adaptability of the synthesized speech and enhancing the immersion and realism of the speech synthesis.

[0086] Furthermore, the step S130 above, "generating scene sound features based on the scene vector," may specifically include sub-steps S130-1 to S130-2:

[0087] Step S130-1: Match the scene vector with each scene sound template in the scene template library to obtain the target scene sound template that matches the scene vector.

[0088] Step S130-2: Generate scene sound features based on the parameters of the target scene sound template and the speech synthesis model.

[0089] In this embodiment, the scene template library contains multiple preset scene sound templates, each representing the scene sound features under different scenes. Matching the scene vector with each scene sound template in the scene template library can be achieved by calculating the similarity between the scene vector and each scene sound template (e.g., calculating the cosine similarity between the scene vector and the scene sound template), and the scene sound template corresponding to the maximum similarity is determined as the target scene sound template that matches the scene vector.

[0090] The parameters of the speech synthesis model include pitch, speech rate, tone and volume. After obtaining the target scene sound template that matches the scene vector, the target scene sound template is combined with the parameters of the speech synthesis model to obtain a scene sound feature that is adapted to the target scene (i.e. the scene described in the first article).

[0091] Thus, this method can identify and adapt to various application scenarios (e.g., news, train stations, dialogues, etc.), and adjust the style and atmosphere of the synthesized speech according to the generated scene sound features, thereby improving the adaptability of the synthesized speech, generating speech that is more in line with actual needs in various scenarios, and enhancing the sense of immersion and realism of speech synthesis.

[0092] Reference Figure 2 As shown, the methods described in steps S110 to S130 above can be implemented based on a scene perception module. Figure 2 This is a schematic diagram of a scene perception module provided in an embodiment of this application. The scene perception module includes: a natural language processing model, a text encoder, a scene classification model, and a scene template library.

[0093] Specifically, the process of generating scene sound features based on the first text is as follows: the first text is segmented and labeled using a natural language processing model to obtain segmented and labeled first text; the segmented and labeled first text is semantically encoded using a text encoder to obtain first semantic features; then, a scene classification model is used to classify and generate scene vectors based on the first semantic features; finally, the scene vectors are matched with various scene sound templates in the scene template library to obtain target scene sound templates that match the scene vectors; and scene sound features are generated based on the target scene sound templates and the parameters of the speech synthesis model.

[0094] In this way, the scene-aware module identifies the specific application scenario and generates speech features suitable for the current scenario, enabling speech synthesis to adjust the style of the synthesized speech according to the scenario of the input text. Therefore, this method can adjust the style and atmosphere of the synthesized speech based on the scenario description of the first text, which not only improves the adaptability of the synthesized speech and generates speech that is more in line with actual needs in various scenarios, but also enhances the immersion and realism of the speech synthesis.

[0095] 1.3 Natural Language Description Parsing:

[0096] In conjunction with the above embodiments, in one embodiment, this application also provides a personalized speech synthesis method based on scene awareness and natural language description. In this method, the second text further includes contextual information of the text content. Specifically, the "semantic analysis of the second text to obtain second semantic features" in step S140 above may include: parsing and encoding the second text through a language parsing model to obtain second semantic features.

[0097] In this embodiment, to better understand the personalized speech style requirements, contextual information of the text content is also included in the second text. The language parsing model can be a BERT or GPT (Generative Pre-trained Transformer) based model. The second text is input into the language parsing model, which decodes the input second text and outputs the second semantic features.

[0098] In this way, by parsing and encoding the second text through the language parsing model, the understanding of the second text can be achieved. Subsequently, based on the second semantic features obtained through the language parsing model, control parameters such as pitch, speech rate, and tone of the speech can be generated to achieve emotion control of the synthesized speech.

[0099] Furthermore, the step S150 above, "predicting the sentiment category and extracting style features based on the second semantic feature to obtain style control parameters," may specifically include sub-steps S150-1 to S150-4:

[0100] Step S150-1: Predict the sentiment category of the second semantic feature using a sentiment analysis model to obtain the first voice control parameters, which include the first pitch, the first speech rate, and the volume.

[0101] The sentiment analysis model can be an LSTM-based model. The second semantic feature contains the semantic information of the second text. The second semantic feature is input into the sentiment analysis model, which predicts the sentiment category based on the input second semantic feature to obtain the first speech control parameters.

[0102] Specifically, the first voice control parameter is obtained by predicting the emotion category of the second semantic feature through the emotion analysis model, including: predicting the emotion category of the second semantic feature through the emotion analysis model to obtain a target emotion category label; and mapping the target emotion category label to the first voice parameter according to the mapping relationship between the emotion category label and the voice parameter.

[0103] In this embodiment of the application, different emotion category labels correspond to different speech parameters (e.g., pitch, speech rate, volume, etc.). The mapping relationship between emotion category labels and speech parameters is predetermined. After obtaining the target emotion category label through the emotion analysis model, the target emotion category label is mapped into the corresponding semantic parameter, i.e., the first speech parameter, based on the mapping relationship between the emotion category label and the speech parameter.

[0104] Step S150-2: Perform tone recognition on the second semantic feature using a tone recognition model to obtain the second speech control parameters, which include tone.

[0105] The tone recognition model can be based on a CNN (Convolutional Neural Network) or a Transformer model. The second semantic feature contains the semantic information of the second text. The second semantic feature is input into the tone recognition model, and the tone recognition model identifies the corresponding tone based on the input second semantic feature to generate the corresponding control parameters.

[0106] Step S150-3: Perform style transfer on the second semantic feature using a style transfer model to obtain the third speech control parameters, which include the second pitch and the second speech rate.

[0107] The style transfer model can be based on Transformer or GAN (Generative Adversarial Network), and the second semantic feature is input into the style transfer model to convert the text description into specific acoustic features, i.e., the third speech control parameter.

[0108] Specifically, the third speech control parameter is obtained by performing style transfer on the second semantic feature through a style transfer model, including: processing the second semantic feature through the style transfer model to obtain a style vector; matching the style vector with each style template in the style library to obtain a target style template that matches the style vector; and using the speech control parameter corresponding to the target style template as the third speech control parameter.

[0109] In this embodiment, the style library contains multiple preset style templates, each representing the voice control parameters corresponding to a different style. Matching the style vector with each style template in the style library can be achieved by calculating the similarity between the style vector and each style template (e.g., calculating the cosine similarity between the style vector and the style template). The style template with the highest similarity is determined as the target style template matching the style vector. Finally, the voice control parameters corresponding to the target style template are used as the third voice control parameter.

[0110] Step S150-4: Integrate the first voice control parameter, the second voice control parameter, and the third voice control parameter to obtain the style control parameter.

[0111] In this embodiment, the first voice control parameters corresponding to the emotion tag, the second voice control parameters corresponding to the tone, and the third voice control parameters corresponding to the style are integrated to obtain a final set of style control parameters. This converts natural language descriptions into corresponding style control parameters, which guide subsequent speech synthesis, ensuring that the voice style highly matches user needs.

[0112] It is understandable that the aforementioned sentiment analysis model, tone recognition model, and style transfer model can be individual neural network models or different sub-models within a single neural network model.

[0113] Reference Figure 3 As shown, the methods in steps S140 to S150 above can be implemented based on a natural language description parsing module. Figure 3 This is a schematic diagram of a natural language description parsing module provided in an embodiment of this application. The natural language description parsing module includes: a language parsing model, a sentiment analysis model, a tone recognition model, a style transfer model, and a style library.

[0114] Specifically, the process of generating style control parameters based on the second text is as follows: the second text is parsed and encoded using a language parsing model to obtain second semantic features; then, the sentiment analysis model is used to predict the sentiment category of the second semantic features to obtain a target sentiment category label; based on the mapping relationship between the sentiment category label and the speech parameters, the target sentiment category label is mapped to a first speech parameter; the tone recognition model is used to identify the tone of the second semantic features to obtain a second speech control parameter; the second semantic features are processed using a style transfer model to obtain a style vector; the style vector is matched with various style templates in the style library to obtain a target style template that matches the style vector; the speech control parameter corresponding to the target style template is used as a third speech control parameter; finally, the first, second, and third speech control parameters are integrated to obtain the style control parameter.

[0115] Thus, the natural language description parsing module can accurately parse and extract natural language descriptions of speech style, thereby adjusting parameters such as emotion category, intonation, pitch, and speech rate of the synthesized speech. This module not only understands rich language descriptions but also flexibly adjusts the speech output to meet users' personalized speech needs, significantly improving the flexibility and controllability of the speech synthesis system.

[0116] 1.4 Adaptive Adjustment:

[0117] In conjunction with the above embodiments, in one embodiment, this application also provides a personalized speech synthesis method based on scene awareness and natural language description. In this method, the step S160 above, "adjusting the synthesized speech parameters of the text content according to the scene sound features and the style control parameters to obtain synthesized speech," may specifically include sub-steps S160-1 to S160-2:

[0118] Step S160-1: Fuse the scene sound features and the style control parameters to obtain comprehensive style adjustment parameters.

[0119] Step S160-2: Based on the integrated style adjustment parameters, dynamically adjust the synthesized speech parameters of the text content to obtain synthesized speech.

[0120] In this embodiment of the application, scene sound features and style control parameters are fused based on the fusion mechanism to obtain comprehensive style adjustment parameters, which are control parameters that include scene information and speech style.

[0121] Among them, the fusion mechanism refers to the process of effectively combining features from different sources or modalities (e.g., scene sound features and style control parameters) through specific strategies or mathematical models to generate unified comprehensive style adjustment parameters. The fusion mechanism can be a weighted linear fusion (i.e., linearly superimposing scene sound features and style parameters after assigning weights), a neural network fusion (i.e., using fully connected layers, LSTM or Transformer to encode feature interactions), an attention mechanism (i.e., dynamically assigning feature importance through attention weights), and other methods.

[0122] The synthesized speech parameters of the text content can be dynamically adjusted based on the comprehensive style adjustment parameters. This can be achieved using a deep learning model (e.g., a temporal model based on LSTM or GRU) to dynamically adjust the synthesized speech parameters of the text content. For example, when the current scenario is a "news broadcast" scenario, the clarity and formality of the synthesized speech will be enhanced; when the second text contains the emotion of "sadness", the tone will be lowered and the speech rate will be reduced, so that the generated speech sounds more in line with the emotional tone.

[0123] Reference Figure 4 As shown, the method of step S160 above can be implemented based on an adaptive control module. Figure 4 This is a schematic diagram of an adaptive control module provided in an embodiment of this application. The adaptive control module includes a fusion mechanism and a parameter adjustment model.

[0124] Specifically, the process of adjusting the synthesized speech parameters based on scene sound features and style control parameters is as follows: the scene sound features from the scene perception module and the style control parameters from the natural language description parsing module are fused together based on the fusion mechanism to obtain a comprehensive style adjustment parameter; then, the synthesized speech parameters of the text content are dynamically adjusted through the parameter adjustment model to obtain the synthesized speech.

[0125] In this way, by combining the outputs of the adaptive control module with the scene perception module and the natural language description and parsing module, the style and speech characteristics of the speech synthesis are intelligently adjusted. This module can dynamically adjust the pitch, speech rate, tone and other features of the speech synthesis according to user needs, scene changes and emotional feedback, so as to ensure the personalization and context adaptability of the speech output.

[0126] Based on the same technical concept, this application also provides a personalized speech synthesis device based on scene awareness and natural language description, referring to... Figure 5 As shown, Figure 5 This is a schematic diagram of a personalized speech synthesis device based on scene awareness and natural language description provided in an embodiment of this application. The device includes:

[0127] The first analysis module 510 is used to perform semantic analysis on the first text to obtain the first semantic features. The first text includes text content and scene description.

[0128] The first generation module 520 is used to generate a scene vector by performing scene classification based on the first semantic feature, wherein the scene vector represents the scene category.

[0129] The second generation module 530 is used to generate scene sound features based on the scene vector, wherein the scene sound features include detailed features of the application scene and audio features related to the scene;

[0130] The second analysis module 540 is used to perform semantic analysis on the second text to obtain second semantic features, wherein the second text includes at least a natural language description of speech style;

[0131] The first extraction module 550 is used to predict the sentiment category and extract style features based on the second semantic features to obtain style control parameters, which include: pitch, speech rate, tone and volume.

[0132] The first adjustment module 560 is used to adjust the synthesized speech parameters of the text content according to the scene sound features and the style control parameters to obtain synthesized speech.

[0133] In an optional embodiment, the first adjustment module includes:

[0134] The parameter fusion module is used to fuse the scene sound features and the style control parameters to obtain comprehensive style adjustment parameters;

[0135] The parameter adjustment module is used to dynamically adjust the synthesized speech parameters of the text content according to the comprehensive style adjustment parameters to obtain synthesized speech.

[0136] In one optional embodiment, the first generation module includes:

[0137] The scene sound template matching module is used to match the scene vector with each scene sound template in the scene template library to obtain a target scene sound template that matches the scene vector.

[0138] The scene sound feature generation module is used to generate scene sound features based on the target scene sound template and the parameters of the speech synthesis model.

[0139] In one optional embodiment, the first analysis module includes:

[0140] The text processing module is used to perform word segmentation and annotation on the first text using a natural language processing model to obtain the first text after word segmentation and annotation.

[0141] The semantic encoding module is used to perform semantic encoding on the first text that has been segmented and annotated by a text encoder to obtain the first semantic features.

[0142] In an optional embodiment, the first extraction module includes:

[0143] The emotion category prediction module is used to predict the emotion category of the second semantic feature through an emotion analysis model to obtain the first voice control parameters, which include the first pitch, the first speech rate and the volume.

[0144] The tone recognition module is used to perform tone recognition on the second semantic feature through a tone recognition model to obtain the second voice control parameters, the second voice control parameters including tone.

[0145] The style transfer module is used to perform style transfer on the second semantic feature through a style transfer model to obtain the third speech control parameters, which include the second pitch and the second speech rate.

[0146] The parameter integration module is used to integrate the first voice control parameter, the second voice control parameter and the third voice control parameter to obtain the style control parameter.

[0147] In an optional embodiment, the emotion category prediction module is specifically used to: predict the emotion category of the second semantic feature through the emotion analysis model to obtain a target emotion category label; and map the target emotion category label to the first speech parameter according to the mapping relationship between the emotion category label and the speech parameter.

[0148] In an optional embodiment, the style conversion module is specifically used to: process the second semantic feature through the style conversion model to obtain a style vector; match the style vector with each style template in the style library to obtain a target style template that matches the style vector; and use the speech control parameters corresponding to the target style template as the third speech control parameters.

[0149] In an optional embodiment, the second text further includes contextual information of the text content, and the second analysis module includes:

[0150] The text parsing module is used to parse and encode the second text using a language parsing model to obtain the second semantic features.

[0151] This application also provides an electronic device, see embodiments thereof. Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 6 As shown, the electronic device 600 includes a memory 610 and a processor 620. The memory 610 and the processor 620 are connected via a bus for communication. The memory 610 stores a computer program that can run on the processor 620 to implement the steps of the personalized speech synthesis method based on scene awareness and natural language description described in the embodiments of this application.

[0152] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the personalized speech synthesis method based on scene awareness and natural language description described in this application.

[0153] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the personalized speech synthesis method based on scene awareness and natural language description described in this application.

[0154] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0155] This application describes embodiments of methods and apparatus according to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0156] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0157] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0158] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0159] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0160] The above provides a detailed description of a personalized speech synthesis method, apparatus, and device based on scene awareness and natural language description provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for personalized speech synthesis based on scene awareness and natural language description, characterized in that, The method comprises: performing semantic analysis on the first text to obtain first semantic features, wherein the first text comprises text content and scene description, and the scene description is a natural language description about emotional color and occasion demand; generating a scene vector according to the first semantic features, wherein the scene vector represents a scene category; generating scene audio features according to the scene vector, wherein the scene audio features comprise detailed features of an application scene and audio features related to the scene; performing semantic analysis on the second text to obtain second semantic features, wherein the second text comprises at least a natural language description of voice style; performing emotion category prediction and style feature extraction according to the second semantic features to obtain style control parameters, wherein the style control parameters comprise pitch, speech rate, tone and volume; adjusting the synthesized speech parameters of the text content according to the scene audio features and the style control parameters to obtain synthesized speech, comprising: fusing the scene audio features and the style control parameters to obtain comprehensive style adjustment parameters; and dynamically adjusting the synthesized speech parameters of the text content according to the comprehensive style adjustment parameters to obtain synthesized speech.

2. The method of claim 1, wherein, Generating scene audio features according to the scene vector comprises: matching the scene vector with each scene audio template in a scene template library to obtain a target scene audio template matched with the scene vector; generating scene audio features according to parameters of the target scene audio template and a speech synthesis model.

3. The method of claim 2, wherein, Performing semantic analysis on the first text to obtain first semantic features comprises: performing word segmentation and annotation processing on the first text by a natural language processing model to obtain the first text after word segmentation and annotation; performing semantic encoding on the first text after word segmentation and annotation by a text encoder to obtain the first semantic features.

4. The method of claim 1, wherein, Performing emotion category prediction and style feature extraction according to the second semantic features to obtain style control parameters comprises: performing emotion category prediction on the second semantic features by an emotion analysis model to obtain first voice control parameters, wherein the first voice control parameters comprise first pitch, first speech rate and volume; performing tone recognition on the second semantic features by a tone recognition model to obtain second voice control parameters, wherein the second voice control parameters comprise tone; performing style conversion on the second semantic features by a style conversion model to obtain third voice control parameters, wherein the third voice control parameters comprise second pitch and second speech rate; integrating the first voice control parameters, the second voice control parameters and the third voice control parameters to obtain the style control parameters.

5. The method of claim 4, wherein, Performing emotion category prediction on the second semantic features by an emotion analysis model to obtain first voice control parameters comprises: performing emotion category prediction on the second semantic features by the emotion analysis model to obtain a target emotion category label; mapping the target emotion category label to the first voice parameters according to a mapping relationship between emotion category labels and voice parameters.

6. The method of claim 4, wherein, Performing style conversion on the second semantic features by a style conversion model to obtain third voice control parameters comprises: The second semantic feature is processed by the style transfer model to obtain a style vector; The style vector is matched with each style template in the style library to obtain the target style template that matches the style vector; The voice control parameters corresponding to the target style template are used as the third voice control parameters.

7. The method of claim 4, wherein, The second text also includes contextual information about the text content; semantic analysis is performed on the second text to obtain second semantic features, including: The second text is parsed and encoded using a language parsing model to obtain the second semantic features.

8. A personalized speech synthesis device based on scene awareness and natural language description, characterized in that, The device includes: The first analysis module is used to perform semantic analysis on the first text to obtain the first semantic features. The first text includes text content and scene description, and the scene description is a natural language description about emotional tone and occasion requirements. The first generation module is used to perform scene classification and generation based on the first semantic feature to obtain a scene vector, wherein the scene vector represents the scene category. The second generation module is used to generate scene audio features based on the scene vector, wherein the scene audio features include detailed features of the application scene and scene-related audio features; The second analysis module is used to perform semantic analysis on the second text to obtain second semantic features, wherein the second text includes at least a natural language description of the speech style; The first extraction module is used to predict the sentiment category and extract style features based on the second semantic features to obtain style control parameters, which include: pitch, speech rate, tone and volume. The first adjustment module is used to adjust the synthesized speech parameters of the text content according to the scene sound features and the style control parameters to obtain synthesized speech, including: fusing the scene sound features and the style control parameters to obtain comprehensive style adjustment parameters; and dynamically adjusting the synthesized speech parameters of the text content according to the comprehensive style adjustment parameters to obtain synthesized speech.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the personalized speech synthesis method based on scene awareness and natural language description as described in any one of claims 1-7.

Images