Speech synthesis method and device, storage medium and electronic equipment
By identifying character dialogue in text and determining intonation and speech speed, and voice synthesis technology based on object voice lines, the problem of inability to distinguish character dialogue in the existing technology is solved, and a more realistic dialogue scene restoration and user experience improvement is achieved.
Patent Information
- Application Number
- CN202411863482.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-17
- Publication Date
- 2025-04-04
AI Technical Summary
Existing speech synthesis technology cannot recognize role relationships in text, resulting in the inability to distinguish between conversations between different characters and the inability to truly reflect the dialogue scenes in text and the interactions between characters.
By obtaining text describing the interaction behavior of multiple objects, the dialogue text of each object is identified, the intonation and speech speed are determined according to the context of the dialogue text, and the dialogue text is broadcasted using virtual digital people based on the basic voice lines of each object.
It realizes voice interaction between different characters, improves the expressiveness of the dialogue and the restoration of the scene, helps users better understand text scenes and character emotions, and improves user experience.
Smart Images
Figure CN119763541A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of financial technology, and more specifically, to a speech synthesis method, device, storage medium and electronic device. It should be noted that the speech synthesis method, device, storage medium and electronic device determined by the present invention can be used for speech synthesis in the field of financial technology, and can also be used for speech synthesis in any field other than the field of financial technology, and the application field of the present invention is not limited. Background Art
[0002] The technology for automatically generating speech corresponding to text has made great progress. With the development of deep learning and neural networks, speech synthesis technology has become more and more advanced. However, in the existing technology, text-to-speech technology cannot recognize the role relationship in the text, that is, it cannot distinguish the dialogue between the characters in the text, and can only use a unified voice to broadcast the same text content. Therefore, it is impossible to achieve voice interaction between different roles. For example, in meeting minutes, the words of each character are often closely intertwined. The existing technology may not be able to distinguish the difference between "Role A said to Role B" and "Role B responded to Role A". Instead, it regards the entire text as a continuous narrative and uses the same pronunciation style and voice to broadcast it. This obviously cannot truly reflect the dialogue scenes in the text and the interaction between the characters.
[0003] With regard to the technical problem of incomplete transmission of scene and dialogue information due to a single voice in speech synthesis in related technologies, no effective solution has been proposed so far. Summary of the invention
[0004] The embodiments of the present invention provide a speech synthesis method, device, storage medium and electronic device to at least solve the technical problem of incomplete transmission of scene and dialogue information due to a single speech synthesis voice.
[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a speech synthesis method is provided. The method comprises: obtaining a text to be processed, wherein the text to be processed includes text describing the interaction behavior between N objects, where N is greater than 1; in the text to be processed, identifying the respective dialogue texts of the N objects, and determining M dialogue texts, wherein M is greater than 1; according to the respective contexts of the M dialogue texts, respectively determining the respective intonations of the M dialogue texts and / or the respective speech speeds of the M dialogue texts; according to the respective intonations of the M dialogue texts and / or the respective speech speeds of the M dialogue texts, using N virtual digital people to broadcast the M dialogue texts based on the respective basic voice lines of the N objects.
[0006] Optionally, the intonation of each of the M dialogue texts and / or the speaking speed of each of the M dialogue texts are determined respectively according to their respective contexts, including: determining the emotion type of each of the M dialogue texts according to their respective contexts; determining the intonation of each of the M dialogue texts according to the emotion type of each of the M dialogue texts and the correspondence between a predetermined emotion type and a intonation strategy, wherein the intonation strategy represents an amplitude for adjusting the intonation based on a predetermined intonation; and / or determining the speaking speed of each of the M dialogue texts according to the emotion type of each of the M dialogue texts and the correspondence between a predetermined emotion type and a speaking speed strategy, wherein the speaking speed strategy represents an amplitude for adjusting the speaking speed based on a predetermined speaking speed.
[0007] Optionally, the emotion type of each of the M dialogue texts is determined according to their respective contexts, including: inputting the respective contexts of the M dialogue texts into a pre-trained neural network model, and the neural network model outputs the emotion type of each of the M dialogue texts, wherein the neural network model is obtained by training an original neural network model with training samples, and the training samples include sample dialogue texts and contexts, and sample emotion types corresponding to the sample dialogue texts.
[0008] Optionally, according to the respective intonations of the M dialogue texts and / or the respective speech speeds of the M dialogue texts, based on the respective basic voice lines of the N objects, N virtual digital humans are respectively used to broadcast the M dialogue texts, including: extracting features of the text to be processed to determine the respective voice line features of the N objects; determining the respective basic voice lines of the N objects according to the respective voice line features of the N objects; broadcasting the M dialogue texts respectively using the respective basic voice lines of the N objects to obtain initial audio of the M dialogue texts; adjusting the initial audio of the M dialogue texts according to the respective intonations of the M dialogue texts and / or the respective speech speeds of the M dialogue texts to obtain broadcast audio of the M dialogue texts; based on the sequence of the M dialogue texts in the text to be processed, N virtual digital humans are respectively used to play the respective broadcast audio of the M dialogue texts.
[0009] Optionally, the initial audio of the M segments of dialogue text is adjusted according to the respective intonations of the M segments of dialogue text and / or the respective speaking speeds of the M segments of dialogue text to obtain the respective broadcast audios of the M segments of dialogue text, including: adjusting the audio frequency of the initial audio of the M segments of dialogue text according to the respective intonations of the M segments of dialogue text to obtain the respective broadcast audios of the M segments of dialogue text; and / or adjusting the playback speed of the initial audio of the M segments of dialogue text according to the respective speaking speeds of the M segments of dialogue text to obtain the respective broadcast audios of the M segments of dialogue text.
[0010] Optionally, feature extraction is performed on the text to be processed to determine the voice line features of each of N objects, including: using natural language processing technology to determine the description texts of each of the N objects in the text to be processed, wherein the description texts of each of the N objects are used to describe the features of the corresponding object; and determining the voice line features of each of the N objects based on the description texts of each of the N objects.
[0011] Optionally, feature extraction is performed on the text to be processed to determine the voice features of each of the N objects, including: feature extraction is performed on the text to be processed to determine the text type of the text to be processed, wherein the text type is used to describe the genre of the text to be processed; and according to the text type of the text to be processed, the voice features of each of the N objects are determined.
[0012] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a speech synthesis device is provided. The device includes: a text acquisition module, used to acquire the text to be processed, wherein the text to be processed includes text describing the interactive behavior between N objects, where N is greater than 1; a dialogue recognition module, used to identify the dialogue texts of the N objects in the text to be processed, and determine M dialogue texts, wherein M is greater than 1; a speech rate and intonation determination module, used to determine the intonation of the M dialogue texts and / or the speech speed of the M dialogue texts according to the context of the M dialogue texts; a voice line determination module, used to use N virtual digital people to broadcast the M dialogue texts according to the intonation of the M dialogue texts and / or the speech speed of the M dialogue texts, based on the basic voice lines of the N objects.
[0013] Optionally, the speech rate and intonation determination module includes: a first determination unit, used to determine the emotion type of each of the M dialogue texts according to their respective contexts; a second determination unit, used to determine the intonation of each of the M dialogue texts according to the emotion type of each of the M dialogue texts and the correspondence between the predetermined emotion type and the intonation strategy, wherein the intonation strategy represents the amplitude of adjusting the intonation based on a predetermined intonation; and / or, determining the speech speed of each of the M dialogue texts according to the emotion type of each of the M dialogue texts and the correspondence between the predetermined emotion type and the speech rate strategy, wherein the speech rate strategy represents the amplitude of adjusting the speech speed based on a predetermined speech speed.
[0014] Optionally, the first determination unit includes: an input-output unit, used to input the contexts of each of the M segments of dialogue text into a pre-trained neural network model, and the neural network model outputs the emotion type of each of the M segments of dialogue text, wherein the neural network model is obtained by training the original neural network model with training samples, and the training samples include sample dialogue text and context, and sample emotion type corresponding to the sample dialogue text.
[0015] Optionally, the voice line determination module includes a third determination unit, which is used to extract features of the text to be processed and determine the voice line features of N objects; a fourth determination unit, which is used to determine the basic voice lines of the N objects according to the voice line features of the N objects; a first broadcasting unit, which is used to broadcast M segments of dialogue text respectively using the basic voice lines of the N objects to obtain initial audio of the M segments of dialogue text; a first adjustment unit, which is used to adjust the initial audio of the M segments of dialogue text according to the intonation of the M segments of dialogue text and / or the speaking speed of the M segments of dialogue text to obtain the broadcast audio of the M segments of dialogue text; and a first playback unit, which is used to play the broadcast audio of the M segments of dialogue text respectively using N virtual digital humans based on the sequence of the M segments of dialogue text in the text to be processed.
[0016] Optionally, the first adjustment unit includes: a second adjustment unit, used to adjust the audio frequency of the initial audio of the M segments of dialogue text according to the respective intonations of the M segments of dialogue text, so as to obtain the respective broadcast audio of the M segments of dialogue text; and / or, to adjust the playback speed of the initial audio of the M segments of dialogue text according to the respective speaking speeds of the M segments of dialogue text, so as to obtain the respective broadcast audio of the M segments of dialogue text.
[0017] Optionally, the third determination unit includes: a fifth determination unit, used to use natural language processing technology to determine the description texts of N objects in the text to be processed, wherein the description texts of the N objects are used to describe the characteristics of the corresponding objects; and a sixth determination unit, used to determine the voice line characteristics of the N objects based on the description texts of the N objects.
[0018] Optionally, the third determination unit includes a seventh determination unit, which is used to extract features from the text to be processed and determine the text type of the text to be processed, wherein the text type is used to describe the genre of the text to be processed; and an eighth determination unit, which is used to determine the voice line features of each of the N objects according to the text type of the text to be processed.
[0019] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a non-volatile storage medium is provided, the non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located controls any one of the above-mentioned speech synthesis methods.
[0020] In order to achieve the above-mentioned purpose, according to another aspect of the present application, an electronic device is provided, comprising one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement any one of the above-mentioned speech synthesis methods.
[0021] In order to achieve the above object, according to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, any of the above-mentioned speech synthesis is realized.
[0022] Through the present application, the following steps are adopted: obtaining a text to be processed, wherein the text to be processed includes text describing the interactive behavior between N objects, where N is greater than 1; in the text to be processed, identifying the respective dialogue texts of the N objects, and determining M dialogue texts, wherein M is greater than 1; determining the respective intonations of the M dialogue texts and / or the respective speech speeds of the M dialogue texts according to the respective contexts of the M dialogue texts; based on the respective intonations of the M dialogue texts and / or the respective speech speeds of the M dialogue texts, using N virtual digital humans to broadcast the M dialogue texts based on the respective basic voice lines of the N objects, mainly determining the speech speed and intonation of the dialogue texts in combination with the context to broadcast the dialogue texts, thereby achieving the purpose of improving the expressiveness of the dialogue and the degree of scene restoration, solving the technical problem of incomplete transmission of scene and dialogue information due to the single voice line of speech synthesis in the related technology, and thus achieving the technical effect of helping users understand text scenes and character emotions and improving user experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0024] Figure 1 is a flow chart of a speech synthesis method provided according to an embodiment of the present application;
[0025] Figure 2 is a schematic diagram of a speech synthesis device provided according to an embodiment of the present application;
[0026] Figure 3 It is a structural schematic diagram of an electronic device for performing a speech synthesis method provided according to an embodiment of the present invention. DETAILED DESCRIPTION
[0027] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0028] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0030] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data comply with the relevant laws, regulations and standards of the relevant regions. For example, an interface is set up between this system and the relevant users or organizations. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.
[0031] The present invention is described below in conjunction with preferred implementation steps. Figure 1 is a flow chart of a speech synthesis method provided according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:
[0032] Step S101: obtaining text to be processed, wherein the text to be processed includes text describing interaction behaviors between N objects, where N is greater than 1.
[0033] In this step, the text to be processed can be obtained from multiple channels, such as input or upload by the user, or the system automatically captures it from a preset data source, such as a web page, social media, email, e-book, online novel, etc. Objects in the text to be processed can include character objects and personified objects, such as characters in novels, personified animals in fairy tales, etc.
[0034] Step S102: In the text to be processed, the conversation texts of N objects are identified, and M conversation texts are determined, where M is greater than 1.
[0035] In this step, in the case of a determined text type, the number of characters (voice lines) involved in the text and the text corresponding to each voice line are identified. Specifically, to automatically identify the number of characters involved in a conversation in a text, natural language processing and machine learning technologies can be used. You can first use text segmentation technology to segment the text into words and delete stop words and punctuation marks. Secondly, use named entity recognition technology to identify the names of characters in the text. Specifically, you can use a ready-made named entity recognition tool or train your own model. Secondly, use relationship extraction technology to analyze the dialogue relationship in the text and identify who is talking to whom. You can first define the possible relationship types, and then determine the dialogue relationship between entities based on the results of entity recognition in the previous model, using the features related to entity relationships in the context, such as the distance between entities, the words before and after the entity, the type information of the entity, etc. Finally, count the number of different characters involved in the text, that is, the number of characters in the conversation, and you can also determine the discourse corresponding to each character, that is, determine the dialogue text of each of the N objects. Among them, since each object may have more than one sentence of dialogue, the number M of recognized dialogue text segments is very likely to be greater than N, and at least equal to N.
[0036] Step S103, determining the intonation of the M dialogue texts and / or the speech speed of the M dialogue texts according to the contexts of the M dialogue texts.
[0037] In this step, natural language processing technology can be used to automatically determine the tone and speed of a sentence in a text. First, text analysis tools can be used to identify the emotional color and tone of a sentence, such as affirmative, negative, happy, sad, etc. Secondly, the speed can be determined according to the context of the sentence. For example, a fast speed can be used when describing a tense scene, and a slow speed can be used when describing a relaxing scene. Finally, the tone of the sentence can be determined in combination with the emotional color of the text. For example, a lower tone can be used to express an affirmative or sad emotional color, and a higher tone can be used to express a negative or happy emotional color. Through these methods, the tone and speed of a sentence can be automatically determined according to the context.
[0038] Step S104, according to the intonation of the M dialogue texts and / or the speech speed of the M dialogue texts, based on the basic voice lines of the N objects, the M dialogue texts are respectively read by N virtual digital humans.
[0039] In this step, the order between multiple sentences of text can be determined according to the order in which the words appear in the text and the context. Multiple sentences of speech are played in order, and the basic voice of the object itself is combined with the speed and intonation of the current dialogue text to realize the presentation of the predetermined text in the form of speech. After the speech synthesis is completed, the intonation, speed and volume of the broadcast voice can also be adjusted according to the emotional color to make it more realistic and vivid to express different emotions.
[0040] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0041] As an optional embodiment, the intonation of each of the M dialogue texts and / or the speaking speed of each of the M dialogue texts are determined according to their respective contexts, including: determining the emotion type of each of the M dialogue texts according to their respective contexts; determining the intonation of each of the M dialogue texts according to the emotion type of each of the M dialogue texts and the correspondence between the predetermined emotion type and the intonation strategy, wherein the intonation strategy represents the amplitude of adjusting the intonation based on a predetermined intonation; and / or determining the speaking speed of each of the M dialogue texts according to the emotion type of each of the M dialogue texts and the correspondence between the predetermined emotion type and the speaking speed strategy, wherein the speaking speed strategy represents the amplitude of adjusting the speaking speed based on a predetermined speaking speed.
[0042] Optionally, the emotional type of each dialogue text is first inferred based on the context of the dialogue text, such as happiness, sadness, anger, tension, fear, etc. After determining the emotional type of each dialogue text, the system will assign a specific tone and speed to each dialogue text according to the correspondence between the preset emotional type and the tone strategy and the speed strategy. For example, for dialogue texts expressing happy or excited emotions, the system will apply the tone strategy corresponding to the happy or excited emotions to increase the pitch, slightly increase the volume, and speed up the speech to convey a positive and energetic emotion; for dialogue texts expressing sad or melancholy emotions, the system will adopt the tone strategy corresponding to the sad or melancholy emotions to lower the pitch, slow down the speech, and reduce the volume to express a low and sad atmosphere; for dialogue texts expressing angry or nervous emotions, the system will adjust the tone and speed to make the tone fluctuate greatly, the speech speed is fast and the volume is loud to reflect the intensity and urgency of the emotions. This embodiment uses emotion-driven tone and speech speed adjustment to highlight the key content of the text, improve the efficiency and clarity of information delivery, greatly enhance the emotional expression ability and scene restoration of speech synthesis, and provide users with a more immersive listening experience.
[0043] As an optional embodiment, the emotion type of each of the M dialogue texts is determined according to their respective contexts, including: inputting the contexts of each of the M dialogue texts into a pre-trained neural network model, and the neural network model outputs the emotion type of each of the M dialogue texts, wherein the neural network model is obtained by training the original neural network model with training samples, and the training samples include sample dialogue texts and contexts, and sample emotion types corresponding to the sample dialogue texts.
[0044] Optionally, first, the context of the dialogue text is defined according to the text type and the complexity of the dialogue. In a dense dialogue, only the 1-2 sentences immediately adjacent are considered as the context; in a long narrative, a wider range of background information is considered, for example, all the paragraphs containing the keywords of the current dialogue text are considered as the context. Secondly, a neural network model needs to be built that can identify the emotion type from the input text and its context. This is usually a classification task, and the model can be based on deep learning, such as using a recurrent neural network (RNN), a long short-term memory network (LSTM), a gated recurrent unit (GRU) or a more advanced transformer (Transformer) architecture. The model uses a large number of training samples during the training phase, which include sample dialogue texts and their contexts, as well as the corresponding emotion type labels. The emotion type can be happy, sad, angry, surprised, calm, etc. Then, before the text is input into the model, data preprocessing is required, including word segmentation, removal of stop words, conversion to word vector representation, etc., so that the model can understand and process the text data. Finally, the contexts of the M dialogue texts are respectively input into the trained neural network model. The model will output the emotion type of each text based on the features and patterns it has learned. This process can be real-time or batch processing, depending on the application scenario and system architecture. Using the trained neural network model, the system can accurately identify the emotion type of each of the M dialogue texts. The accuracy of this emotion recognition is much higher than that of simple rule matching, and it can capture the emotional changes in the text more delicately, so that the emotion of the text can be expressed more realistically during speech synthesis, enhancing the appeal of the speech.
[0045] As an optional embodiment, according to the respective intonations of M dialogue texts and / or the respective speech speeds of the M dialogue texts, based on the respective basic voice lines of N objects, N virtual digital humans are respectively used to broadcast the M dialogue texts, including: extracting features of the text to be processed to determine the respective voice line features of the N objects; determining the respective basic voice lines of the N objects according to the respective voice line features of the N objects; broadcasting the M dialogue texts respectively using the respective basic voice lines of the N objects to obtain initial audio of the M dialogue texts; adjusting the initial audio of the M dialogue texts according to the respective intonations of the M dialogue texts and / or the respective speech speeds of the M dialogue texts to obtain the respective broadcast audio of the M dialogue texts; based on the sequence of the M dialogue texts in the text to be processed, using N virtual digital humans to play the respective broadcast audio of the M dialogue texts.
[0046] Optionally, first, feature extraction can be performed on the text to be processed to identify the voice features of N objects in the text. Feature extraction may include semantic analysis, dialogue structure analysis, sentiment analysis, etc. to determine the role features of each object in the dialogue, and finally determine the voice features of each of the N objects. Voice features may include timbre, pitch, etc. Based on the voice features of each of the N objects, the system will select the most suitable basic voice of the object from the preset virtual digital human voice library. The system uses the selected basic voice to broadcast the corresponding dialogue text respectively, and generates the initial audio of M segments of dialogue text.
[0047] Then, according to the intonation and speech speed of each of the M dialogue texts, the initial audio is adjusted to better match the emotional expression of the dialogue text. Specifically, it can include using audio processing technology to adjust the pitch, volume, speech speed and other parameters of the audio to reflect the emotional changes in the text. The system can automatically adjust the audio parameters based on the mapping relationship between the emotion type and the intonation strategy and the speech speed strategy, or it can provide a manual adjustment option so that the user can adjust the voice effect according to their preferences. Finally, based on the order of the M dialogue texts in the text to be processed, the system uses N virtual digital people to play the broadcast audio of each of the M dialogue texts to restore the entire dialogue scene. When playing the audio, the fluency and coherence of the dialogue can be taken into account to ensure that the transition between the audios is natural and the entire dialogue sounds like a real communication between people. Through the comprehensive application of the above embodiments, the system can generate high-quality, high-emotional expression and personalized matching synthetic speech, providing users with an immersive, natural and smooth auditory experience that highly restores the scene.
[0048] As an optional embodiment, the initial audio of the M segments of dialogue text is adjusted according to the respective intonations of the M segments of dialogue text and / or the respective speaking speeds of the M segments of dialogue text to obtain the respective broadcast audios of the M segments of dialogue text, including: adjusting the audio frequency of the initial audio of the M segments of dialogue text according to the respective intonations of the M segments of dialogue text to obtain the respective broadcast audios of the M segments of dialogue text; and / or adjusting the playback speed of the initial audio of the M segments of dialogue text according to the respective speaking speeds of the M segments of dialogue text to obtain the respective broadcast audios of the M segments of dialogue text.
[0049] Optionally, changes in intonation are usually reflected in the pitch and tone of the voice. In order to better express the intonation of different emotions, the system can adjust the frequency of the audio to change the pitch and tone. For example, if a text is analyzed as having a high intonation, the system will increase the audio frequency to make the tone rise and the timbre brighter to match this intonation feature; on the contrary, if the text has a flat or low tone, the system will reduce the audio frequency to make the tone more flat or low and the timbre softer. The speed of speech is reflected in the playback speed of the speech. The system adjusts the playback speed of the audio according to the recognized speech speed. For example, for a fast-speech text, the system will speed up the audio playback speed to make the speech sound more hurried; for a slow-speech text, the system will slow down the audio playback speed to make the speech sound more stable and steady. By adjusting the audio frequency to change the pitch and tone, and by adjusting the audio playback speed to match the speech speed of the text, the abstract description of speech features can be converted into specific audio parameters, thereby improving the user's auditory experience.
[0050] As an optional embodiment, feature extraction is performed on the text to be processed to determine the voice line features of N objects, including: using natural language processing technology to determine the description texts of the N objects in the text to be processed, wherein the description texts of the N objects are respectively used to describe the features of the corresponding objects; and determining the voice line features of the N objects according to the description texts of the N objects.
[0051] Optionally, first, natural language processing technology is used to conduct an in-depth analysis of the text, identify the N objects involved in the text, and extract the description text related to each object. The description text may include a direct description of the object, such as personality, age, gender, occupation, etc., or it may be the object characteristics inferred from the content of the conversation, such as speaking style, tone habits, emotional tendencies, etc. Then, the system will perform semantic analysis based on the description texts of each of the N objects to determine the voice characteristics of each object. Once the voice characteristics of each of the N objects are determined, the system can use speech synthesis technology to generate a personalized voice for each object, that is, adjust the parameters of the speech engine according to the object voice characteristics, such as pitch, timbre, speech speed, etc., to generate a synthesized voice that meets the object characteristics. Synthesizing speech based on object characteristics makes the character dialogue closer to its image setting in the text, improving the accuracy of speech expression and character recognition.
[0052] As an optional embodiment, feature extraction is performed on the text to be processed to determine the voice features of each of N objects, including: feature extraction is performed on the text to be processed to determine the text type of the text to be processed, wherein the text type is used to describe the genre of the text to be processed; and according to the text type of the text to be processed, the voice features of each of the N objects are determined.
[0053] Optionally, first, natural language processing technology can be used to extract features from the text to be processed to identify the genre or type of the text. The text type can include novels, poems, news reports, academic articles, script dialogues, etc. This step can be achieved by training a classification model, which automatically determines the type of the text based on the vocabulary, grammatical structure, emotional expression and other features of the text. Once the type of the text is determined, the system can more accurately determine the voice features of each of the N objects based on the characteristics of the text type. For example: for novels or script dialogues, the system can analyze the character descriptions in the text, as well as the context and emotions of the dialogue, and generate voice features for each character that match their personality and context; for poetry, since poetry usually has a specific rhythm and rhythm, the system can generate more rhythmic and musical voice features to enhance the rhythmic sense and emotional expression of the poetry; for news reports or academic articles, the system tends to generate clearer and more objective voice features to avoid excessive emotional colors and ensure accurate communication of information. The system can adjust the style of speech synthesis according to the text type, generate voice features that match the genre, make the synthesized speech more in line with the original style of the text, and enhance the realism and professionalism of the content. For text types with rich emotions or obvious rhythms, such as novels and poems, the system can generate emotional and musical voices to better convey the emotional fluctuations and rhythmic rhythms in the text, creating a more immersive listening experience for users.
[0054] As a specific embodiment, the speech synthesis method can be used in financial education and training, significantly improving the learning experience and educational effect. Using multi-voice technology, the speaking styles of experts in different financial fields can be simulated, such as gentle, aggressive, and stable, providing students with a diverse learning experience. This personalized approach helps to stimulate students' interest in learning, allowing them to concentrate more and understand more deeply when listening to lecturers with their preferred styles.
[0055] When explaining complex financial cases or simulating trading scenarios, the system can use different voices to play different roles, such as analysts, investors, bankers, etc., and demonstrate and explain through dialogue between roles, so that students can more intuitively understand the financial transaction and decision-making process, and enhance their sense of practice and situation; each role uses a different voice and speed and tone, which can help students better distinguish and understand the views and positions of different characters in the case, so as to deeply analyze the financial logic and strategy behind the case; through scenario-based interpretation, students seem to be in a real financial environment, which can enhance their sensitivity and adaptability to the financial market. On the education platform, virtual people can play the role of the host of the question-and-answer session, use multiple voices to interact with students, simulate the questions and answers in real classrooms, enhance students' sense of participation, and can also be used to simulate financial interview scenarios to improve students' answering skills. For financial students with visual impairments, multi-voice technology can convert course content into high-quality voices, distinguish between explainers and case roles through different voices, and help them learn financial knowledge without obstacles and enjoy equal educational opportunities.
[0056] The embodiment of the present application also provides a speech synthesis device. It should be noted that the speech synthesis device of the embodiment of the present application can be used to execute the speech synthesis method provided in the embodiment of the present application. The speech synthesis device provided in the embodiment of the present application is introduced below.
[0057] Figure 2 is a schematic diagram of a speech synthesis device according to an embodiment of the present application. Figure 2 As shown, the device includes: a text acquisition module 21, used to acquire the text to be processed, wherein the text to be processed includes text describing the interactive behavior between N objects, N is greater than 1; a dialogue recognition module 22, used to identify the dialogue texts of the N objects in the text to be processed, and determine M dialogue texts, wherein M is greater than 1; a speech rate and intonation determination module 23, used to determine the intonation of the M dialogue texts and / or the speech speed of the M dialogue texts according to the context of the M dialogue texts; a voice line determination module 24, used to use N virtual digital people to broadcast the M dialogue texts according to the intonation of the M dialogue texts and / or the speech speed of the M dialogue texts based on the basic voice lines of the N objects.
[0058] Optionally, the speech rate and intonation determination module includes: a first determination unit, used to determine the emotion type of each of the M dialogue texts according to their respective contexts; a second determination unit, used to determine the intonation of each of the M dialogue texts according to the emotion type of each of the M dialogue texts and the correspondence between the predetermined emotion type and the intonation strategy, wherein the intonation strategy represents the amplitude of adjusting the intonation based on a predetermined intonation; and / or, determining the speech speed of each of the M dialogue texts according to the emotion type of each of the M dialogue texts and the correspondence between the predetermined emotion type and the speech rate strategy, wherein the speech rate strategy represents the amplitude of adjusting the speech speed based on a predetermined speech speed.
[0059] Optionally, the first determination unit includes: an input-output unit, used to input the contexts of each of the M segments of dialogue text into a pre-trained neural network model, and the neural network model outputs the emotion type of each of the M segments of dialogue text, wherein the neural network model is obtained by training the original neural network model with training samples, and the training samples include sample dialogue text and context, and sample emotion type corresponding to the sample dialogue text.
[0060] Optionally, the voice line determination module includes a third determination unit, which is used to extract features of the text to be processed and determine the voice line features of N objects; a fourth determination unit, which is used to determine the basic voice lines of the N objects according to the voice line features of the N objects; a first broadcasting unit, which is used to broadcast M segments of dialogue text respectively using the basic voice lines of the N objects to obtain initial audio of the M segments of dialogue text; a first adjustment unit, which is used to adjust the initial audio of the M segments of dialogue text according to the intonation of the M segments of dialogue text and / or the speaking speed of the M segments of dialogue text to obtain the broadcast audio of the M segments of dialogue text; and a first playback unit, which is used to play the broadcast audio of the M segments of dialogue text respectively using N virtual digital humans based on the sequence of the M segments of dialogue text in the text to be processed.
[0061] Optionally, the first adjustment unit includes: a second adjustment unit, used to adjust the audio frequency of the initial audio of the M segments of dialogue text according to the respective intonations of the M segments of dialogue text, so as to obtain the respective broadcast audio of the M segments of dialogue text; and / or, to adjust the playback speed of the initial audio of the M segments of dialogue text according to the respective speaking speeds of the M segments of dialogue text, so as to obtain the respective broadcast audio of the M segments of dialogue text.
[0062] Optionally, the third determination unit includes: a fifth determination unit, used to use natural language processing technology to determine the description texts of N objects in the text to be processed, wherein the description texts of the N objects are used to describe the characteristics of the corresponding objects; and a sixth determination unit, used to determine the voice line characteristics of the N objects based on the description texts of the N objects.
[0063] Optionally, the third determination unit includes a seventh determination unit, which is used to extract features from the text to be processed and determine the text type of the text to be processed, wherein the text type is used to describe the genre of the text to be processed; and an eighth determination unit, which is used to determine the voice line features of each of the N objects according to the text type of the text to be processed.
[0064] The speech synthesis device provided in the embodiment of the present application obtains the text to be processed, identifies the object and its corresponding dialogue text, determines the speech speed and intonation of the dialogue text in combination with the context, and uses a virtual digital person to broadcast the dialogue text based on the basic voice of each object, thereby achieving the purpose of improving the expressiveness of the dialogue and the degree of scene restoration, and solves the technical problem of incomplete transmission of scene and dialogue information caused by the single voice of speech synthesis in the related technology, thereby achieving the technical effect of helping users understand the text scene and character emotions and improving the user experience.
[0065] The speech synthesis device includes a processor and a memory. The above-mentioned text acquisition module, dialogue recognition module, speech rate and intonation determination module, voice line determination module, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.
[0066] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and multi-voice speech synthesis can be achieved by adjusting kernel parameters.
[0067] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.
[0068] An embodiment of the present invention provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the speech synthesis method is implemented.
[0069] An embodiment of the present invention provides a processor, which is used to run a program, wherein the speech synthesis method is executed when the program is run.
[0070] like Figure 3As shown, an embodiment of the present invention provides an electronic device, the device includes a processor, a memory, and a program stored in the memory and executable on the processor, and the processor implements the following steps when executing the program: obtaining a text to be processed, wherein the text to be processed includes text describing interactive behaviors between N objects, where N is greater than 1; identifying the dialogue texts of the N objects in the text to be processed, and determining M dialogue texts, wherein M is greater than 1; determining the intonation of the M dialogue texts and / or the speech speed of the M dialogue texts according to the contexts of the M dialogue texts; and using N virtual digital humans to broadcast the M dialogue texts based on the basic voices of the N objects according to the intonation of the M dialogue texts and / or the speech speed of the M dialogue texts. Optionally, the intonation of each of the M dialogue texts and / or the speaking speed of each of the M dialogue texts are determined respectively according to the respective contexts of the M dialogue texts, including: determining the emotion type of each of the M dialogue texts according to the respective contexts of the M dialogue texts; determining the intonation of each of the M dialogue texts according to the emotion type of each of the M dialogue texts and the correspondence between the predetermined emotion type and the intonation strategy, wherein the intonation strategy represents the amplitude of adjusting the intonation based on a predetermined intonation; and / or determining the speaking speed of each of the M dialogue texts according to the emotion type of each of the M dialogue texts and the correspondence between the predetermined emotion type and the speaking speed strategy, wherein the speaking speed strategy represents the amplitude of adjusting the speaking speed based on a predetermined speaking speed.
[0071] Optionally, the emotion type of each of the M dialogue texts is determined according to their respective contexts, including: inputting the respective contexts of the M dialogue texts into a pre-trained neural network model, and the neural network model outputs the emotion type of each of the M dialogue texts, wherein the neural network model is obtained by training the original neural network model with training samples, and the training samples include sample dialogue texts and contexts, and sample emotion types corresponding to the sample dialogue texts.
[0072] Optionally, according to the intonation of each of the M dialogue texts and / or the speech speed of each of the M dialogue texts, based on the basic voice lines of each of the N objects, N virtual digital humans are respectively used to broadcast the M dialogue texts, including: extracting features of the text to be processed to determine the voice line features of each of the N objects; determining the basic voice lines of each of the N objects according to the voice line features of each of the N objects; broadcasting the M dialogue texts respectively using the basic voice lines of each of the N objects to obtain initial audio of the M dialogue texts; adjusting the initial audio of the M dialogue texts according to the intonation of each of the M dialogue texts and / or the speech speed of each of the M dialogue texts to obtain broadcast audio of each of the M dialogue texts; based on the sequence of the M dialogue texts in the text to be processed, using N virtual digital humans to play the broadcast audio of each of the M dialogue texts.
[0073] Optionally, the initial audio of the M segments of dialogue text is adjusted according to the respective intonations of the M segments of dialogue text and / or the respective speaking speeds of the M segments of dialogue text to obtain the respective broadcast audios of the M segments of dialogue text, including: adjusting the audio frequency of the initial audio of the M segments of dialogue text according to the respective intonations of the M segments of dialogue text to obtain the respective broadcast audios of the M segments of dialogue text; and / or adjusting the playback speed of the initial audio of the M segments of dialogue text according to the respective speaking speeds of the M segments of dialogue text to obtain the respective broadcast audios of the M segments of dialogue text.
[0074] Optionally, feature extraction is performed on the text to be processed to determine the voice line features of N objects, including: using natural language processing technology to determine the description texts of N objects in the text to be processed, wherein the description texts of the N objects are respectively used to describe the features of the corresponding objects; and determining the voice line features of the N objects according to the description texts of the N objects.
[0075] Optionally, feature extraction is performed on the text to be processed to determine the voice features of each of the N objects, including: feature extraction is performed on the text to be processed to determine the text type of the text to be processed, wherein the text type is used to describe the genre of the text to be processed; and according to the text type of the text to be processed, the voice features of each of the N objects are determined.
[0076] The devices in this article can be servers, PCs, PADs, mobile phones, etc.
[0077] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that is initialized with the following method steps: obtaining a text to be processed, wherein the text to be processed includes text describing interactive behaviors between N objects, where N is greater than 1; identifying the dialogue texts of the N objects in the text to be processed, and determining M segments of dialogue text, wherein M is greater than 1; determining the intonation of the M segments of dialogue text and / or the speaking speed of the M segments of dialogue text according to the context of each of the M segments of dialogue text; and using N virtual digital humans to broadcast the M segments of dialogue text based on the basic voices of the N objects according to the intonation of the M segments of dialogue text and / or the speaking speed of the M segments of dialogue text.
[0078] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0079] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0080] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0081] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0082] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0083] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0084] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0085] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0086] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0087] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.
Claims
1. A speech synthesis method, characterized in that: include: Acquire a text to be processed, wherein the text to be processed includes text describing interaction behaviors between N objects, where N is greater than 1; In the text to be processed, identifying the respective dialogue texts of the N objects, and determining M dialogue texts, where M is greater than 1; Determining the intonation of each of the M dialogue texts and / or the speech speed of each of the M dialogue texts according to the contexts of each of the M dialogue texts; According to the intonation of each of the M dialogue texts and / or the speaking speed of each of the M dialogue texts, based on the basic voice lines of each of the N objects, N virtual digital humans are respectively used to read the M dialogue texts.
2. The method according to claim 1, characterized in that The determining, according to the respective contexts of the M dialogue texts, the respective intonations of the M dialogue texts and / or the respective speech speeds of the M dialogue texts comprises: Determine the emotion type of each of the M dialogue texts according to the context of each of the M dialogue texts; Determine the intonation of each of the M dialogue texts according to the emotion type of each of the M dialogue texts and the predetermined correspondence between the emotion type and the intonation strategy, wherein the intonation strategy represents the amplitude of adjusting the intonation based on the predetermined intonation; And / or, the speech speed of each of the M segments of dialogue text is determined according to the emotion type of each of the M segments of dialogue text and the correspondence between the predetermined emotion type and the speech speed strategy, wherein the speech speed strategy represents the amplitude of adjusting the speech speed based on the predetermined speech speed.
3. The method according to claim 2, characterized in that Determining the emotion types of the M dialogue texts according to the respective contexts of the M dialogue texts includes: The contexts of the M dialogue texts are respectively input into a pre-trained neural network model, and the neural network model outputs the emotion types of the M dialogue texts. The neural network model is obtained by training the original neural network model with training samples, and the training samples include sample dialogue texts and contexts, and the sample emotion types corresponding to the sample dialogue texts.
4. The method according to claim 1, characterized in that: The method of using N virtual digital humans to broadcast the M dialogue texts according to the intonations of the M dialogue texts and / or the speech speeds of the M dialogue texts and based on the basic voices of the N objects includes: Extracting features from the text to be processed to determine the voice line features of each of the N objects; Determining basic sound lines of the N objects according to the sound line features of the N objects; Using the basic voice lines of the N objects to respectively broadcast the M dialogue texts to obtain initial audio of the M dialogue texts; According to the intonation of each of the M dialogue texts and / or the speech speed of each of the M dialogue texts, the initial audio of the M dialogue texts is adjusted to obtain the broadcast audio of each of the M dialogue texts; Based on the sequence of the M dialogue texts in the text to be processed, the N virtual digital humans are respectively used to play the respective broadcast audios of the M dialogue texts.
5. The method according to claim 4, characterized in that The method of adjusting the initial audio of the M segments of dialogue text according to the respective intonations of the M segments of dialogue text and / or the respective speech speeds of the M segments of dialogue text to obtain the respective broadcast audio of the M segments of dialogue text comprises: According to the intonation of each of the M dialogue texts, the audio frequency of the initial audio of the M dialogue texts is adjusted to obtain the broadcast audio of each of the M dialogue texts; And / or, according to the speech speed of each of the M segments of dialogue text, the playback speed of the initial audio of the M segments of dialogue text is adjusted to obtain the broadcast audio of each of the M segments of dialogue text.
6. The method according to claim 4, characterized in that The extracting features of the text to be processed to determine the sound line features of each of the N objects includes: Using natural language processing technology, respectively determine the description texts of the N objects in the text to be processed, wherein the description texts of the N objects are respectively used to describe the characteristics of the corresponding objects; and determine the sound line characteristics of the N objects according to the description texts of the N objects.
7. The method according to claim 4, characterized in that The extracting features of the text to be processed to determine the sound line features of each of the N objects includes: Performing feature extraction on the text to be processed to determine the text type of the text to be processed, wherein the text type is used to describe the genre of the text to be processed; According to the text type of the text to be processed, the sound line features of each of the N objects are determined.
8. A speech synthesis device, characterized in that: include: A text acquisition module, used to acquire text to be processed, wherein the text to be processed includes text describing interaction behaviors between N objects, where N is greater than 1; A dialogue recognition module, used to identify the dialogue texts of the N objects in the text to be processed, and determine M dialogue texts, where M is greater than 1; A speech rate and intonation determination module, used to determine the intonation of each of the M dialogue texts and / or the speech rate of each of the M dialogue texts according to the context of each of the M dialogue texts; The voice line determination module is used to respectively use N virtual digital people to read the M dialogue texts according to the respective intonations of the M dialogue texts and / or the respective speech speeds of the M dialogue texts and based on the respective basic voice lines of the N objects.
9. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the speech synthesis method according to any one of claims 1 to 7.
10. An electronic device, characterized in that: The invention comprises one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any one of claims 1 to 7.
11. A computer program product comprising computer instructions, characterized in that: The computer instructions are used by a processor to execute the method according to any one of claims 1 to 7.