Speech stream synthesis method and device based on emotion recognition

Abstract

The application discloses a speech stream synthesis method and device based on emotion recognition, and the method comprises the following steps: determining semantic features of to-be-analyzed materials according to text content of the to-be-analyzed materials, wherein the to-be-analyzed materials comprise any one of to-be-analyzed texts, to-be-analyzed speech streams and to-be-analyzed images containing at least one character; extracting emotion features of the to-be-analyzed materials according to the semantic features; and generating a first speech stream corresponding to the to-be-analyzed materials based on the emotion features, wherein the first speech stream is a speech stream containing the emotion features. It can be seen that, by implementing the application, emotion features of to-be-analyzed materials can be extracted according to semantic features of the to-be-analyzed materials, and a first speech stream corresponding to the to-be-analyzed materials can be generated based on the emotion features, so that the synthesized speech stream is full of emotion and more vivid.

Description

Technical Field

[0001] This invention relates to the field of speech synthesis technology, and in particular to a speech stream synthesis method and apparatus based on emotion recognition. Background Technology

[0002] In the modern digital age, text has become an indispensable part of people's daily lives and work. However, for some groups, such as the visually impaired, reading text is a difficult task. To help the visually impaired access information more conveniently and improve their productivity and quality of life, many text-to-speech tools have emerged. However, existing text-to-speech tools simply connect and integrate the fixed sounds corresponding to each character in the text into a speech stream, resulting in a rather stiff and unnatural speech stream. Therefore, synthesizing more vivid speech streams is particularly important. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a speech stream synthesis method and apparatus based on emotion recognition, which can make the synthesized speech stream more emotional and vivid.

[0004] To address the aforementioned technical problems, the first aspect of this invention discloses a speech stream synthesis method based on emotion recognition, the method comprising:

[0005] Based on the text content of the material to be analyzed, the semantic features of the material to be analyzed are determined. The material to be analyzed includes any one of the following: the text to be analyzed, the speech stream to be analyzed, and the image to be analyzed containing at least one character.

[0006] Based on the semantic features, extract the sentiment features of the material to be analyzed;

[0007] Based on the emotional features, a first speech stream corresponding to the material to be analyzed is generated, wherein the first speech stream is a speech stream containing the emotional features.

[0008] As an optional implementation, in the first aspect of the present invention, the method further includes:

[0009] If the material to be analyzed is a speech stream or an image, the text information in the material to be analyzed is recognized based on the text recognition method that matches the material to be analyzed, and the text content of the material to be analyzed is obtained.

[0010] And, generating the first speech stream corresponding to the material to be analyzed based on the emotional features includes:

[0011] In the process of generating a speech stream based on the text content, the emotional features are synchronously fused into the generated speech stream to obtain the first speech stream corresponding to the material to be analyzed; or...

[0012] The pre-generated initial speech stream is adjusted according to the emotional features to integrate the emotional features into the initial speech stream, thereby obtaining the first speech stream corresponding to the material to be analyzed. The initial speech stream includes the speech stream to be analyzed or a speech stream generated based on the text content.

[0013] As an optional implementation, in the first aspect of the present invention, the semantic features include global semantic features of the material to be analyzed and local semantic features of one or more segmented statements arranged in positional order in the text content;

[0014] Before extracting the sentiment features from the material to be analyzed based on the semantic features, the method further includes:

[0015] Extract the target keywords contained in each of the segmented statements;

[0016] And, the step of extracting the sentiment features of the material to be analyzed based on the semantic features includes:

[0017] Based on the global semantic features, the global sentiment features of the material to be analyzed are determined, wherein the global sentiment features include the sentiment tone of the material to be analyzed;

[0018] Based on the local semantic features of each segmented statement and the target keywords contained in each segmented statement, the local sentiment features of each segmented statement are determined. The local sentiment features include one or more of sentiment degree, sentiment type, and sentiment orientation. The sentiment degree includes one or more of sentiment pleasure degree, sentiment intensity degree, and sentiment controllability degree. The sentiment orientation includes introverted sentiment orientation or extroverted sentiment orientation.

[0019] Based on the global sentiment features, the sentiment level in the local sentiment features of each segmented sentence is adjusted to obtain new local sentiment features for each segmented sentence, which are then used as the sentiment features of the material to be analyzed.

[0020] As an optional implementation, in a first aspect of the invention, determining the local sentiment features of each segmented statement based on its local semantic features and the target keywords contained in each segmented statement includes:

[0021] For the first segmented statement in the order of all the segmented statements, the local sentiment features of the first segmented statement are determined based on the local semantic features of the first segmented statement and the target keywords contained in the first segmented statement.

[0022] For each of the segmented statements other than the first segmented statement, based on the local semantic features of the segmented statement, it is determined whether the degree of association between the segmented statement and its preceding adjacent segmented statement is greater than or equal to the degree of association threshold. If the determination result is yes, the local sentiment feature of the segmented statement is determined based on the local semantic features of the segmented statement, the target keywords contained in the segmented statement, and the local sentiment features of the preceding adjacent segmented statements. If the determination result is no, the local sentiment feature of the segmented statement is determined based on the local semantic features of the segmented statement and the target keywords contained in the segmented statement.

[0023] As an optional implementation, in the first aspect of the present invention, the step of synchronously fusing the emotional features into the generated speech stream during the process of generating a speech stream based on the text content to obtain a first speech stream corresponding to the material to be analyzed includes:

[0024] Based on the standard pronunciation of each morpheme in the text content and the standard pronunciation of its adjacent morphemes, the appropriate pronunciation of the morpheme is determined, and the morpheme is used to represent the smallest sound-meaning combination in the text content;

[0025] In the process of generating speech based on the adapted pronunciation of each morpheme, the emotional representation attribute that matches the morpheme and is determined in advance according to the emotional features is synchronously fused into the speech corresponding to the morpheme to synthesize the first speech stream corresponding to the material to be analyzed.

[0026] The step of adjusting the pre-generated initial speech stream according to the emotional features to fuse the emotional features into the initial speech stream, thereby obtaining the first speech stream corresponding to the material to be analyzed, includes:

[0027] Based on the emotional representation attributes that match each morpheme in the text content determined in advance according to the emotional features, the speech corresponding to the morpheme in the pre-generated initial speech stream is adjusted to obtain the first speech stream corresponding to the material to be analyzed;

[0028] Furthermore, the method further includes:

[0029] Based on the emotional characteristics, determine the basic intonation of each segmented sentence;

[0030] For each morpheme in each segmented sentence, based on the morpheme's morpheme attributes and the basic intonation of the segmented sentence, a sentiment representation attribute matching the morpheme is determined. The morpheme attributes of each morpheme include the word type to which the morpheme belongs and / or the position of the morpheme in the corresponding segmented sentence. The sentiment representation attribute includes one or more of stress, tone, and pronunciation duration.

[0031] As an optional implementation, in the first aspect of the present invention, the method further includes:

[0032] Identify the voice features that match one or more characters in the material to be analyzed;

[0033] The first speech stream is adjusted according to the voice features that match each character, so as to fuse the voice features that match the character into the sub-speech stream corresponding to the character in the first speech stream, thereby obtaining the second speech stream corresponding to the material to be analyzed;

[0034] The step of determining the voice features that match one or more characters in the material to be analyzed includes:

[0035] The system identifies one or more voice tags matching characters within the material to be analyzed, and synthesizes voice features matching each character based on these voice tags. Each voice tag matching a character includes a voice tag determined by the user for that character and / or a voice tag matching that character determined based on the feature information of the material to be analyzed. The feature information includes the semantic features and / or the emotional features; and / or...

[0036] For each character in the material to be analyzed, based on one or more pre-set voice feature extraction factors, the voice features matching the character are extracted from the pre-collected voice recordings for that character.

[0037] As an optional implementation, in the first aspect of the present invention, the method further includes:

[0038] If the material to be analyzed is the speech stream to be analyzed and the voiceprint recognition result of the material to be analyzed includes at least two types of voiceprint information, the person corresponding to each type of voiceprint information is determined as a character in the material to be analyzed;

[0039] If the material to be analyzed is the speech stream to be analyzed and the voiceprint recognition result of the material to be analyzed includes only one type of voiceprint information, or if the material to be analyzed is the text to be analyzed or the image to be analyzed, the role corresponding to each segmentation statement is determined based on the semantic features and the target keywords contained in one or more segmentation statements in the text content, as all the roles contained in the material to be analyzed, wherein each role corresponds to one or more segmentation statements.

[0040] As an optional implementation, in the first aspect of the present invention, the method further includes:

[0041] Determine the criteria for segmenting sentences that match the material to be analyzed;

[0042] Based on the stated statement segmentation criteria, the text content is segmented to obtain one or more segmented statements arranged in positional order within the text content;

[0043] The criteria for determining the sentence segmentation that matches the material to be analyzed include:

[0044] If the material to be analyzed is the speech stream to be analyzed, when the voiceprint recognition result of the material to be analyzed includes at least two types of voiceprint information, the voiceprint recognition result and the speech pause time in the material to be analyzed are determined as the basis for sentence segmentation matching the material to be analyzed; when the voiceprint recognition result of the material to be analyzed includes only one type of voiceprint information, the speech pause time in the material to be analyzed is determined as the basis for sentence segmentation matching the material to be analyzed.

[0045] If the material to be analyzed is the text to be analyzed or the image to be analyzed, determine whether the text content includes a first text element of a preset element type; if the text content includes the first text element, determine the first text element as the basis for sentence segmentation matching the material to be analyzed; if the text content does not include the first text element, determine the gap between every two adjacent morphemes in the text content, and determine whether there is a target gap greater than or equal to a preset length between every two adjacent morphemes; if the target gap exists, determine the target gap as the basis for sentence segmentation matching the material to be analyzed.

[0046] As an optional implementation, in a first aspect of the invention, after determining the voice features matching one or more characters included in the material to be analyzed, the method further includes:

[0047] Determine the voice state that matches one or more of the segmented statements corresponding to each of the aforementioned roles, wherein the voice state includes at least the voice age;

[0048] Based on the sound state matching each segmented statement corresponding to each character, the sub-sound features corresponding to the segmented statement in the sound features matching the character are adjusted to obtain new sound features matching the character, and the operation of adjusting the first speech stream based on the sound features matching each character is triggered to fuse the sound features matching the character into the sub-speech stream corresponding to the character in the first speech stream to obtain the second speech stream corresponding to the material to be analyzed.

[0049] Determining the sound state that matches one or more segmented statements for each role includes:

[0050] Based on the sound state description input by the user for one or more segmentation statements corresponding to each of the aforementioned roles, determine the sound state that matches the segmentation statement; and / or,

[0051] Based on the semantic features, determine the sound state that matches one or more of the segmented statements corresponding to each character.

[0052] A second aspect of the present invention discloses a speech stream synthesis apparatus based on emotion recognition, the apparatus comprising:

[0053] The determination module is used to determine the semantic features of the material to be analyzed based on the text content of the material to be analyzed. The material to be analyzed includes any one of the following: the text to be analyzed, the speech stream to be analyzed, and the image to be analyzed containing at least one character.

[0054] The extraction module is used to extract the sentiment features of the material to be analyzed based on the semantic features;

[0055] The speech generation module is used to generate a first speech stream corresponding to the material to be analyzed based on the emotional features, wherein the first speech stream is a speech stream containing the emotional features.

[0056] As an optional implementation, in a second aspect of the invention, the apparatus further includes:

[0057] The recognition module is used to identify the text information in the material to be analyzed, based on a text recognition method that matches the material to be analyzed, if the material to be analyzed is a speech stream or an image to be analyzed, and to obtain the text content of the material to be analyzed.

[0058] Furthermore, the specific method by which the speech generation module generates the first speech stream corresponding to the material to be analyzed based on the emotional features includes:

[0059] In the process of generating a speech stream based on the text content, the emotional features are synchronously fused into the generated speech stream to obtain the first speech stream corresponding to the material to be analyzed; or...

[0060] The pre-generated initial speech stream is adjusted according to the emotional features to integrate the emotional features into the initial speech stream, thereby obtaining the first speech stream corresponding to the material to be analyzed. The initial speech stream includes the speech stream to be analyzed or a speech stream generated based on the text content.

[0061] As an optional implementation, in a second aspect of the present invention, the semantic features include global semantic features of the material to be analyzed and local semantic features of one or more segmented statements arranged in positional order in the text content;

[0062] The extraction module is further configured to extract target keywords contained in each segmented statement before extracting the sentiment features in the material to be analyzed based on the semantic features;

[0063] Furthermore, the specific method by which the extraction module extracts the sentiment features of the material to be analyzed based on the semantic features includes:

[0064] Based on the global semantic features, the global sentiment features of the material to be analyzed are determined, wherein the global sentiment features include the sentiment tone of the material to be analyzed;

[0065] Based on the local semantic features of each segmented statement and the target keywords contained in each segmented statement, the local sentiment features of each segmented statement are determined. The local sentiment features include one or more of sentiment degree, sentiment type, and sentiment orientation. The sentiment degree includes one or more of sentiment pleasure degree, sentiment intensity degree, and sentiment controllability degree. The sentiment orientation includes introverted sentiment orientation or extroverted sentiment orientation.

[0066] Based on the global sentiment features, the sentiment level in the local sentiment features of each segmented sentence is adjusted to obtain new local sentiment features for each segmented sentence, which are then used as the sentiment features of the material to be analyzed.

[0067] As an optional implementation, in a second aspect of the invention, the extraction module determines the local sentiment features of each segmented statement based on the local semantic features of each segmented statement and the target keywords contained in each segmented statement in a specific manner, including:

[0068] For the first segmented statement in the order of all the segmented statements, the local sentiment features of the first segmented statement are determined based on the local semantic features of the first segmented statement and the target keywords contained in the first segmented statement.

[0069] For each of the segmented statements other than the first segmented statement, based on the local semantic features of the segmented statement, it is determined whether the degree of association between the segmented statement and its preceding adjacent segmented statement is greater than or equal to the degree of association threshold. If the determination result is yes, the local sentiment feature of the segmented statement is determined based on the local semantic features of the segmented statement, the target keywords contained in the segmented statement, and the local sentiment features of the preceding adjacent segmented statements. If the determination result is no, the local sentiment feature of the segmented statement is determined based on the local semantic features of the segmented statement and the target keywords contained in the segmented statement.

[0070] As an optional implementation, in a second aspect of the present invention, the specific method by which the speech generation module synchronously integrates the emotional features into the generated speech stream during the process of generating a speech stream based on the text content to obtain the first speech stream corresponding to the material to be analyzed includes:

[0071] Based on the standard pronunciation of each morpheme in the text content and the standard pronunciation of its adjacent morphemes, the appropriate pronunciation of the morpheme is determined, and the morpheme is used to represent the smallest sound-meaning combination in the text content;

[0072] In the process of generating speech based on the adapted pronunciation of each morpheme, the emotional representation attribute that matches the morpheme and is determined in advance according to the emotional features is synchronously fused into the speech corresponding to the morpheme to synthesize the first speech stream corresponding to the material to be analyzed.

[0073] The specific method by which the speech generation module adjusts the pre-generated initial speech stream according to the emotional features to integrate the emotional features into the initial speech stream and obtain the first speech stream corresponding to the material to be analyzed includes:

[0074] Based on the emotional representation attributes that match each morpheme in the text content determined in advance according to the emotional features, the speech corresponding to the morpheme in the pre-generated initial speech stream is adjusted to obtain the first speech stream corresponding to the material to be analyzed;

[0075] Furthermore, the determining module is also configured to determine the basic intonation of each segmented statement based on the emotional features; for each morpheme in each segmented statement, determine the emotional representation attribute matching the morpheme based on the morpheme's morpheme attributes and the basic intonation of the segmented statement, wherein the morpheme attributes of each morpheme include the word type to which the morpheme belongs and / or the position of the morpheme in the corresponding segmented statement, and the emotional representation attribute includes one or more of stress, tone, and pronunciation duration.

[0076] As an optional implementation, in a second aspect of the invention, the determining module is further configured to determine the voice features that match one or more characters contained in the material to be analyzed;

[0077] The speech generation module is further configured to adjust the first speech stream according to the voice features that match each of the characters, so as to fuse the voice features that match the character into the sub-speech stream corresponding to the character in the first speech stream, thereby obtaining the second speech stream corresponding to the material to be analyzed;

[0078] The specific method by which the determining module determines the voice features matching one or more characters contained in the material to be analyzed includes:

[0079] The system identifies one or more voice tags matching characters within the material to be analyzed, and synthesizes voice features matching each character based on these voice tags. Each voice tag matching a character includes a voice tag determined by the user for that character and / or a voice tag matching that character determined based on the feature information of the material to be analyzed. The feature information includes the semantic features and / or the emotional features; and / or...

[0080] For each character in the material to be analyzed, based on one or more pre-set voice feature extraction factors, the voice features matching the character are extracted from the pre-collected voice recordings for that character.

[0081] As an optional implementation, in a second aspect of the present invention, the determining module is further configured to: if the material to be analyzed is the speech stream to be analyzed and the voiceprint recognition result of the material to be analyzed includes at least two types of voiceprint information, determine the person corresponding to each type of voiceprint information as a character in the material to be analyzed; if the material to be analyzed is the speech stream to be analyzed and the voiceprint recognition result of the material to be analyzed includes only one type of voiceprint information, or if the material to be analyzed is the text to be analyzed or the image to be analyzed, determine the character corresponding to each segmentation statement based on the semantic features and the target keywords contained in one or more segmentation statements in the text content, as all the characters contained in the material to be analyzed, wherein each character corresponds to one or more segmentation statements.

[0082] As an optional implementation, in a second aspect of the present invention, the determining module is further configured to determine the sentence segmentation criteria that match the material to be analyzed;

[0083] The device further includes:

[0084] The segmentation module is used to segment the text content based on the statement segmentation criteria to obtain one or more segmented statements contained in the text content arranged in positional order;

[0085] The specific method by which the determining module determines the sentence segmentation criteria that match the material to be analyzed includes:

[0086] If the material to be analyzed is the speech stream to be analyzed, when the voiceprint recognition result of the material to be analyzed includes at least two types of voiceprint information, the voiceprint recognition result and the speech pause time in the material to be analyzed are determined as the basis for sentence segmentation matching the material to be analyzed; when the voiceprint recognition result of the material to be analyzed includes only one type of voiceprint information, the speech pause time in the material to be analyzed is determined as the basis for sentence segmentation matching the material to be analyzed.

[0087] If the material to be analyzed is the text to be analyzed or the image to be analyzed, determine whether the text content includes a first text element of a preset element type; if the text content includes the first text element, determine the first text element as the basis for sentence segmentation matching the material to be analyzed; if the text content does not include the first text element, determine the gap between every two adjacent morphemes in the text content, and determine whether there is a target gap greater than or equal to a preset length between every two adjacent morphemes; if the target gap exists, determine the target gap as the basis for sentence segmentation matching the material to be analyzed.

[0088] As an optional implementation, in a second aspect of the invention, the determining module is further configured to, after determining the voice features that match one or more characters contained in the material to be analyzed, determine the voice state that matches one or more segmented statements corresponding to each character, wherein the voice state includes at least voice age;

[0089] The device further includes:

[0090] The adjustment module is used to adjust the sub-sound features corresponding to the segmented statement in the sound features matched by each of the segmented statements corresponding to each of the roles, based on the sound state matched by each segmented statement for each role, to obtain new sound features matched by the role, and to trigger the speech generation module to perform the operation of adjusting the first speech stream based on the sound features matched by each of the roles, so as to fuse the sound features matched by the role into the sub-speech stream corresponding to the role in the first speech stream, to obtain the second speech stream corresponding to the material to be analyzed;

[0091] The specific method by which the determining module determines the sound state that matches one or more segmented statements for each role includes:

[0092] Based on the sound state description input by the user for one or more segmentation statements corresponding to each of the aforementioned roles, determine the sound state that matches the segmentation statement; and / or,

[0093] Based on the semantic features, determine the sound state that matches one or more of the segmented statements corresponding to each character.

[0094] A third aspect of the present invention discloses another speech stream synthesis apparatus based on emotion recognition, the apparatus comprising:

[0095] Memory containing executable program code;

[0096] A processor coupled to the memory;

[0097] The processor calls the executable program code stored in the memory to execute the speech stream synthesis method based on emotion recognition disclosed in the first aspect of the present invention.

[0098] The fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute the speech stream synthesis method based on emotion recognition disclosed in the first aspect of the present invention.

[0099] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0100] In this embodiment of the invention, semantic features of the material to be analyzed are determined based on its text content. The material to be analyzed includes any one of the following: text to be analyzed, speech stream to be analyzed, and image to be analyzed containing at least one character. Emotional features are extracted from the material to be analyzed based on the semantic features. Based on the emotional features, a first speech stream corresponding to the material to be analyzed is generated, and the first speech stream is a speech stream containing emotional features. Therefore, implementing this invention enables the extraction of emotional features from the material to be analyzed based on its semantic features, and the generation of a first speech stream corresponding to the material to be analyzed based on these emotional features, resulting in a more emotional and vivid synthesized speech stream. Attached Figure Description

[0101] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0102] Figure 1 This is a schematic flowchart of a speech stream synthesis method based on emotion recognition disclosed in an embodiment of the present invention;

[0103] Figure 2 This is a flowchart illustrating another speech stream synthesis method based on emotion recognition disclosed in an embodiment of the present invention;

[0104] Figure 3 This is a schematic diagram of the structure of a speech stream synthesis device based on emotion recognition disclosed in an embodiment of the present invention;

[0105] Figure 4 This is a schematic diagram of another speech stream synthesis device based on emotion recognition disclosed in an embodiment of the present invention;

[0106] Figure 5 This is a schematic diagram of another speech stream synthesis device based on emotion recognition disclosed in an embodiment of the present invention. Detailed Implementation

[0107] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0108] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0109] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0110] This invention discloses a speech stream synthesis method and apparatus based on emotion recognition. It can extract the emotional features of the material to be analyzed based on its semantic features, and generate a first speech stream corresponding to the material to be analyzed based on these emotional features, resulting in a more emotional and vivid synthesized speech stream. Detailed descriptions follow.

[0111] Example 1

[0112] Please see Figure 1 , Figure 1 This is a flowchart illustrating a speech stream synthesis method based on emotion recognition disclosed in an embodiment of the present invention. Figure 1 The described emotion-based speech stream synthesis method can be applied to any terminal and server with text-to-speech requirements, such as smart terminals for visually impaired individuals, e-readers, intelligent dubbing software or platforms, etc., and the embodiments of this invention are not limited thereto. Figure 1 As shown, the speech stream synthesis method based on emotion recognition may include the following operations:

[0113] 101. Determine the semantic features of the material to be analyzed based on its text content.

[0114] Optionally, the material to be analyzed may include any one of the following: the text to be analyzed, the audio stream to be analyzed, and the image to be analyzed containing at least one character. This embodiment of the invention does not limit the scope of the analysis.

[0115] Optionally, semantic features may include global semantic features of the material to be analyzed and local semantic features of one or more segmented statements arranged in positional order in the text content. Further, optionally, global semantic features may include the text type to which the text content of the material to be analyzed belongs, the central idea (or text intent) of the text content, the text title, etc. Local semantic features of each segmented statement may include one or more of the following: natural semantic features used to represent the basic concepts and logical meanings in the segmented statement; auxiliary semantic features used to represent the non-natural and subjective content in the segmented statement; aggregate semantic features used to represent the association between different words after being separated from the segmented statement; and combinatorial semantic features used to represent the semantics of different word combinations in the segmented statement.

[0116] 102. Extract the emotional features of the material to be analyzed based on semantic features.

[0117] As an optional implementation method, extracting the sentiment features of the material to be analyzed based on semantic features may include:

[0118] Based on the global semantic features, the global sentiment features of the material to be analyzed are determined. The global sentiment features can include the emotional tone of the material to be analyzed. For example, if the global semantic features indicate that the material to be analyzed is a fairy tale, then its emotional tone is relatively cheerful. If the global semantic features indicate that the material to be analyzed is a prose, then its emotional tone is relatively tranquil.

[0119] Based on the local semantic features of each segmented statement and the target keywords contained in each segmented statement, the local sentiment features of each segmented statement are determined. Optionally, the local sentiment features may include one or more of sentiment degree, sentiment type, and sentiment orientation. Further optionally, the sentiment degree may include one or more of sentiment pleasure degree, sentiment intensity degree, and sentiment controllability degree, and the sentiment orientation may include introverted sentiment orientation or extroverted sentiment orientation.

[0120] Based on the global sentiment features, the sentiment level in the local sentiment features of each segmented sentence is adjusted to obtain new local sentiment features for each segmented sentence, which serve as the sentiment features of the material to be analyzed.

[0121] In this optional implementation, introverted affective orientation is used to represent affective orientation towards self-factors, while extroverted affective orientation is used to represent affective orientation towards external factors. For example, the affective orientation of "Xiaoming is not satisfied with his appearance" is introverted affective orientation, while the affective orientation of "Xiaohong especially likes spicy hot pot" is extroverted affective orientation. It should be noted that the intensity of emotion is positively correlated with the controllability of emotion, but different emotion types may not have the same level of controllability under the same intensity of emotion. For example, under the same intensity of emotion, the controllability of anger is usually higher than that of fear.

[0122] In this optional implementation, before extracting the sentiment features from the material to be analyzed based on semantic features, the method may further include: extracting target keywords contained in each segmented sentence. Optionally, the target keywords may include a combination of one or more of the following: sentiment words, degree particles, modal particles, affirmative words, negative words, sentiment subject, and sentiment object. Here, the sentiment subject represents the owner of the sentiment, and the sentiment object represents the object of the sentiment. For example, in "Xiaoming likes Xiaohong," "Xiaoming" is the sentiment subject, and "Xiaohong" is the sentiment object.

[0123] As can be seen, this optional implementation method determines the emotional tone of the material to be analyzed through global semantic features, and after determining the corresponding local emotional features based on the local semantic features of the segmented sentences and their target keywords, it adjusts the local emotional features using the emotional tone so that the emotional features of each whole text content match its emotional tone. This not only improves the accuracy of emotional feature determination, but also helps to keep the emotional tone of the synthesized speech stream consistent.

[0124] In this optional implementation, determining the local sentiment features of each segmented statement based on its local semantic features and the target keywords contained in each segmented statement may include:

[0125] For the first segmented statement in the order of all segmented statements, the local sentiment features of the first segmented statement are determined based on its local semantic features and the target keywords it contains.

[0126] For each segmented statement other than the first one, based on its local semantic features, determine whether the degree of association between the segmented statement and its preceding adjacent segmented statement is greater than or equal to the degree of association threshold. If the determination result is yes, determine the local sentiment feature of the segmented statement based on its local semantic features, the target keywords contained in the segmented statement, and the local sentiment features of its preceding adjacent segmented statements. If the determination result is no, determine the local sentiment feature of the segmented statement based on its local semantic features and the target keywords contained in the segmented statement.

[0127] It is evident that implementing this optional implementation method can determine the local sentiment features of a segmented statement by combining the local sentiment features of the preceding adjacent segmented statements when the correlation between the segmented statement and its preceding adjacent segmented statements is strong, thereby improving the efficiency, accuracy, and completeness of determining the local sentiment features of the segmented statement.

[0128] In this optional implementation, further optionally, determining the local sentiment features of the segmented statement based on the local semantic features of the segmented statement, the target keywords contained in the segmented statement, and the local sentiment features of the preceding adjacent segmented statements may include:

[0129] Determine whether the target keywords contained in the segmented statement contain sentiment keywords;

[0130] When the judgment result is yes, the local sentiment features of the segmented statement are determined based on the local sentiment features of the segmented statement and the target keywords contained in the segmented statement.

[0131] When the judgment result is negative, the association between the segmented statement and its preceding adjacent segmented statements is determined based on the local semantic features of the segmented statement and the target keywords contained in the segmented statement. The local sentiment features of the segmented statement are determined based on the local sentiment features of the preceding adjacent segmented statements and the association between the segmented statement and its preceding adjacent segmented statements. The association is either a positive association or a negative association. A positive association indicates that the similarity of the sentiment type between the segmented statement and its preceding adjacent segmented statements is greater than or equal to a first similarity threshold. A negative association indicates that the similarity of the sentiment type between the segmented statement and its preceding adjacent segmented statements is less than a second similarity threshold. The first similarity threshold is greater than or equal to the second similarity threshold.

[0132] Furthermore, after determining the local sentiment features of the segmented statement based on the local sentiment features of its preceding adjacent segmented statements and the relationship between the segmented statement and its preceding adjacent segmented statements, the method further includes: modifying the sentiment degree and sentiment orientation in the local sentiment features of the segmented statement based on the local semantic features of the segmented statement and the target keywords contained in the segmented statement, so as to obtain new local sentiment features of the segmented statement.

[0133] As can be seen, implementing this optional implementation method can determine the local sentiment features of a segmented statement based on the relationship between the segmented statement and its preceding adjacent segmented statements, as well as the local sentiment features of the preceding adjacent segmented statements, thereby further improving the efficiency and accuracy of sentiment feature determination.

[0134] 103. Based on emotional features, generate the first speech stream corresponding to the material to be analyzed.

[0135] The first speech stream is a speech stream that includes emotional features.

[0136] As an optional implementation method, generating a first speech stream corresponding to the material to be analyzed based on emotional features may include:

[0137] In the process of generating a speech stream based on text content, emotional features are synchronously fused into the generated speech stream to obtain the first speech stream corresponding to the material to be analyzed; or...

[0138] The pre-generated initial speech stream is adjusted based on emotional features to integrate the emotional features into the initial speech stream, resulting in the first speech stream corresponding to the material to be analyzed. The initial speech stream may include the speech stream to be analyzed or a speech stream generated based on text content.

[0139] It is evident that implementing this optional implementation method can simultaneously integrate emotional features during the process of generating speech streams based on text content, thereby improving the synthesis efficiency of speech streams, or it can integrate emotional features into the speech stream to be analyzed or the already generated speech stream, thereby improving the synthesis accuracy of speech streams and enriching the speech stream synthesis methods.

[0140] In this optional implementation, further optionally, during the process of generating a speech stream based on text content, emotional features are synchronously fused into the generated speech stream to obtain a first speech stream corresponding to the material to be analyzed. This may include:

[0141] Based on the standard pronunciation of each morpheme in the text content and the standard pronunciation of its adjacent morphemes, the appropriate pronunciation of the morpheme is determined. The morpheme is used to represent the smallest combination of sound and meaning in the text content.

[0142] In the process of generating speech based on the adapted pronunciation of each morpheme, the affective representation attribute matching the morpheme determined in advance according to the affective features is synchronously fused into the speech corresponding to the morpheme, so as to synthesize the first speech stream corresponding to the material to be analyzed.

[0143] It can be seen that the implementation of this optional embodiment can further determine the adapted pronunciation of each morpheme according to the standard pronunciation of each morpheme and its adjacent morphemes in the text content, which is beneficial to improving the fluency of the synthesized speech and reducing speech stuttering, and the speech stream is synthesized according to the adapted pronunciation and affective representation attribute of each morpheme, so that the integration of speech and emotion is more natural, vivid and smooth.

[0144] In this optional embodiment, further optionally, adjusting the pre-generated initial speech stream according to the affective features to fuse the affective features into the initial speech stream to obtain the first speech stream corresponding to the material to be analyzed may include:

[0145] Adjusting the speech corresponding to each morpheme in the pre-generated initial speech stream based on the affective representation attribute matching each morpheme in the text content determined in advance according to the affective features, to obtain the first speech stream corresponding to the material to be analyzed.

[0146] It can be seen that the implementation of this optional embodiment can further adjust the speech corresponding to each morpheme in the initial speech stream according to the affective representation attribute of the morpheme, which improves the precision of adjusting the initial speech stream.

[0147] In this optional embodiment, still further optionally, the method may further include:

[0148] Determining the basic intonation of each segmented sentence according to the affective features;

[0149] For each morpheme in each segmented sentence, determining the affective representation attribute matching the morpheme according to the morpheme attribute of the morpheme and the basic intonation of the segmented sentence.

[0150] Optionally, the basic intonation may include a rising intonation, a falling intonation, a flat intonation, a winding intonation, and the like, the morpheme attribute of each morpheme may include the word type to which the morpheme belongs and / or the position of the morpheme in the corresponding segmented sentence, and the affective representation attribute may include one or more of stress degree, tone, pronunciation duration, and the like. For example, according to the semantic features of the sentence "Have you eaten?", it can be known that the sentence is an interrogative sentence, and the corresponding basic intonation should be a rising intonation, so the tone of the final word "ma" at the end of the sentence should be higher than the tones of "ni chi le" at the beginning and in the middle of the sentence.

[0151] It is evident that implementing this optional implementation method improves the accuracy of determining the emotional representation attributes of morphemes by determining the stress, tone, and pronunciation duration of morphemes through the word type and position to which the morpheme belongs, as well as the basic intonation of the segmented sentence.

[0152] As can be seen, implementing the embodiments of the present invention can extract the emotional features of the material to be analyzed based on the semantic features of the material to be analyzed, and generate the first speech stream corresponding to the material to be analyzed based on the emotional features, so that the synthesized speech stream is rich in emotion and more vivid.

[0153] In an optional embodiment, the method may further include: if the material to be analyzed is a speech stream or an image, recognizing the text information in the material based on a text recognition method that matches the material to be analyzed, thereby obtaining the text content of the material to be analyzed; optionally, the text recognition method matching the image to be analyzed may be OCR (Optical Character Recognition) technology, and the text recognition method matching the speech stream to be analyzed may be any speech-to-text technology. This method of obtaining text content by recognizing the text information in the speech stream or image to be analyzed facilitates the conversion of text in different materials into speech.

[0154] In another alternative embodiment, the method may further include:

[0155] Determine the criteria for segmenting sentences that match the material to be analyzed;

[0156] Based on the sentence segmentation criteria, the text content is segmented to obtain one or more segmented sentences arranged in positional order within the text content;

[0157] The criteria for determining the sentence segmentation that matches the material to be analyzed may include:

[0158] If the material to be analyzed is a speech stream, when the voiceprint recognition result of the material to be analyzed includes at least two types of voiceprint information, the voiceprint recognition result and the speech pause time in the material to be analyzed are determined as the basis for matching the sentence segmentation of the material to be analyzed; when the voiceprint recognition result of the material to be analyzed includes only one type of voiceprint information, the speech pause time in the material to be analyzed is determined as the basis for matching the sentence segmentation of the material to be analyzed.

[0159] If the material to be analyzed is a text to be analyzed or an image to be analyzed, it is determined whether the text content includes a first text element whose element type is a preset element type; if the text content includes the first text element, the first text element is determined as the sentence segmentation basis matching the material to be analyzed; if the text content does not include the first text element, gaps between every two adjacent morphemes in the text content are determined, and it is determined whether there is a target gap greater than or equal to a preset length in the gaps between every two adjacent morphemes; if the target gap exists, the target gap is determined as the sentence segmentation basis matching the material to be analyzed; optionally, the first text element may include a preset punctuation mark, such as "。", etc., further optionally, the first text element may further include a preset sentence-ending modal particle type, such as "ba", etc.

[0160] It can be seen that implementing this optional embodiment segments the text content of the speech stream to be analyzed through the voiceprint recognition result and speech pause information, which not only improves the efficiency of segmenting the text content of the speech stream to be analyzed, but also can separate the segmented sentences corresponding to different roles. In addition, segmenting the text content through the first text element whose element type is the preset element type and the gap between every two adjacent morphemes improves the accuracy of text content segmentation.

[0161] Embodiment 2

[0162] Please refer to Figure 2 , Figure 2 which is a schematic flowchart of another speech stream synthesis method based on emotion recognition disclosed in the embodiments of the present invention. Wherein, Figure 2 the described speech stream synthesis method based on emotion recognition can be applied to any terminal and server with text-to-speech requirements, such as intelligent terminals for visually impaired people, e-readers, intelligent dubbing software or platforms, etc., which is not limited in the embodiments of the present invention. In addition, it can also be used to store people's voices, for example, to permanently preserve the voices of deceased people. As shown in Figure 2 , the speech stream synthesis method based on emotion recognition may include the following operations:

[0163] 201. Determine the semantic feature of the material to be analyzed according to the text content of the material to be analyzed.

[0164] 202. Extract the emotion feature of the material to be analyzed according to the semantic feature.

[0165] 203. Generate the first speech stream corresponding to the material to be analyzed based on the emotion feature.

[0166] In the embodiment of the present invention, for other descriptions of steps 201 to 203, please refer to the detailed descriptions of steps 101 to 103 in Embodiment 1, which will not be repeated in this embodiment of the present invention.

[0167] 204. Identify the voice features that match one or more characters in the material to be analyzed.

[0168] As an optional implementation, determining the voice features that match one or more characters contained in the material to be analyzed may include:

[0169] Identify voice tags matching one or more characters within the material to be analyzed, and synthesize voice features matching each character based on the matching voice tags. Optionally, the voice tags matching each character may include voice tags determined by the user for that character and / or voice tags matching that character determined based on the feature information of the material to be analyzed. Feature information may include semantic features and / or emotional features. Optionally, voice tags may include one or more of the following: volume, frequency, timbre, sound source distance, sound direction, vocal position, accent, resonance cavity, etc.; and / or...

[0170] For each character in the material to be analyzed, based on one or more pre-defined voice feature extraction factors, the voice features matching the character are extracted from the pre-collected voice recordings for that character. Optionally, the voice feature extraction factors may include one or more of the following: volume feature extraction factor, timbre feature extraction factor, frequency feature extraction factor, vocal position feature extraction factor, accent feature extraction factor, resonance cavity feature extraction factor, etc.

[0171] As can be seen, implementing this optional implementation method can synthesize voice features based on user-defined voice tags to meet users' personalized needs. Alternatively, voice tags can be determined based on semantic and emotional features to further synthesize voice features, which improves the intelligence of voice feature synthesis. It can also extract voice features from existing voice recordings so that users can choose their favorite real human voices.

[0172] 205. Adjust the first speech stream according to the matching voice features of each character, so as to merge the matching voice features of the character into the sub-speech stream corresponding to the character in the first speech stream, and obtain the second speech stream corresponding to the material to be analyzed.

[0173] Optionally, the voice features matched for each character may include sub-voice features matched for one or more segmented statements corresponding to that character, where each segmented statement is used to represent the character's lines.

[0174] As can be seen, implementing the embodiments of the present invention can extract the emotional features of the material to be analyzed based on the semantic features of the material to be analyzed, and generate a first speech stream corresponding to the material to be analyzed based on the emotional features, so that the synthesized speech stream is rich in emotion. It can also integrate the voice features of the characters contained in the material to be analyzed into the first speech stream, further enriching the content of the speech stream, making the synthesized speech stream more similar to real human speech and more vivid, and improving the personalization of the synthesized speech stream.

[0175] In an optional embodiment, the method may further include:

[0176] If the material to be analyzed is a speech stream and the voiceprint recognition result of the material to be analyzed includes at least two types of voiceprint information, the person corresponding to each type of voiceprint information is identified as a character in the material to be analyzed.

[0177] If the material to be analyzed is a speech stream and the voiceprint recognition result of the material to be analyzed includes only one type of voiceprint information, or if the material to be analyzed is text or an image to be analyzed, the role corresponding to each segmentation statement is determined based on semantic features and the target keywords contained in one or more segmentation statements in the text content, and is used as all the roles contained in the material to be analyzed, wherein each role corresponds to one or more segmentation statements.

[0178] As can be seen, implementing this optional embodiment can determine the roles contained in the speech stream to be analyzed by the voiceprint information in the speech stream to be analyzed, thereby improving the efficiency of role determination in the material to be analyzed, or determine all the roles contained in the material to be analyzed by the target keywords and their semantic features contained in each segmented sentence in the text content, thereby improving the accuracy and comprehensiveness of role determination in the material to be analyzed.

[0179] In another alternative embodiment, after determining the matching voice features of one or more characters contained in the material to be analyzed, the method may further include:

[0180] Determine the sound state that matches one or more segmented statements for each character;

[0181] Based on the sound state that matches each segmented statement corresponding to each character, the sub-sound features corresponding to the segmented statement in the sound features that match the character are adjusted to obtain new sound features that match the character. This triggers the above-mentioned adjustment of the first speech stream based on the sound features that match each character, so as to merge the sound features that match the character into the sub-speech stream corresponding to the character in the first speech stream, and obtain the second speech stream corresponding to the material to be analyzed.

[0182] Optionally, determining the sound state that matches one or more segmented statements corresponding to each role may include: determining the sound state that matches the segmented statement based on the sound state description input by the user for one or more segmented statements corresponding to each role; and / or, determining the sound state that matches one or more segmented statements corresponding to each role based on semantic features.

[0183] Optionally, voice state may include at least voice age. For example, if an article describes a character's youth and old age, then the character's voice age in youth and old age should be different. Further optional, voice state may also include one or more of the following: fluency of speech, voice health, voice fullness, and voice hoarseness. For example, if a character needs to have a long dialogue or deliver a long speech, the voice state corresponding to the segmented sentences should be increasingly hoarse; if a character is ill, their voice health is relatively low, and their voice state should be relatively hoarse, nasal, occasionally coughing, and low in volume.

[0184] As can be seen, implementing this optional embodiment can adjust the sub-sound features corresponding to each segmented sentence according to the matching sound state of each segmented sentence corresponding to each role, so that each sentence in the synthesized speech stream matches the sound state of the corresponding role, making the synthesized speech stream more vivid, more personalized, and more vivid, which helps to enhance the sense of immersion in the speech stream.

[0185] Example 3

[0186] Please see Figure 3 , Figure 3 This is a schematic diagram of a speech stream synthesis device based on emotion recognition disclosed in an embodiment of the present invention. Figure 3 The described emotion-based speech synthesis device can be applied to any terminal and server with text-to-speech requirements, such as smart terminals for visually impaired individuals, e-readers, intelligent dubbing software or platforms, etc. This invention is not limited in its embodiments. Furthermore, it can also be used to store people's voices, such as for permanently preserving the voices of deceased individuals. Figure 3 As shown, the emotion recognition-based speech stream synthesis device may include:

[0187] The determination module 301 is used to determine the semantic features of the material to be analyzed based on the text content of the material to be analyzed. The material to be analyzed includes any one of the following: the text to be analyzed, the speech stream to be analyzed, and the image to be analyzed containing at least one character.

[0188] Extraction module 302 is used to extract the sentiment features of the material to be analyzed based on semantic features;

[0189] The speech generation module 303 is used to generate a first speech stream corresponding to the material to be analyzed based on emotional features. The first speech stream is a speech stream containing emotional features.

[0190] It is evident that implementation Figure 3 The described device can extract the emotional features of the material to be analyzed based on its semantic features, and generate a first speech stream corresponding to the material to be analyzed based on the emotional features, making the synthesized speech stream more emotional and vivid.

[0191] In an optional embodiment, such as Figure 4 As shown, the device may further include:

[0192] The recognition module 304 is used to recognize the text information in the material to be analyzed based on the text recognition method that matches the material to be analyzed, if the material to be analyzed is a speech stream or an image to be analyzed, and to obtain the text content of the material to be analyzed.

[0193] Optionally, the specific method by which the speech generation module 303 generates the first speech stream corresponding to the material to be analyzed based on emotional features may include:

[0194] In the process of generating a speech stream based on text content, emotional features are synchronously fused into the generated speech stream to obtain the first speech stream corresponding to the material to be analyzed; or...

[0195] The pre-generated initial speech stream is adjusted based on emotional features to integrate the emotional features into the initial speech stream, resulting in the first speech stream corresponding to the material to be analyzed. The initial speech stream includes the speech stream to be analyzed or a speech stream generated based on text content.

[0196] It is evident that implementation Figure 4 The described device can recognize text information in the speech stream or image to be analyzed to obtain text content, and can easily convert text in different materials into speech. In addition, it can simultaneously integrate emotional features during the process of generating speech streams based on text content to improve the synthesis efficiency of speech streams, and can also integrate emotional features into the speech stream to be analyzed or the already generated speech stream to improve the synthesis accuracy of speech streams, while enriching the speech stream synthesis methods.

[0197] In another alternative embodiment, such as Figure 4 As shown, semantic features can include global semantic features of the material to be analyzed and local semantic features of one or more segmented sentences arranged in positional order in the text content;

[0198] The extraction module 302 is also used to extract the target keywords contained in each segmented sentence before extracting the sentiment features in the material to be analyzed based on semantic features;

[0199] Furthermore, the specific methods by which the extraction module 302 extracts the sentiment features of the material to be analyzed based on semantic features may include:

[0200] Based on the global semantic features, the global sentiment features of the material to be analyzed are determined. The global sentiment features include the sentiment tone of the material to be analyzed.

[0201] Based on the local semantic features of each segmented statement and the target keywords contained in each segmented statement, the local sentiment features of each segmented statement are determined. The local sentiment features include one or more of sentiment degree, sentiment type, and sentiment orientation. Among them, sentiment degree includes one or more of sentiment pleasure degree, sentiment intensity degree, and sentiment controllability degree, and sentiment orientation includes introverted sentiment orientation or extroverted sentiment orientation.

[0202] Based on the global sentiment features, the sentiment level in the local sentiment features of each segmented sentence is adjusted to obtain new local sentiment features for each segmented sentence, which serve as the sentiment features of the material to be analyzed.

[0203] It is evident that implementation Figure 4 The described device can also determine the emotional tone of the material to be analyzed through global semantic features, and after determining the corresponding local emotional features based on the local semantic features of the segmented sentences and their target keywords, it adjusts the local emotional features using the emotional tone so that the emotional features of each whole text content match its emotional tone. This not only improves the accuracy of emotional feature determination, but also helps to keep the emotional tone consistent in the synthesized speech stream.

[0204] In yet another alternative embodiment, such as Figure 4 As shown, the extraction module 302 determines the local sentiment features of each segmented statement based on its local semantic features and the target keywords contained in each segmented statement. This determination may include:

[0205] For the first segmented statement in the order of all segmented statements, the local sentiment features of the first segmented statement are determined based on its local semantic features and the target keywords it contains.

[0206] For each segmented statement other than the first one, based on its local semantic features, determine whether the degree of association between the segmented statement and its preceding adjacent segmented statement is greater than or equal to the degree of association threshold. If the determination result is yes, determine the local sentiment feature of the segmented statement based on its local semantic features, the target keywords contained in the segmented statement, and the local sentiment features of its preceding adjacent segmented statements. If the determination result is no, determine the local sentiment feature of the segmented statement based on its local semantic features and the target keywords contained in the segmented statement.

[0207] It is evident that implementation Figure 4 The described apparatus can also determine the local sentiment features of a segmented statement by combining the local sentiment features of the preceding adjacent segmented statements when the correlation between the segmented statement and its preceding adjacent segmented statements is strong, thereby improving the efficiency, accuracy and completeness of the determination of the local sentiment features of the segmented statement.

[0208] In yet another alternative embodiment, such as Figure 4 As shown, the specific method by which the speech generation module 303 synchronously integrates emotional features into the generated speech stream during the process of generating a speech stream based on text content, thereby obtaining the first speech stream corresponding to the material to be analyzed, may include:

[0209] Based on the standard pronunciation of each morpheme in the text content and the standard pronunciation of its adjacent morphemes, the appropriate pronunciation of the morpheme is determined. The morpheme is used to represent the smallest combination of sound and meaning in the text content.

[0210] In the process of generating speech based on the adapted pronunciation of each morpheme, the emotional representation attribute that matches the morpheme and is determined in advance according to the emotional features is synchronously fused into the speech corresponding to the morpheme to synthesize the first speech stream corresponding to the material to be analyzed.

[0211] Optionally, the specific method by which the speech generation module 303 adjusts the pre-generated initial speech stream according to emotional features to integrate the emotional features into the initial speech stream and obtain the first speech stream corresponding to the material to be analyzed may include:

[0212] Based on the emotional representation attributes that match each morpheme in the text content determined in advance according to emotional features, the speech corresponding to the morpheme in the pre-generated initial speech stream is adjusted to obtain the first speech stream corresponding to the material to be analyzed.

[0213] Furthermore, the determining module 301 is also used to determine the basic intonation of each segmented sentence based on the emotional features; for each morpheme in each segmented sentence, the emotional representation attribute matching the morpheme is determined based on the morpheme's morpheme attributes and the basic intonation of the segmented sentence, the morpheme attributes of each morpheme include the word type to which the morpheme belongs and / or the position of the morpheme in the corresponding segmented sentence, and the emotional representation attributes include one or more of stress, tone and pronunciation duration.

[0214] It is evident that implementation Figure 4 The described device can also determine the appropriate pronunciation of each morpheme in the text content based on the standard pronunciation of each morpheme and its adjacent morphemes, thereby improving the fluency of the synthesized speech and reducing speech stuttering. Furthermore, it can synthesize speech streams based on the appropriate pronunciation and emotional representation attributes of each morpheme, making the integration of speech and emotion more natural, vivid, and fluent. It can also adjust the speech corresponding to each morpheme in the initial speech stream based on the emotional representation attributes of each morpheme, thereby improving the accuracy of adjusting the initial speech stream.

[0215] In yet another alternative embodiment, such as Figure 4 As shown, the determining module 301 is also used to determine the voice features that match one or more characters contained in the material to be analyzed;

[0216] The speech generation module 303 is also used to adjust the first speech stream according to the voice features matched by each character, so as to fuse the voice features matched by the character into the sub-speech stream corresponding to the character in the first speech stream, so as to obtain the second speech stream corresponding to the material to be analyzed;

[0217] The specific method by which the determining module 301 determines the voice features that match one or more characters in the material to be analyzed may include:

[0218] Identify voice tags matching one or more characters within the material to be analyzed, and synthesize voice features matching each character based on the matching voice tags. Each matching voice tag includes a voice tag determined by the user for that character and / or a voice tag matching that character determined based on the feature information of the material to be analyzed. The feature information includes semantic features and / or emotional features; and / or...

[0219] For each character in the material to be analyzed, based on one or more pre-defined voice feature extraction factors, the voice features matching that character are extracted from the pre-collected voice recordings for that character.

[0220] It is evident that implementation Figure 4The described device can also integrate the voice features of the characters contained in the material to be analyzed into the speech stream, further enriching the content of the speech stream, making the speech stream synthesized from text more vivid and more similar to real human voice, and improving the personalization of the synthesized speech stream.

[0221] In yet another alternative embodiment, such as Figure 4 As shown, the determining module 301 is further configured to: if the material to be analyzed is a speech stream to be analyzed and the voiceprint recognition result of the material to be analyzed includes at least two types of voiceprint information, determine the person corresponding to each type of voiceprint information as a role in the material to be analyzed; if the material to be analyzed is a speech stream to be analyzed and the voiceprint recognition result of the material to be analyzed includes only one type of voiceprint information; or if the material to be analyzed is text to be analyzed or an image to be analyzed, determine the role corresponding to each segmentation statement based on semantic features and the target keywords contained in one or more segmentation statements in the text content, as all roles contained in the material to be analyzed, wherein each role corresponds to one or more segmentation statements.

[0222] It is evident that implementation Figure 4 The described device can also determine the roles contained in the speech stream to be analyzed based on the voiceprint information in the speech stream to be analyzed, thereby improving the efficiency of role determination in the material to be analyzed, or determine all the roles contained in the material to be analyzed based on the target keywords and their semantic features contained in each segmented sentence in the text content of the material to be analyzed, thereby improving the accuracy and comprehensiveness of role determination in the material to be analyzed.

[0223] In yet another alternative embodiment, such as Figure 4 As shown, the determination module 301 is also used to determine the sentence segmentation criteria that match the material to be analyzed;

[0224] The device may also include:

[0225] The segmentation module 305 is used to segment the text content based on the statement segmentation criteria to obtain one or more segmented statements contained in the text content arranged in positional order.

[0226] Optionally, the specific method by which module 301 determines the segmentation criteria that match the material to be analyzed may include:

[0227] If the material to be analyzed is a speech stream, when the voiceprint recognition result of the material to be analyzed includes at least two types of voiceprint information, the voiceprint recognition result and the speech pause time in the material to be analyzed are determined as the basis for matching the sentence segmentation of the material to be analyzed; when the voiceprint recognition result of the material to be analyzed includes only one type of voiceprint information, the speech pause time in the material to be analyzed is determined as the basis for matching the sentence segmentation of the material to be analyzed.

[0228] If the material to be analyzed is text or an image, determine whether the text content includes a first text element of a preset element type. If the text content includes the first text element, determine the first text element as the basis for sentence segmentation matching the material to be analyzed. If the text content does not include the first text element, determine the gap between every two adjacent morphemes in the text content, and determine whether there is a target gap greater than or equal to a preset length between every two adjacent morphemes. If there is a target gap, determine the target gap as the basis for sentence segmentation matching the material to be analyzed.

[0229] It is evident that implementation Figure 4 The described device also segments the text content of the speech stream to be analyzed using voiceprint recognition results and speech pause information. This not only improves the efficiency of text content segmentation of the speech stream to be analyzed, but also separates the segmented sentences corresponding to different roles. In addition, by segmenting the text content according to the first text element with a preset element type and the gap between every two adjacent morphemes, the accuracy of text content segmentation is improved.

[0230] In yet another alternative embodiment, such as Figure 4 As shown,

[0231] The determination module 301 is further configured to determine the voice state matching one or more segmented statements corresponding to each character after determining the voice features matching one or more characters contained in the material to be analyzed, wherein the voice state includes at least the voice age;

[0232] The device may also include:

[0233] The adjustment module 306 is used to adjust the sub-sound features corresponding to the segmented statement in the sound features matched by each character according to the sound state matched by each segmented statement, to obtain the new sound features matched by the character, and trigger the speech generation module 303 to perform the above-mentioned adjustment of the first speech stream according to the sound features matched by each character, so as to merge the sound features matched by the character into the sub-speech stream corresponding to the character in the first speech stream, and obtain the second speech stream corresponding to the material to be analyzed;

[0234] Optionally, the specific method by which the determining module 301 determines the sound state that matches one or more segmented statements corresponding to each character may include:

[0235] Based on the sound state description input by the user for one or more segmentation statements corresponding to each role, determine the sound state that matches the segmentation statement; and / or,

[0236] Based on semantic features, determine the sound state that matches one or more segmented statements for each character.

[0237] It is evident that implementation Figure 4 The described device can also adjust the sub-sound features corresponding to each segmented sentence based on the matching sound state of each segmented sentence corresponding to each character, so that each sentence in the synthesized speech stream matches the sound state of the corresponding character, making the synthesized speech stream more vivid, more personalized, and more visual, which helps to enhance the immersion of the speech stream.

[0238] Example 4

[0239] Please see Figure 5 , Figure 5 This is a schematic diagram of another speech stream synthesis device based on emotion recognition disclosed in an embodiment of the present invention. Figure 5 As shown, the emotion recognition-based speech stream synthesis device may include:

[0240] Memory 401 storing executable program code;

[0241] Processor 402 coupled to memory 401;

[0242] The processor 402 calls the executable program code stored in the memory 401 to execute the steps in the speech stream synthesis method based on emotion recognition described in Embodiment 1 or Embodiment 2 of the present invention.

[0243] Example 5

[0244] This invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute the steps in the emotion recognition-based speech stream synthesis method described in Embodiment 1 or Embodiment 2 of this invention.

[0245] Example 6

[0246] This invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps in the emotion recognition-based speech stream synthesis method described in Embodiment 1 or Embodiment 2.

[0247] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0248] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0249] Finally, it should be noted that the speech stream synthesis method and apparatus based on emotion recognition disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech stream synthesis method based on emotion recognition, characterized in that, The method includes: Based on the text content of the material to be analyzed, the semantic features of the material to be analyzed are determined. The material to be analyzed includes any one of the following: the text to be analyzed, the speech stream to be analyzed, and the image to be analyzed containing at least one character. Based on the semantic features, extract the sentiment features of the material to be analyzed; Based on the emotional features, a first speech stream corresponding to the material to be analyzed is generated, wherein the first speech stream is a speech stream containing the emotional features; The method further includes: If the material to be analyzed is a speech stream or an image, the text information in the material to be analyzed is recognized based on the text recognition method that matches the material to be analyzed, and the text content of the material to be analyzed is obtained. And, generating the first speech stream corresponding to the material to be analyzed based on the emotional features includes: In the process of generating a speech stream based on the text content, the emotional features are synchronously fused into the generated speech stream to obtain the first speech stream corresponding to the material to be analyzed; or... The pre-generated initial speech stream is adjusted according to the emotional features to integrate the emotional features into the initial speech stream, thereby obtaining the first speech stream corresponding to the material to be analyzed. The initial speech stream includes the speech stream to be analyzed or a speech stream generated based on the text content. The semantic features include the global semantic features of the material to be analyzed and the local semantic features of one or more segmented sentences arranged in positional order in the text content; Before extracting the sentiment features from the material to be analyzed based on the semantic features, the method further includes: Extract the target keywords contained in each of the segmented statements; And, the step of extracting the sentiment features of the material to be analyzed based on the semantic features includes: Based on the global semantic features, the global sentiment features of the material to be analyzed are determined, wherein the global sentiment features include the sentiment tone of the material to be analyzed; Based on the local semantic features of each segmented statement and the target keywords contained in each segmented statement, the local sentiment features of each segmented statement are determined. The local sentiment features include sentiment degree, sentiment type, and sentiment orientation. The sentiment degree includes one or more of the following: sentiment pleasure degree, sentiment intensity degree, and sentiment controllability degree. The sentiment orientation includes introverted sentiment orientation or extroverted sentiment orientation. Based on the global sentiment features, the sentiment level in the local sentiment features of each segmented sentence is adjusted to obtain new local sentiment features for each segmented sentence, which are then used as the sentiment features of the material to be analyzed.

2. The speech stream synthesis method based on emotion recognition according to claim 1, characterized in that, The step of determining the local sentiment features of each segmented statement based on its local semantic features and the target keywords contained in each segmented statement includes: For the first segmented statement in the order of all the segmented statements, the local sentiment features of the first segmented statement are determined based on the local semantic features of the first segmented statement and the target keywords contained in the first segmented statement. For each of the segmented statements other than the first segmented statement, based on the local semantic features of the segmented statement, it is determined whether the degree of association between the segmented statement and its preceding adjacent segmented statement is greater than or equal to the degree of association threshold. If the determination result is yes, the local sentiment feature of the segmented statement is determined based on the local semantic features of the segmented statement, the target keywords contained in the segmented statement, and the local sentiment features of the preceding adjacent segmented statements. If the determination result is no, the local sentiment feature of the segmented statement is determined based on the local semantic features of the segmented statement and the target keywords contained in the segmented statement.

3. The speech stream synthesis method based on emotion recognition according to claim 1, characterized in that, The step of synchronously fusing the emotional features into the generated speech stream during the process of generating a speech stream based on the text content to obtain the first speech stream corresponding to the material to be analyzed includes: Based on the standard pronunciation of each morpheme in the text content and the standard pronunciation of its adjacent morphemes, the appropriate pronunciation of the morpheme is determined, and the morpheme is used to represent the smallest sound-meaning combination in the text content; In the process of generating speech based on the adapted pronunciation of each morpheme, the emotional representation attribute that matches the morpheme and is determined in advance according to the emotional features is synchronously fused into the speech corresponding to the morpheme to synthesize the first speech stream corresponding to the material to be analyzed. The step of adjusting the pre-generated initial speech stream according to the emotional features to fuse the emotional features into the initial speech stream, thereby obtaining the first speech stream corresponding to the material to be analyzed, includes: Based on the emotional representation attributes that match each morpheme in the text content determined in advance according to the emotional features, the speech corresponding to the morpheme in the pre-generated initial speech stream is adjusted to obtain the first speech stream corresponding to the material to be analyzed; Furthermore, the method further includes: Based on the emotional characteristics, determine the basic intonation of each segmented sentence; For each morpheme in each segmented sentence, based on the morpheme's morpheme attributes and the basic intonation of the segmented sentence, a sentiment representation attribute matching the morpheme is determined. The morpheme attributes of each morpheme include the word type to which the morpheme belongs and / or the position of the morpheme in the corresponding segmented sentence. The sentiment representation attribute includes one or more of stress, tone, and pronunciation duration.

4. The speech stream synthesis method based on emotion recognition according to any one of claims 1-3, characterized in that, The method further includes: Identify the voice features that match one or more characters in the material to be analyzed; The first speech stream is adjusted according to the voice features that match each character, so as to fuse the voice features that match the character into the sub-speech stream corresponding to the character in the first speech stream, thereby obtaining the second speech stream corresponding to the material to be analyzed; The step of determining the voice features that match one or more characters in the material to be analyzed includes: The system identifies one or more voice tags matching characters within the material to be analyzed, and synthesizes voice features matching each character based on these voice tags. Each voice tag matching a character includes a voice tag determined by the user for that character and / or a voice tag matching that character determined based on the feature information of the material to be analyzed. The feature information includes the semantic features and / or the emotional features; and / or... For each character in the material to be analyzed, based on one or more pre-set voice feature extraction factors, the voice features matching the character are extracted from the pre-collected voice recordings for that character.

5. The speech stream synthesis method based on emotion recognition according to claim 4, characterized in that, The method further includes: If the material to be analyzed is the speech stream to be analyzed and the voiceprint recognition result of the material to be analyzed includes at least two types of voiceprint information, the person corresponding to each type of voiceprint information is determined as a character in the material to be analyzed; If the material to be analyzed is the speech stream to be analyzed and the voiceprint recognition result of the material to be analyzed includes only one type of voiceprint information, or if the material to be analyzed is the text to be analyzed or the image to be analyzed, the role corresponding to each segmentation statement is determined based on the semantic features and the target keywords contained in one or more segmentation statements in the text content, as all the roles contained in the material to be analyzed, wherein each role corresponds to one or more segmentation statements.

6. The speech stream synthesis method based on emotion recognition according to any one of claims 1, 2, 3, and 5, characterized in that, The method further includes: Determine the criteria for segmenting sentences that match the material to be analyzed; Based on the stated statement segmentation criteria, the text content is segmented to obtain one or more segmented statements arranged in positional order within the text content; The criteria for determining the sentence segmentation that matches the material to be analyzed include: If the material to be analyzed is the speech stream to be analyzed, when the voiceprint recognition result of the material to be analyzed includes at least two types of voiceprint information, the voiceprint recognition result and the speech pause time in the material to be analyzed are determined as the basis for sentence segmentation matching the material to be analyzed; when the voiceprint recognition result of the material to be analyzed includes only one type of voiceprint information, the speech pause time in the material to be analyzed is determined as the basis for sentence segmentation matching the material to be analyzed. If the material to be analyzed is the text to be analyzed or the image to be analyzed, determine whether the text content includes a first text element of a preset element type; if the text content includes the first text element, determine the first text element as the basis for sentence segmentation matching the material to be analyzed; if the text content does not include the first text element, determine the gap between every two adjacent morphemes in the text content, and determine whether there is a target gap greater than or equal to a preset length between every two adjacent morphemes; if the target gap exists, determine the target gap as the basis for sentence segmentation matching the material to be analyzed.

7. The speech stream synthesis method based on emotion recognition according to claim 5, characterized in that, After determining the voice features that match one or more characters contained in the material to be analyzed, the method further includes: Determine the voice state that matches one or more of the segmented statements corresponding to each of the aforementioned roles, wherein the voice state includes at least the voice age; Based on the sound state matching each segmented statement corresponding to each character, the sub-sound features corresponding to the segmented statement in the sound features matching the character are adjusted to obtain new sound features matching the character, and the operation of adjusting the first speech stream based on the sound features matching each character is triggered to fuse the sound features matching the character into the sub-speech stream corresponding to the character in the first speech stream to obtain the second speech stream corresponding to the material to be analyzed. Determining the sound state that matches one or more segmented statements for each role includes: Based on the sound state description input by the user for one or more segmentation statements corresponding to each of the aforementioned roles, determine the sound state that matches the segmentation statement; and / or, Based on the semantic features, determine the sound state that matches one or more of the segmented statements corresponding to each character.

8. A speech stream synthesis device based on emotion recognition, characterized in that, The device includes: The determination module is used to determine the semantic features of the material to be analyzed based on the text content of the material to be analyzed. The material to be analyzed includes any one of the following: the text to be analyzed, the speech stream to be analyzed, and the image to be analyzed containing at least one character. The extraction module is used to extract the sentiment features of the material to be analyzed based on the semantic features; The speech generation module is used to generate a first speech stream corresponding to the material to be analyzed based on the emotional features, wherein the first speech stream is a speech stream containing the emotional features; The device further includes: The recognition module is used to identify the text information in the material to be analyzed, based on a text recognition method that matches the material to be analyzed, if the material to be analyzed is a speech stream or an image to be analyzed, and to obtain the text content of the material to be analyzed. Furthermore, the specific method by which the speech generation module generates the first speech stream corresponding to the material to be analyzed based on the emotional features includes: In the process of generating a speech stream based on the text content, the emotional features are synchronously fused into the generated speech stream to obtain the first speech stream corresponding to the material to be analyzed; or... The pre-generated initial speech stream is adjusted according to the emotional features to integrate the emotional features into the initial speech stream, thereby obtaining the first speech stream corresponding to the material to be analyzed. The initial speech stream includes the speech stream to be analyzed or a speech stream generated based on the text content. The semantic features include the global semantic features of the material to be analyzed and the local semantic features of one or more segmented sentences arranged in positional order in the text content; The extraction module is further configured to extract target keywords contained in each segmented statement before extracting the sentiment features in the material to be analyzed based on the semantic features; Furthermore, the specific method by which the extraction module extracts the sentiment features of the material to be analyzed based on the semantic features includes: Based on the global semantic features, the global sentiment features of the material to be analyzed are determined, wherein the global sentiment features include the sentiment tone of the material to be analyzed; Based on the local semantic features of each segmented statement and the target keywords contained in each segmented statement, the local sentiment features of each segmented statement are determined. The local sentiment features include sentiment degree, sentiment type, and sentiment orientation. The sentiment degree includes one or more of the following: sentiment pleasure degree, sentiment intensity degree, and sentiment controllability degree. The sentiment orientation includes introverted sentiment orientation or extroverted sentiment orientation. Based on the global sentiment features, the sentiment level in the local sentiment features of each segmented sentence is adjusted to obtain new local sentiment features for each segmented sentence, which are then used as the sentiment features of the material to be analyzed.

Images