Speech synthesis method, device, equipment and medium

Abstract

The invention belongs to the field of artificial intelligence, and relates to a speech synthesis method, which comprises the following steps: acquiring a text of a speech to be synthesized and a reference audio representing a target emotion; performing feature extraction on the reference audio by adopting an emotion coding model, and constructing a target three-dimensional emotion coordinate for describing a target emotion; performing coordinate conversion processing on the target three-dimensional emotion coordinate to obtain a target spherical emotion coordinate describing the target emotion; carrying out coding conversion on the target spherical emotion coordinates by adopting an emotion encoder to obtain a target emotion embedding vector for describing target emotion; and fusing the text and the target emotion embedding vector by adopting a speech synthesis model to generate a target synthesized speech with the semantics of the text and the target emotion. The invention further provides a device, equipment and a medium. In addition, the invention also relates to a block chain technology, and the text, the reference audio and the target synthetic voice can be stored in a block chain. According to the method and the device, more precise and flexible control on emotion expression in speech synthesis can be realized.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a speech synthesis method, apparatus, device and medium. Background Art

[0002] In the field of speech synthesis technology, the emotional speech generation model is an important technology that can generate speech with different emotions, greatly enriching the diversity and authenticity of speech interaction. There are currently many methods for emotional speech generation.

[0003] One mainstream method is to use emotion labels to control the model to generate different emotional expressions. This method models based on the complexity of emotions and uses the learned emotional features as auxiliary inputs to the speech generation model to achieve emotional control. However, this label-based method has a significant flaw, which is that it focuses on using discrete labels to express emotions, which ignores the complexity and continuity of emotions conveyed in human language, resulting in the generated emotional speech to appear too simple and stereotyped. Another method is to control the emotional style of the generated audio based on the emotional features of the reference audio. This method controls the intensity of emotion by introducing a scaling factor and the emotional embedding of the reference audio. However, this method faces many challenges in practical applications, such as the relatively difficult determination of the scaling factor, which limits the scope of application of this method.

[0004] In summary, although the existing emotion-based speech generation models have made some progress in emotion control and expression, there are still problems such as single emotion expression and complex determination of scaling factors, making it difficult to achieve more refined and flexible control of emotion expression in speech synthesis. Summary of the invention

[0005] The purpose of the embodiments of the present application is to propose a speech synthesis method, apparatus, device and medium to solve the problem that it is difficult to achieve more sophisticated and flexible control of emotional expression in speech synthesis.

[0006] In order to solve the above technical problems, the present application provides a speech synthesis method, which adopts the following technical solutions:

[0007] Acquire the text of the speech to be synthesized and the reference audio representing the target emotion; use the preset emotion coding model to extract features from the reference audio and construct target three-dimensional emotion coordinates describing the target emotion; perform coordinate transformation processing on the target three-dimensional emotion coordinates to obtain target spherical emotion coordinates describing the target emotion; use the preset emotion encoder to encode and transform the target spherical emotion coordinates to obtain a target emotion embedding vector describing the target emotion; use the preset speech synthesis model to fuse the text and the target emotion embedding vector to generate a target synthesized speech with the semantics of the text and the target emotion.

[0008] Further, the steps of using a preset emotion coding model to extract features from the reference audio and constructing a target three-dimensional emotion coordinate describing the target emotion specifically include:

[0009] The three-dimensional emotional feature values ​​of the reference audio are obtained, and the three-dimensional emotional feature values ​​include arousal value, valence value and dominance value; according to the arousal value, valence value and dominance value, a preset emotional coding model is used to construct the three-dimensional coordinates of the reference audio to obtain the target three-dimensional emotional coordinates describing the target emotion.

[0010] Further, the step of performing coordinate conversion processing on the target three-dimensional emotion coordinates to obtain the target spherical emotion coordinates describing the target emotion specifically includes:

[0011] Obtain preset neutral emotion coordinates, and set the neutral emotion coordinates as the neutral emotion origin of the preset spherical coordinate system; based on the neutral emotion origin, use the preset spherical coordinate conversion formula to convert the target three-dimensional emotion coordinates into target spherical emotion coordinates that describe the target emotion.

[0012] Further, a preset emotion encoder is used to encode and transform the target spherical emotion coordinates to obtain a target emotion embedding vector describing the target emotion, specifically including:

[0013] Calculate the distance from the target spherical emotion coordinates to the neutral emotion origin to obtain the emotion intensity value; according to the angle range formed by the target spherical emotion coordinates and the neutral emotion origin, use the preset emotion encoder to predict the emotion space to which the target spherical emotion coordinates belong; based on the emotion intensity value, project the target spherical emotion coordinates to the emotion space to obtain the target emotion style vector, target emotion intensity vector and target emotion category vector; perform fusion transformation on the target emotion style vector, target emotion intensity vector and target emotion category vector to obtain the target emotion embedding vector that describes the target emotion.

[0014] Further, the target emotion style vector, the target emotion intensity vector and the target emotion category vector are fused and transformed to obtain a target emotion embedding vector describing the target emotion, which specifically includes:

[0015] The target emotion style vector, the target emotion intensity vector and the target emotion category vector are normalized respectively to obtain a normalized emotion feature set; the vectors in the emotion feature set are concatenated to obtain a target emotion embedding vector describing the target emotion.

[0016] Furthermore, the steps of using a preset speech synthesis model to fuse the text and the target emotion embedding vector to generate a target synthesized speech with the semantics of the text and the target emotion specifically include:

[0017] The semantic information of the text is extracted, and based on the semantic information, a basic speech sequence is generated using a preset speech synthesis model; based on the target emotion embedding vector, the basic speech sequence is emotionally modulated to obtain a modulated speech sequence; the speech sequence is converted into acoustic parameters, and a target synthetic speech with the semantics of the text and the target emotion is synthesized based on the acoustic parameters.

[0018] Furthermore, before the step of using a preset emotion coding model to extract features from the reference audio to obtain target three-dimensional emotion coordinates describing the target emotion, the method further includes:

[0019] Obtain target training audio representing a reference emotion, input the target training audio into a preset initial emotion coding model, and output three-dimensional emotion coordinates representing the reference emotion; perform coordinate conversion processing on the three-dimensional emotion coordinates to obtain spherical emotion coordinates describing the reference emotion; use a preset initial emotion encoder to encode and convert the spherical emotion coordinates to obtain an emotion embedding vector describing the reference emotion; obtain target training text of the target training audio, use a preset initial speech synthesis model to fuse the target training text with the emotion embedding vector to generate synthesized speech; calculate a loss function based on the difference between the synthesized speech and the target training audio, and based on the loss function, adjust the parameters of the initial emotion coding model and the parameters of the initial speech synthesis model respectively to construct a preset emotion coding model and a preset speech synthesis model.

[0020] In order to solve the above technical problems, the embodiment of the present application further provides a speech synthesis device, which adopts the following technical solution:

[0021] An acquisition module is used to acquire the text of the speech to be synthesized and the reference audio representing the target emotion;

[0022] A feature extraction module is used to extract features from the reference audio using a preset emotion coding model, and construct a target three-dimensional emotion coordinate describing the target emotion;

[0023] A coordinate conversion module is used to perform coordinate conversion processing on the target three-dimensional emotion coordinates to obtain the target spherical emotion coordinates that describe the target emotion;

[0024] A coding conversion module is used to use a preset emotion encoder to perform coding conversion on the target spherical emotion coordinates to obtain a target emotion embedding vector describing the target emotion;

[0025] The fusion module is used to fuse the text and the target emotion embedding vector using a preset speech synthesis model to generate a target synthesized speech with the semantics of the text and the target emotion.

[0026] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the above-mentioned speech synthesis method are implemented.

[0027] In order to solve the above technical problems, an embodiment of the present application also provides a computer-readable storage medium, which stores computer-readable instructions. The computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned speech synthesis method.

[0028] Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects: by deeply analyzing the reference audio through the emotion coding model, the emotion features are accurately extracted, and then the target three-dimensional emotion coordinates are constructed, thereby realizing a preliminary quantitative description of the target emotion. On this basis, the innovative coordinate transformation process maps the three-dimensional coordinates to the spherical emotion space, further refines the division of the emotion dimension, and provides the possibility for fine control of emotion. Subsequently, the emotion encoder encodes the target spherical emotion coordinates into a target emotion embedding vector, which is used as one of the inputs of the speech synthesis model and is deeply integrated with the text of the speech to be synthesized. This process not only retains the semantic content of the text, but also incorporates delicate and flexible emotional expression into speech generation, so that the target synthesized speech can accurately convey the preset emotional color, and realize more refined and flexible control of emotional expression in speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0030] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0031] Figure 2 It is a flowchart of a speech synthesis method provided by the present application;

[0032] Figure 3 It is a structural schematic diagram of a speech synthesis device provided by the present application;

[0033] Figure 4 It is a structural schematic diagram of a computer device provided by this application. DETAILED DESCRIPTION

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0035] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0036] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0037] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0038] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0039] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0040] The server 103 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal device 101 .

[0041] It should be noted that the speech synthesis method provided in the embodiment of the present application is generally executed by a server, and accordingly, the speech synthesis device is generally arranged in the server.

[0042] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0043] Continue to refer Figure 2 , shows a flow chart of an embodiment of a speech synthesis method according to the present application. The speech synthesis method comprises the following steps:

[0044] Step S201, obtaining the text of the speech to be synthesized and the reference audio representing the target emotion.

[0045] In this embodiment, the electronic device (eg, Figure 1 The server shown in the figure can obtain the text of the speech to be synthesized and the reference audio representing the target emotion through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0046] The speech to be synthesized refers to the speech that is ultimately expected to be generated through technical means. It combines the specified text content (text) and specific emotional expression (the target emotion of the reference audio) and is the direct product of the application of speech synthesis technology.

[0047] Among them, text refers to the original text information that needs to be synthesized for speech. It contains the semantic content of the speech to be synthesized and is an indispensable source of information in the speech synthesis process.

[0048] Among them, target emotion refers to the specific emotional state that one hopes to express in synthesized speech, such as happiness, sadness, anger, etc. It is an important guide for emotional expression in synthesized speech.

[0049] The reference audio is an audio sample containing the target emotion, which is used to provide a reference for emotional expression for the synthesized speech. Through the reference audio, the characteristics of the target emotion can be extracted, and then the emotion can be reproduced in the synthesized speech.

[0050] Step S202: Using a preset emotion coding model, extracting features from the reference audio, and constructing target three-dimensional emotion coordinates describing the target emotion.

[0051] Among them, the emotion coding model is used to process and analyze emotion data. It can extract emotion information based on audio features and convert it into quantifiable emotion coordinates.

[0052] Among them, feature extraction refers to the process of extracting key information that can represent the target emotion from the reference audio. This key information usually includes the three-dimensional feature values ​​of emotion such as arousal, valence and dominance of the reference audio.

[0053] Among them, the target three-dimensional emotional coordinates are the coordinate points used to describe the target emotion constructed under the emotion coding model based on the extracted three-dimensional emotional feature values. It contains the quantitative values ​​of the target emotion in the three dimensions of arousal, valence and dominance.

[0054] Step S203, performing coordinate transformation processing on the target three-dimensional emotion coordinates to obtain target spherical emotion coordinates describing the target emotion.

[0055] Among them, coordinate conversion processing refers to the process of converting the target three-dimensional emotional coordinates from the three-dimensional coordinate system to the spherical coordinate system. This conversion helps to better understand and process the spatial distribution of emotional information.

[0056] Among them, the target spherical emotional coordinates are obtained after coordinate conversion processing, and describe the coordinate points of the target emotion in the spherical coordinate system. It is located based on the preset neutral emotional coordinates (i.e., the neutral emotional origin of the spherical coordinate system), which can more intuitively reflect the intensity and type of emotional information.

[0057] Step S204, using a preset emotion encoder to perform encoding conversion on the target spherical emotion coordinates to obtain a target emotion embedding vector describing the target emotion.

[0058] Among them, the emotion encoder is a model that can convert the target spherical emotion coordinates into emotion embedding vectors. It predicts the emotion space by analyzing the relationship between the spherical emotion coordinates and the neutral emotion origin, and encodes the emotion information into a vector form that is easy to process.

[0059] Among them, encoding conversion refers to the process of converting the target spherical emotion coordinates into the target emotion embedding vector using the emotion encoder. This conversion enables the emotion information to be represented in a more compact and easier to process form, which is convenient for subsequent application in speech synthesis.

[0060] The target emotion embedding vector is a vector representation output by the emotion encoder to describe the target emotion. It integrates information such as the style, intensity, and category of the target emotion and is the key input for achieving emotional expression in the speech synthesis process.

[0061] Step S205, using a preset speech synthesis model, fuses the text and the target emotion embedding vector to generate a target synthesized speech with the semantics of the text and the target emotion.

[0062] Among them, the speech synthesis model is a model that can generate speech with specific emotions (such as target emotions) and semantic content based on input text and emotional information.

[0063] Fusion refers to the process of combining text and target emotion embedding vectors in the speech synthesis model. Through fusion processing, it can be ensured that the generated target synthesized speech contains both the semantic content of the text and can accurately express the target emotion.

[0064] The target synthesized speech is the final output of the speech synthesis model. It integrates the semantic content of the text and the target emotion, and is a speech signal with specific emotion and semantic content. Through auditory perception, the semantic information and emotional expression can be clearly distinguished.

[0065] The embodiment of the present application can deeply analyze the reference audio through the emotion coding model, accurately extract the emotion features, and then construct the target three-dimensional emotion coordinates, thereby realizing a preliminary quantitative description of the target emotion. On this basis, the innovative coordinate transformation process maps the three-dimensional coordinates to the spherical emotion space, further refines the division of the emotion dimension, and provides the possibility for fine control of emotion. Subsequently, the emotion encoder encodes the target spherical emotion coordinates into a target emotion embedding vector, which is used as one of the inputs of the speech synthesis model and deeply integrated with the text of the speech to be synthesized. This process not only retains the semantic content of the text, but also incorporates delicate and flexible emotion expression into speech generation, so that the target synthesized speech can accurately convey the preset emotional color, and realize more refined and flexible control of emotional expression in speech synthesis.

[0066] In some optional implementations of this embodiment, step 202, using a preset emotion coding model to extract features from the reference audio and constructing a target three-dimensional emotion coordinate describing the target emotion, specifically includes:

[0067] The three-dimensional emotional feature values ​​of the reference audio are obtained, and the three-dimensional emotional feature values ​​include arousal value, valence value and dominance value; according to the arousal value, valence value and dominance value, a preset emotional coding model is used to construct the three-dimensional coordinates of the reference audio to obtain the target three-dimensional emotional coordinates describing the target emotion.

[0068] In one example, first, the reference audio is preprocessed by an audio processing module to extract its emotional three-dimensional feature values. Specifically, advanced audio signal processing techniques, such as short-time Fourier transform and Mel-frequency cepstral coefficients, can be used to analyze the spectral characteristics of the audio signal of the reference audio, and then calculate the arousal value, valence value, and dominance value. The arousal value reflects the excitement or calmness of the reference audio, the valence value indicates the positive or negative emotional tendency of the reference audio, and the dominance value reflects the sense of control or obedience in the reference audio. Subsequently, based on the extracted arousal, valence, and dominance values, the three-dimensional coordinates of the reference audio are constructed using a pre-trained emotional coding model to obtain the target three-dimensional emotional coordinates that describe the target emotion. The emotional coding model is trained based on a large amount of audio data labeled with emotional dimensions, and can accurately map emotional features into three-dimensional space.

[0069] The embodiment of the present application can achieve comprehensive quantification of the emotional characteristics of the reference audio by accurately extracting the three-dimensional characteristic values ​​of the reference audio (arousal value, valence value and dominance value). On this basis, the characteristic values ​​of these three dimensions are converted into three-dimensional coordinates using a preset emotion coding model to construct a target three-dimensional emotion coordinate that describes the target emotion. This process not only improves the accuracy of the emotion description, but also allows the emotional information to be intuitively presented in three-dimensional space, which is convenient for subsequent emotion analysis and processing, and can achieve detailed characterization and precise positioning of the reference audio emotion.

[0070] In some optional implementations of this embodiment, step 203, performing coordinate conversion processing on the target three-dimensional emotion coordinates to obtain the target spherical emotion coordinates describing the target emotion, specifically includes the following steps:

[0071] Obtain preset neutral emotion coordinates, and set the neutral emotion coordinates as the neutral emotion origin of the preset spherical coordinate system; based on the neutral emotion origin, use the preset spherical coordinate conversion formula to convert the target three-dimensional emotion coordinates into target spherical emotion coordinates that describe the target emotion.

[0072] In one example, first, the three dimensional features of emotion in speech, including arousal, valence, and dominance, are extracted using a pre-trained emotion coding model. The emotion coding model is a deep neural network model that can accurately predict the emotion coordinates of each speech segment by training a large amount of speech data labeled with emotion dimensions. a ,d d ,d v ), where d a d d and d v Represent the values ​​of arousal, valence, and dominance, all ranging from 0 to 1. Next, obtain the preset neutral emotion coordinates, which represent a neutral state without specific emotional tendencies and are the reference point for subsequent emotion encoding. In this example, the neutral emotion coordinates are set to the neutral emotion origin of the preset spherical coordinate system, that is, the center of the spherical coordinate system M (0,0,0). After obtaining the three-dimensional coordinates of the target emotion, the preset spherical coordinate conversion formula is used to convert the target three-dimensional emotion coordinates into target spherical emotion coordinates that describe the target emotion. The conversion process is as follows: First, taking the neutral emotion origin M as the reference, calculate the vector distance r between the target emotion coordinate e and the origin M, as well as the two angles θ and Specifically, it can be calculated by the following formula:

[0073]

[0074] Among them, in practical applications, when calculating r, θ and Finally, in order to standardize the intensity of emotion, r is scaled to the range of 0 to 1 to obtain r'. This step can be achieved through a simple linear transformation. By scaling the distance r from the neutral emotion origin to the range of 0 to 1, the emotional intensity is standardized, which helps to improve the accuracy and consistency of sentiment analysis. Finally, the target spherical emotional coordinates (r', θ, ), through two angles θ and The value of can divide the entire emotional space into eight parts, providing a more refined division for emotional classification and recognition. The target spherical emotional coordinates not only retain the three-dimensional information of the original emotional coordinates, but also make the emotional features more intuitive and easy to analyze in space through the representation of the spherical coordinate system.

[0075] The embodiment of the present application can achieve accurate modeling of emotional features in a three-dimensional spherical coordinate system by introducing preset neutral emotional coordinates as the neutral emotional origin of the spherical coordinate system. The scheme uses a pre-trained emotional coding model to effectively extract the three-dimensional characteristics of emotions in the reference audio, namely arousal, valence, and dominance, providing a basis for the quantitative analysis of emotions. Based on the neutral emotional origin, the preset spherical coordinate conversion formula is used to convert the target three-dimensional emotional coordinates into target spherical emotional coordinates that describe the target emotion. This conversion process not only retains the three-dimensional information of the emotional features, but also makes the emotional features more intuitive and easy to analyze in space through the representation of the spherical coordinate system.

[0076] In some optional implementations of this embodiment, step S204, using a preset emotion encoder to encode and convert the target spherical emotion coordinates to obtain a target emotion embedding vector describing the target emotion, specifically includes the following steps:

[0077] Calculate the distance from the target spherical emotion coordinates to the neutral emotion origin to obtain the emotion intensity value; according to the angle range formed by the target spherical emotion coordinates and the neutral emotion origin, use the preset emotion encoder to predict the emotion space to which the target spherical emotion coordinates belong; based on the emotion intensity value, project the target spherical emotion coordinates to the emotion space to obtain the target emotion style vector, target emotion intensity vector and target emotion category vector; perform fusion transformation on the target emotion style vector, target emotion intensity vector and target emotion category vector to obtain the target emotion embedding vector that describes the target emotion.

[0078] In one example, first, the distance from the target spherical emotion coordinates (x, y, z) to the neutral emotion origin (0, 0, 0) can be calculated, and this distance is the emotion intensity value r. The emotion intensity value reflects the degree of deviation between the target emotion and the neutral emotion, and its calculation formula is:

[0079]

[0080] Next, the preset emotion encoder can be used to predict the emotion space to which the target spherical emotion coordinates belong according to the angle range formed by the target spherical emotion coordinates and the neutral emotion origin. The emotion space is divided according to the angle range of the emotion dimensions (such as arousal, valence and dominance), and each emotion space corresponds to one or more emotion categories. Eight emotion spaces can be set, corresponding to different emotion categories. The emotion encoder maps the target spherical emotion coordinates to the corresponding emotion space according to the angle range. Based on the emotion intensity value r, the target spherical emotion coordinates are projected to the predicted emotion space to obtain the target emotion style vector, the target emotion intensity vector and the target emotion category vector. These three vectors describe the style characteristics, intensity characteristics and category characteristics of the target emotion respectively. Finally, the target emotion style vector, the target emotion intensity vector and the target emotion category vector are fused and transformed to obtain the target emotion embedding vector describing the target emotion. For example, taking a certain reference audio as an example, its emotion dimension can be extracted and the target spherical emotion coordinates (0.5, 0.3, -0.4) can be calculated. According to the distance between the coordinate and the neutral emotion origin, the emotion intensity value r = 0.74 is obtained. Then, the emotion encoder is used to predict that the coordinate belongs to the "sad" emotion space. Next, the target spherical emotion coordinates are projected into the emotion space to obtain the target emotion style vector, target emotion intensity vector, and target emotion category vector. Finally, these three vectors are fused and transformed to obtain the target emotion embedding vector that describes the target emotion of the speech segment.

[0081] The embodiment of the present application can accurately quantify the emotional intensity value by calculating the distance from the target spherical emotional coordinates to the neutral emotional origin, thereby providing an objective basis for emotional analysis. By utilizing the angle range formed by the target spherical emotional coordinates and the neutral emotional origin, combined with a preset emotional encoder, accurate prediction of the emotional space is achieved, laying a solid foundation for emotional classification. At the same time, based on the emotional intensity value, the target spherical emotional coordinates are projected to the corresponding emotional space, which not only obtains the target emotional style vector, target emotional intensity vector, and target emotional category vector, but also comprehensively describes the multi-dimensional characteristics of emotions. Finally, through the fusion transformation of these three vectors, a highly generalized target emotional embedding vector rich in emotional information is generated, which effectively improves the accuracy and efficiency of emotion recognition and analysis.

[0082] In some optional implementations of this embodiment, the step of “performing a fusion transformation on the target emotion style vector, the target emotion intensity vector, and the target emotion category vector to obtain a target emotion embedding vector describing the target emotion” specifically includes the following steps:

[0083] The target emotion style vector, the target emotion intensity vector and the target emotion category vector are normalized respectively to obtain a normalized emotion feature set; the vectors in the emotion feature set are concatenated to obtain a target emotion embedding vector describing the target emotion.

[0084] In one example, the min-max normalization method can be used to normalize the three emotional feature vectors of the target emotional style vector, the target emotional intensity vector, and the target emotional category vector. The purpose of normalization is to unify the feature vectors with different value ranges into the range of [0,1] for subsequent vector splicing. After obtaining the normalized emotional feature set, the vectors in the emotional feature set can be spliced ​​to obtain the target emotional embedding vector that describes the target emotion. Specifically, the vectors in the emotional feature set can be connected end to end to form a new long vector (target emotional embedding vector).

[0085] The embodiment of the present application can effectively unify the magnitude of emotional features by normalizing the target emotional style vector, target emotional intensity vector and target emotional category vector, which provides convenience for subsequent processing. The normalized emotional feature set is concatenated to form a target emotional embedding vector, which integrates the style, intensity and category information of the emotion, and realizes efficient integration of emotional features.

[0086] In some optional implementations of this embodiment, step S205 uses a preset speech synthesis model to fuse the text and the target emotion embedding vector to generate a target synthesized speech with the semantics of the text and the target emotion, which specifically includes the following steps:

[0087] The semantic information of the text is extracted, and based on the semantic information, a basic speech sequence is generated using a preset speech synthesis model; based on the target emotion embedding vector, the basic speech sequence is emotionally modulated to obtain a modulated speech sequence; the speech sequence is converted into acoustic parameters, and a target synthetic speech with the semantics of the text and the target emotion is synthesized based on the acoustic parameters.

[0088] In one example, the semantic information of the text can be extracted by natural language processing technology. Then, based on the semantic information, a preset speech synthesis model (such as Tacotron, FastSpeech, etc.) is used to generate a basic speech sequence. The basic speech sequence contains the speech content of the text, but no emotional features have been added. The target emotional embedding vector is input into the emotional modulation module, which can be a network structure based on deep learning, which is used to emotionally modulate the basic speech sequence. The modulated speech sequence will contain the text content and the target emotional features. The modulated speech sequence is converted into acoustic parameters, which may include fundamental frequency, formant, timbre, etc. Then, the target synthesized speech with text semantics and target emotion is synthesized based on the acoustic parameters. For example, suppose that a financial speech expressing positive emotions needs to be synthesized: "Today, A shares rose 5%." First, a reference audio expressing positive emotions can be obtained. Then, features are extracted and three-dimensional emotional coordinates are constructed through the emotional encoding model. Next, the three-dimensional coordinates are converted into spherical coordinates, and the emotional embedding vector is obtained through the emotional encoder. After extracting the text semantic information, the speech synthesis model is used to generate the basic speech sequence. Finally, through emotion modulation and acoustic parameter synthesis, a financial voice with positive emotion can be obtained: "Today's A-shares rose 5%". This financial voice not only accurately conveys the text information, but also expresses positive emotional colors, improving the user experience.

[0089] The embodiment of the present application can accurately generate a basic speech sequence that matches the text content by deeply mining the text semantic information and adopting an advanced preset speech synthesis model accordingly. On this basis, the target emotion embedding vector is innovatively introduced to implement fine emotion modulation on the basic speech sequence, ensuring that the speech not only accurately conveys the text semantics, but also perfectly integrates the target emotional color, significantly improving the authenticity and appeal of the speech expression. Through precise acoustic parameter conversion and synthesis technology, the modulated speech sequence is converted into high-quality acoustic features, and then synthesized speech that contains both rich text semantics and target emotions is synthesized.

[0090] In some optional implementations of this embodiment, in step S205, after the preset emotion coding model is used to extract features from the reference audio to obtain the target three-dimensional emotion coordinates describing the target emotion, the following steps are specifically included:

[0091] Obtain target training audio representing a reference emotion, input the target training audio into a preset initial emotion coding model, and output three-dimensional emotion coordinates representing the reference emotion; perform coordinate conversion processing on the three-dimensional emotion coordinates to obtain spherical emotion coordinates describing the reference emotion; use a preset initial emotion encoder to encode and convert the spherical emotion coordinates to obtain an emotion embedding vector describing the reference emotion; obtain target training text of the target training audio, use a preset initial speech synthesis model to fuse the target training text with the emotion embedding vector to generate synthesized speech; calculate a loss function based on the difference between the synthesized speech and the target training audio, and based on the loss function, adjust the parameters of the initial emotion coding model and the parameters of the initial speech synthesis model respectively to construct a preset emotion coding model and a preset speech synthesis model.

[0092] In one example, first, target training audio representing reference emotions can be obtained. These target training audios should cover a variety of emotional expressions, such as happiness, sadness, anger, etc. At the same time, target training texts corresponding to the target training audios are obtained. These target training texts should match the audio content for subsequent speech synthesis. Next, a pre-trained initial emotion encoding model is used to extract the features of the three dimensions of emotion in the target training audio, namely, arousal, valence, and dominance. The initial emotion encoding model can predict a three-dimensional coordinate vector e=(d a ,d d ,d v ), where d a d d and d v They represent arousal, valence and dominance respectively, and their values ​​are all between 0 and 1. Next, the extracted three-dimensional emotion coordinates are transformed to obtain the spherical emotion coordinates describing the reference emotion. The specific transformation process is as follows:

[0093] First, the difference between each emotion coordinate and the neutral emotion center M is calculated to obtain the distance r from the neutral emotion origin. Then, the spherical coordinates θ and Component. Finally, the distance r from the neutral emotion origin is scaled to the range of 0 to 1 to standardize the intensity of the emotion. Then, the preset initial emotion encoder is used to encode and transform the spherical emotion coordinates to obtain the emotion embedding vector that describes the reference emotion. This vector will be used as an auxiliary input for the subsequent speech synthesis model to control the emotional expression of the generated speech. Then, the preset initial speech synthesis model (such as Tacotron, FastSpeech, etc.) is used to fuse the target training text with the emotion embedding vector. The fusion process involves steps such as normalization and splicing of the vector. Then, synthesized speech is generated, and the loss function is calculated based on the difference between the synthesized speech and the target training audio. According to the calculated loss function, the parameters of the initial emotion encoding model and the initial speech synthesis model are adjusted respectively. Through multiple iterative training, the model parameters are gradually optimized until the preset training effect is achieved. After the training is completed, the preset emotion encoding model and the preset speech synthesis model can be applied to actual scenarios.

[0094] The embodiment of the present application can achieve high-quality speech synthesis with controllable emotions by deeply integrating emotion coding and speech synthesis technology. First, the three-dimensional emotion coordinates of the target training audio are accurately extracted using the initial emotion coding model, and mapped to spherical emotion coordinates through clever coordinate transformation, which effectively improves the delicacy and accuracy of emotional expression. Subsequently, the emotion encoder is used to efficiently encode the spherical emotion coordinates to generate an emotion embedding vector, which can accurately capture and express complex emotional features. In the speech synthesis stage, by fusing the emotion embedding vector with the target training text, the initial speech synthesis model can generate emotion-rich synthesized speech. Finally, based on the difference between the synthesized speech and the target training audio, the emotion coding model and speech synthesis model are constructed by accurately calculating the loss function and adjusting the model parameters. The emotion coding model and speech synthesis model not only have powerful emotional expression capabilities, but also ensure the natural fluency and high fidelity of the speech.

[0095] It should be emphasized that in order to further ensure the privacy and security of the text of the speech to be synthesized, the reference audio and the target synthesized speech, the text of the speech to be synthesized, the reference audio and the target synthesized speech can also be stored in a node of a blockchain.

[0096] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0097] The embodiments of the present application can build and optimize related models and networks based on artificial intelligence technology, such as initial emotion encoders, emotion encoding models, initial speech synthesis models, and speech synthesis models. Among them, artificial intelligence (AI) models are the crystallization of theory and practice that simulates the decision-making process of human intelligence through algorithms and data analysis to solve complex problems, predict future trends, or realize automation tasks. These models use a large amount of historical data and real-time information, and are trained and optimized through a specific algorithm framework to achieve efficient, accurate, and reliable performance.

[0098] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0099] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0100] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a speech synthesis device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0101] like Figure 3 As shown, the speech synthesis device 400 of this embodiment includes: an acquisition module 401, a feature extraction module 402, a coordinate conversion module 403, a coding conversion module 404 and a fusion module 405. Among them:

[0102] An acquisition module 401 is used to acquire the text of the speech to be synthesized and the reference audio representing the target emotion;

[0103] The feature extraction module 402 is used to extract features from the reference audio using a preset emotion coding model, and construct a target three-dimensional emotion coordinate describing the target emotion;

[0104] A coordinate conversion module 403 is used to perform coordinate conversion processing on the target three-dimensional emotion coordinates to obtain the target spherical emotion coordinates describing the target emotion;

[0105] The encoding conversion module 404 is used to use a preset emotion encoder to perform encoding conversion on the target spherical emotion coordinates to obtain a target emotion embedding vector describing the target emotion;

[0106] The fusion module 405 is used to use a preset speech synthesis model to fuse the text and the target emotion embedding vector to generate a target synthesized speech with the semantics of the text and the target emotion.

[0107] The embodiment of the present application can deeply analyze the reference audio through the emotion coding model, accurately extract the emotion features, and then construct the target three-dimensional emotion coordinates, thereby realizing a preliminary quantitative description of the target emotion. On this basis, the innovative coordinate transformation process maps the three-dimensional coordinates to the spherical emotion space, further refines the division of the emotion dimension, and provides the possibility for fine control of emotion. Subsequently, the emotion encoder encodes the target spherical emotion coordinates into a target emotion embedding vector, which is used as one of the inputs of the speech synthesis model and deeply integrated with the text of the speech to be synthesized. This process not only retains the semantic content of the text, but also incorporates delicate and flexible emotion expression into speech generation, so that the target synthesized speech can accurately convey the preset emotional color, and realize more refined and flexible control of emotional expression in speech synthesis.

[0108] In one embodiment, the feature extraction module 402 includes:

[0109] A value acquisition submodule is used to obtain the three-dimensional characteristic values ​​of emotion of the reference audio, and the three-dimensional characteristic values ​​of emotion include arousal value, valence value and dominance value;

[0110] The submodule is used to construct the three-dimensional coordinates of the reference audio according to the arousal value, valence value and dominance value, using a preset emotion coding model to obtain the target three-dimensional emotion coordinates describing the target emotion.

[0111] The embodiment of the present application can achieve comprehensive quantification of the emotional characteristics of the reference audio by accurately extracting the three-dimensional characteristic values ​​of the reference audio (arousal value, valence value and dominance value). On this basis, the characteristic values ​​of these three dimensions are converted into three-dimensional coordinates using a preset emotion coding model to construct a target three-dimensional emotion coordinate that describes the target emotion. This process not only improves the accuracy of the emotion description, but also allows the emotional information to be intuitively presented in three-dimensional space, which is convenient for subsequent emotion analysis and processing, and can achieve detailed characterization and precise positioning of the reference audio emotion.

[0112] In one embodiment, the coordinate conversion module 403 includes:

[0113] A coordinate acquisition submodule, used to acquire preset neutral emotion coordinates and set the neutral emotion coordinates as the neutral emotion origin of a preset spherical coordinate system;

[0114] The coordinate conversion submodule is used to convert the target three-dimensional emotion coordinates into target spherical emotion coordinates describing the target emotion based on the neutral emotion origin and using a preset spherical coordinate conversion formula.

[0115] The embodiment of the present application can achieve accurate modeling of emotional features in a three-dimensional spherical coordinate system by introducing preset neutral emotional coordinates as the neutral emotional origin of the spherical coordinate system. The scheme uses a pre-trained emotional coding model to effectively extract the three-dimensional characteristics of emotions in the reference audio, namely arousal, valence, and dominance, providing a basis for the quantitative analysis of emotions. Based on the neutral emotional origin, the preset spherical coordinate conversion formula is used to convert the target three-dimensional emotional coordinates into target spherical emotional coordinates that describe the target emotion. This conversion process not only retains the three-dimensional information of the emotional features, but also makes the emotional features more intuitive and easy to analyze in space through the representation of the spherical coordinate system.

[0116] In one embodiment, the encoding conversion module 404 includes:

[0117] The calculation submodule is used to calculate the distance from the target spherical emotion coordinates to the neutral emotion origin to obtain the emotion intensity value;

[0118] A prediction submodule, for predicting the emotional space to which the target spherical emotional coordinates belong based on the angle range formed by the target spherical emotional coordinates and the neutral emotional origin, using a preset emotional encoder;

[0119] A projection submodule is used to project the target spherical emotion coordinates to the emotion space based on the emotion intensity value to obtain a target emotion style vector, a target emotion intensity vector and a target emotion category vector;

[0120] The fusion submodule is used to fuse the target emotion style vector, the target emotion intensity vector and the target emotion category vector to obtain a target emotion embedding vector describing the target emotion.

[0121] The embodiment of the present application can accurately quantify the emotional intensity value by calculating the distance from the target spherical emotional coordinates to the neutral emotional origin, thereby providing an objective basis for emotional analysis. By utilizing the angle range formed by the target spherical emotional coordinates and the neutral emotional origin, combined with a preset emotional encoder, accurate prediction of the emotional space is achieved, laying a solid foundation for emotional classification. At the same time, based on the emotional intensity value, the target spherical emotional coordinates are projected to the corresponding emotional space, which not only obtains the target emotional style vector, target emotional intensity vector, and target emotional category vector, but also comprehensively describes the multi-dimensional characteristics of emotions. Finally, through the fusion transformation of these three vectors, a highly generalized target emotional embedding vector rich in emotional information is generated, which effectively improves the accuracy and efficiency of emotion recognition and analysis.

[0122] In one embodiment, the fusion submodule is also used to normalize the target emotion style vector, the target emotion intensity vector and the target emotion category vector respectively to obtain a normalized emotion feature set; and perform vector concatenation on the vectors in the emotion feature set to obtain a target emotion embedding vector that describes the target emotion.

[0123] The embodiment of the present application can effectively unify the magnitude of emotional features by normalizing the target emotional style vector, target emotional intensity vector and target emotional category vector, which provides convenience for subsequent processing. The normalized emotional feature set is concatenated to form a target emotional embedding vector, which integrates the style, intensity and category information of the emotion, and realizes efficient integration of emotional features.

[0124] In one embodiment, the fusion module 405 includes:

[0125] The sequence generation submodule is used to extract the semantic information of the text and generate a basic speech sequence based on the semantic information using a preset speech synthesis model;

[0126] A modulation submodule is used to perform emotion modulation on the basic speech sequence based on the target emotion embedding vector to obtain a modulated speech sequence;

[0127] The synthesis submodule is used to convert the speech sequence into acoustic parameters, and synthesize the target synthesized speech with the semantics and target emotion of the text based on the acoustic parameters.

[0128] The embodiment of the present application can accurately generate a basic speech sequence that matches the text content by deeply mining the text semantic information and adopting an advanced preset speech synthesis model accordingly. On this basis, the target emotion embedding vector is innovatively introduced to implement fine emotion modulation on the basic speech sequence, ensuring that the speech not only accurately conveys the text semantics, but also perfectly integrates the target emotional color, significantly improving the authenticity and appeal of the speech expression. Through precise acoustic parameter conversion and synthesis technology, the modulated speech sequence is converted into high-quality acoustic features, and then synthesized speech that contains both rich text semantics and target emotions is synthesized.

[0129] In one embodiment, the speech synthesis apparatus 400 further includes:

[0130] An audio acquisition module is used to acquire a target training audio representing a reference emotion, input the target training audio into a preset initial emotion encoding model, and output a three-dimensional emotion coordinate representing the reference emotion;

[0131] A conversion processing module, used for performing coordinate conversion processing on the three-dimensional emotion coordinates to obtain spherical emotion coordinates describing the reference emotion;

[0132] A conversion module, used to use a preset initial emotion encoder to encode and convert the spherical emotion coordinates to obtain an emotion embedding vector describing the reference emotion;

[0133] A text acquisition module is used to acquire a target training text of a target training audio, and to fuse the target training text with the emotion embedding vector using a preset initial speech synthesis model to generate a synthesized speech;

[0134] The calculation module is used to calculate the loss function based on the difference between the synthesized speech and the target training audio, and adjust the parameters of the initial emotion coding model and the parameters of the initial speech synthesis model based on the loss function to construct a preset emotion coding model and a preset speech synthesis model.

[0135] The embodiment of the present application can achieve high-quality speech synthesis with controllable emotions by deeply integrating emotion coding and speech synthesis technology. First, the three-dimensional emotion coordinates of the target training audio are accurately extracted using the initial emotion coding model, and mapped to spherical emotion coordinates through clever coordinate transformation, which effectively improves the delicacy and accuracy of emotional expression. Subsequently, the emotion encoder is used to efficiently encode the spherical emotion coordinates to generate an emotion embedding vector, which can accurately capture and express complex emotional features. In the speech synthesis stage, by fusing the emotion embedding vector with the target training text, the initial speech synthesis model can generate emotion-rich synthesized speech. Finally, based on the difference between the synthesized speech and the target training audio, the emotion coding model and speech synthesis model are constructed by accurately calculating the loss function and adjusting the model parameters. The emotion coding model and speech synthesis model not only have powerful emotional expression capabilities, but also ensure the natural fluency and high fidelity of the speech.

[0136] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0137] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 6 having a memory 61, a processor 62, and a network interface 63, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0138] Computer devices can be computing devices such as desktop computers, notebooks, PDAs, and cloud servers. Computer devices can interact with users through keyboards, mice, remote controls, touch pads, or voice control devices.

[0139] The memory 61 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 61 can be an internal storage unit of the computer device 6, such as a hard disk or memory of the computer device 6. In other embodiments, the memory 61 can also be an external storage device of the computer device 6, such as a plug-in hard disk equipped on the computer device 6, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 61 can also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions of the speech synthesis method, etc. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or are to be output.

[0140] The processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to run computer-readable instructions stored in the memory 61 or process data, such as computer-readable instructions for running a speech synthesis method.

[0141] The network interface 63 may include a wireless network interface or a wired network interface, and the network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0142] The embodiment of the present application can deeply analyze the reference audio through the emotion coding model, accurately extract the emotion features, and then construct the target three-dimensional emotion coordinates, thereby realizing a preliminary quantitative description of the target emotion. On this basis, the innovative coordinate transformation process maps the three-dimensional coordinates to the spherical emotion space, further refines the division of the emotion dimension, and provides the possibility for fine control of emotion. Subsequently, the emotion encoder encodes the target spherical emotion coordinates into a target emotion embedding vector, which is used as one of the inputs of the speech synthesis model and deeply integrated with the text of the speech to be synthesized. This process not only retains the semantic content of the text, but also incorporates delicate and flexible emotion expression into speech generation, so that the target synthesized speech can accurately convey the preset emotional color, and realize more refined and flexible control of emotional expression in speech synthesis.

[0143] The present application also provides another implementation, namely, providing a computer-readable storage medium, which stores computer-readable instructions. The computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the speech synthesis method as described above.

[0144] The embodiment of the present application can deeply analyze the reference audio through the emotion coding model, accurately extract the emotion features, and then construct the target three-dimensional emotion coordinates, thereby realizing a preliminary quantitative description of the target emotion. On this basis, the innovative coordinate transformation process maps the three-dimensional coordinates to the spherical emotion space, further refines the division of the emotion dimension, and provides the possibility for fine control of emotion. Subsequently, the emotion encoder encodes the target spherical emotion coordinates into a target emotion embedding vector, which is used as one of the inputs of the speech synthesis model and deeply integrated with the text of the speech to be synthesized. This process not only retains the semantic content of the text, but also incorporates delicate and flexible emotion expression into speech generation, so that the target synthesized speech can accurately convey the preset emotional color, and realize more refined and flexible control of emotional expression in speech synthesis.

[0145] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0146] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

[0147] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A speech synthesis method, characterized in that: The steps include: Obtain the text of the speech to be synthesized and the reference audio representing the target emotion; Using a preset emotion coding model, extracting features from the reference audio, and constructing a target three-dimensional emotion coordinate describing the target emotion; Performing coordinate conversion processing on the target three-dimensional emotion coordinates to obtain target spherical emotion coordinates that describe the target emotion; Using a preset emotion encoder, encoding and converting the target spherical emotion coordinates to obtain a target emotion embedding vector describing the target emotion; The text and the target emotion embedding vector are fused by using a preset speech synthesis model to generate a target synthesized speech having the semantics of the text and the target emotion.

2. The method according to claim 1, characterized in that The step of using a preset emotion coding model to extract features from the reference audio and constructing a target three-dimensional emotion coordinate describing the target emotion specifically includes: Acquire the emotional three-dimensional feature value of the reference audio, wherein the emotional three-dimensional feature value includes an arousal value, a valence value, and a dominance value; According to the arousal value, the valence value and the dominance value, a preset emotion coding model is used to construct the three-dimensional coordinates of the reference audio to obtain the target three-dimensional emotion coordinates that describe the target emotion.

3. The method according to claim 1, characterized in that The step of performing coordinate conversion processing on the target three-dimensional emotion coordinates to obtain the target spherical emotion coordinates describing the target emotion specifically includes: Obtaining preset neutral emotion coordinates, and setting the neutral emotion coordinates as the neutral emotion origin of a preset spherical coordinate system; Based on the neutral emotion origin, a preset spherical coordinate conversion formula is used to convert the target three-dimensional emotion coordinates into target spherical emotion coordinates that describe the target emotion.

4. The method according to claim 3, characterized in that The step of using a preset emotion encoder to encode and convert the target spherical emotion coordinates to obtain a target emotion embedding vector describing the target emotion specifically includes: Calculate the distance from the target spherical emotion coordinates to the neutral emotion origin to obtain an emotion intensity value; According to the angle range formed by the target spherical emotional coordinates and the neutral emotional origin, a preset emotional encoder is used to predict the emotional space to which the target spherical emotional coordinates belong; Based on the emotion intensity value, projecting the target spherical emotion coordinates to the emotion space to obtain a target emotion style vector, a target emotion intensity vector, and a target emotion category vector; The target emotion style vector, the target emotion intensity vector and the target emotion category vector are fused and transformed to obtain a target emotion embedding vector describing the target emotion.

5. The method according to claim 4, characterized in that The step of fusing and transforming the target emotion style vector, the target emotion intensity vector, and the target emotion category vector to obtain a target emotion embedding vector describing the target emotion specifically includes: Normalizing the target emotion style vector, the target emotion intensity vector, and the target emotion category vector respectively to obtain a normalized emotion feature set; Vector concatenation is performed on the vectors in the emotion feature set to obtain a target emotion embedding vector describing the target emotion.

6. The method according to claim 1, characterized in that The step of adopting a preset speech synthesis model to fuse the text with the target emotion embedding vector to generate a target synthesized speech having the semantics of the text and the target emotion specifically includes: Extracting semantic information of the text, and based on the semantic information, using a preset speech synthesis model to generate a basic speech sequence; Based on the target emotion embedding vector, the basic speech sequence is subjected to emotion modulation to obtain a modulated speech sequence; The speech sequence is converted into acoustic parameters, and a target synthesized speech having the semantics of the text and the target emotion is synthesized based on the acoustic parameters.

7. The method according to claim 1, characterized in that Before the step of using a preset emotion coding model to extract features from the reference audio to obtain target three-dimensional emotion coordinates describing the target emotion, the method further includes: Acquire target training audio representing a reference emotion, input the target training audio into a preset initial emotion coding model, and output three-dimensional emotion coordinates representing the reference emotion; Performing coordinate conversion processing on the three-dimensional emotion coordinates to obtain spherical emotion coordinates describing the reference emotion; Using a preset initial emotion encoder, the spherical emotion coordinates are encoded and converted to obtain an emotion embedding vector describing the reference emotion; Acquire a target training text of the target training audio, and use a preset initial speech synthesis model to fuse the target training text with the emotion embedding vector to generate a synthesized speech; Based on the difference between the synthesized speech and the target training audio, a loss function is calculated, and based on the loss function, the parameters of the initial emotion coding model and the parameters of the initial speech synthesis model are adjusted respectively to construct the preset emotion coding model and the preset speech synthesis model.

8. A speech synthesis device, characterized in that: include: An acquisition module is used to acquire the text of the speech to be synthesized and the reference audio representing the target emotion; A feature extraction module, used to extract features from the reference audio using a preset emotion coding model, and construct a target three-dimensional emotion coordinate describing the target emotion; A coordinate conversion module, used for performing coordinate conversion processing on the target three-dimensional emotion coordinates to obtain target spherical emotion coordinates describing the target emotion; A coding conversion module, used to use a preset emotion encoder to perform coding conversion on the target spherical emotion coordinates to obtain a target emotion embedding vector describing the target emotion; The fusion module is used to use a preset speech synthesis model to fuse the text with the target emotion embedding vector to generate a target synthesized speech with the semantics of the text and the target emotion.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the speech synthesis method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the speech synthesis method according to any one of claims 1 to 7 are implemented.

Images