Speech synthesis method, device, electronic device, computer-readable storage medium, and computer program product

Abstract

The present application provides a speech synthesis method, apparatus, electronic device, computer program product, and computer-readable storage medium. The method includes: determining a phoneme sequence of a first text and a second text, the second text being used to describe the attributes of a first speech; performing feature extraction on the phoneme sequence to obtain phoneme features, and performing feature extraction on the second text to obtain first attribute features; based on the phoneme features, performing attention adjustment on the first attribute features to obtain second attribute features; fusing the second attribute features and the phoneme features to obtain fused features; performing spectrum prediction based on the fused features to obtain a predicted spectrum, and performing speech synthesis based on the predicted spectrum to obtain a first speech corresponding to the first text. Through the present application, refined control of the attributes of the first speech can be achieved, thereby improving the accuracy of speech synthesis.

Description

Technical Field

[0001] The present application relates to the field of speech synthesis technology, and in particular to a speech synthesis method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] Text-to-speech (TTS), also known as text-to-speech technology, is a computer-generated technique that converts arbitrary text into fluent speech through analysis. As one of the core technologies for human-computer language interaction systems, speech synthesis is gaining increasing attention. As speech synthesis technology is applied across various industries and scenarios, the synthesized speech needs to possess different properties, such as emotional and pronunciation characteristics. Therefore, controlling these properties of speech synthesis is a key research topic. Summary of the Invention

[0003] The embodiments of the present application provide a speech synthesis method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can integrate attribute information of a first speech into the first speech, achieve refined control of the attributes of the first speech, and improve the accuracy of speech synthesis.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present invention provides a method for speech synthesis, comprising:

[0006] Determine a phoneme sequence of a first text and a second text, wherein the second text is used to describe a second text of a first speech attribute; perform feature extraction on the phoneme sequence to obtain phoneme features; perform feature extraction on the second text to obtain first attribute features; based on the phoneme features, perform attention adjustment on the first attribute features to obtain second attribute features; perform fusion processing on the second attribute features and the phoneme features to obtain fusion features; perform spectrum prediction based on the fusion features to obtain a predicted spectrum, and perform speech synthesis based on the predicted spectrum to obtain a first speech corresponding to the first text.

[0007] The present invention provides a speech synthesis device, comprising:

[0008] a determination module, configured to determine a phoneme sequence of a first text and a second text for describing a first speech attribute;

[0009] a feature extraction module, configured to perform feature extraction on the phoneme sequence to obtain phoneme features, and perform feature extraction on the second text to obtain first attribute features;

[0010] an adjustment module, configured to perform attention adjustment on the first attribute feature based on the phoneme feature to obtain a second attribute feature;

[0011] a fusion module, configured to fuse the second attribute feature and the phoneme feature to obtain a fusion feature;

[0012] A synthesis module is used to perform spectrum prediction based on the fusion feature to obtain a predicted spectrum, and perform speech synthesis based on the predicted spectrum to obtain a first speech corresponding to the first text.

[0013] An embodiment of the present application provides an electronic device, comprising:

[0014] a memory for storing computer-executable instructions;

[0015] The processor is used to implement the language synthesis method provided in the embodiment of the present application when executing the computer-executable instructions stored in the memory.

[0016] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the speech synthesis method provided in the embodiment of the present application when executed by a processor.

[0017] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the speech synthesis method provided in the embodiment of the present application is implemented.

[0018] The embodiments of the present application have the following beneficial effects:

[0019] Through the above embodiment, when synthesizing speech for a first text, the phoneme sequence of the first text and the second text used to describe the attributes of the first speech are obtained, and feature extraction is performed on the phoneme sequence and the second text respectively to obtain phoneme features and first attribute features; then, the attention of the first attribute features is adjusted using the phoneme features, and the adjusted second attribute features are fused with the phoneme features to obtain fused features including the phoneme information and the attribute information described by the second text; then, the fused features are used to perform spectrum prediction to obtain a predicted spectrum, and the first speech corresponding to the first text and conforming to the attributes described by the second text is generated through the predicted spectrum. In this way, by utilizing attention adjustment and fusion processing, the attribute information described by the second text can be integrated into the phoneme features to synthesize a first speech conforming to the attributes described by the second text, thereby achieving refined control of the attributes of the first speech through the second text and improving the accuracy of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1This is a schematic diagram of the structure of the speech synthesis system architecture provided by an embodiment of the present application;

[0021] Figure 2 is a structural diagram of an electronic device provided in an embodiment of the present application;

[0022] Figure 3A Schematic diagram of the flow of the speech synthesis method provided in the embodiment of the present application;

[0023] Figure 3B Schematic diagram of the process of extracting features from a phoneme sequence according to an embodiment of the present application;

[0024] Figure 3C is a schematic diagram of a process for extracting features from a second text provided in an embodiment of the present application;

[0025] Figure 3D is a flowchart of attention adjustment provided by an embodiment of the present application;

[0026] Figure 4A Schematic diagram of the network structure of the FastSpeech2 model provided in the embodiment of the present application;

[0027] Figure 4B Schematic diagram of the network structure of the speech synthesis model provided in the embodiment of the present application;

[0028] Figure 5 Schematic diagram of the network structure of the feature enhancement layer provided in an embodiment of the present application;

[0029] Figure 6 Schematic diagram of the predicted spectrum provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0031] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0032] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0033] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0034] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0035] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0036] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0037] 1) Phoneme sequence: A phoneme sequence is a collection of phonemes arranged in sequence. Each phoneme represents the smallest, most basic unit of speech that can distinguish meaning. In phonetics, a phoneme is defined as the smallest distinguishing feature of speech that can be used to distinguish different languages ​​or dialects.

[0038] 2) Attention mechanism. The attention mechanism is a technique widely used in deep learning. It allows models to selectively focus on important parts of information while ignoring unimportant information when processing information. This mechanism mimics the human behavior of focusing attention: when processing a task, people generally focus on the most critical information while ignoring other distracting information. In deep neural networks, the attention mechanism is often used in sequence-to-sequence (seq2seq) models, particularly in fields such as machine translation, speech recognition, and natural language processing. Its basic principle is to determine which parts are important by calculating the correlation between the query, key, and value.

[0039] 3) Spectrum, a frequency representation method used in audio signal processing, such as the Mel spectrum (Mel spectrum). It performs a nonlinear transformation of frequencies based on the human auditory perception. This transformation renders the frequency axis no longer linear, but rather logarithmic, simulating the differences in the human auditory system's perception of sounds of different frequencies. In speech synthesis, the spectrum is often used as an intermediate representation to generate pitch and sound quality. In speech recognition, the Mel spectrum can improve the performance of the recognition system because it more closely resembles the processing method of the human auditory system. In audio signal processing, the Mel spectrum can help with tasks such as sound source separation and echo cancellation.

[0040] Non-autoregressive speech synthesis models are attracting increasing attention from both industry and academia. The FastSpeech2 model has garnered the most attention. Its excellent speech synthesis performance and responsiveness have won the attention of many companies, who are now using it for online tasks. The FastSpeech2 model takes a phoneme sequence as input and outputs a mel spectrum, which is then synthesized into speech using a vocoder. However, the FastSpeech2 model lacks precise control over the properties of the synthesized speech (such as pronunciation and emotion). For example, speech synthesis in customer service or debt collection scenarios requires a gentle and pleasant voice, while debt collection scenarios require a more stern voice to encourage customers to repay quickly. To address this issue, it is necessary to incorporate information describing the properties of the synthesized speech into the synthesis process when synthesizing phoneme sequences into speech, enabling fine-grained control of the synthesized speech properties.

[0041] The embodiments of the present application provide a speech synthesis method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can integrate information describing the attributes of a first speech into the first speech, thereby achieving refined control of the attributes of the synthesized speech and improving the accuracy of speech synthesis.

[0042] See also Figure 1 , Figure 1This is a schematic diagram of the architecture of the speech synthesis system 100 provided in an embodiment of the present application. To support a speech synthesis application, the terminal 401 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0043] Terminal 401 is configured to send a speech synthesis request to server 200 in response to a speech synthesis instruction. Server 200 is configured to, in response to the speech synthesis request, obtain a phoneme sequence of a first text and a second text describing an attribute of the first speech from terminal 401, then perform feature extraction on the phoneme sequence to obtain phoneme features, and perform feature extraction on the second text to obtain a first attribute feature; based on the phoneme features, perform attention adjustment on the first attribute feature to obtain a second attribute feature; fuse the second attribute feature and the phoneme feature to obtain a fused feature; perform spectrum prediction based on the fused feature to obtain a predicted spectrum, and perform speech synthesis based on the predicted spectrum to obtain a first speech corresponding to the first text.

[0044] In some embodiments, the server 200 may directly send the processing result "first voice" in response to the speech synthesis request to the terminal 401; the terminal 401 may also actively obtain the "first voice" from the server 200.

[0045] In some embodiments, terminal 401 can be implemented as various types of terminals, such as a laptop computer, tablet computer, desktop computer, set-top box, smartphone, smart speaker, smart watch, smart TV, vehicle-mounted terminal, etc., or can be implemented as a server. Below, exemplary applications when the device is implemented as a terminal will be described.

[0046] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal and the server may be connected directly or indirectly via wired or wireless communication, which is not limited in the embodiments of the present application.

[0047] See also Figure 2 , Figure 2 is a structural diagram of an electronic device 400 provided in an embodiment of the present application, Figure 2The electronic device 400 shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the electronic device 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figure 2 Various buses are labeled as bus system 440 .

[0048] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0049] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0050] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0051] The memory 450 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0052] In some embodiments, the memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0053] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0054] A network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include Bluetooth, Wi-Fi, and Universal Serial Bus (USB);

[0055] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0056] The input processing module 454 is configured to detect one or more user inputs or interactions from one of the one or more input devices 432 and to translate the detected inputs or interactions.

[0057] In some embodiments, the apparatus provided in the embodiments of the present application may be implemented in software. Figure 2 A speech synthesis device 455 stored in memory 450 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: a determination module 4551, a feature extraction module 4552, an adjustment module 4553, a fusion module 4554, and a synthesis module 4555. These modules are logical and can be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0058] In other embodiments, the speech synthesis device provided in the embodiments of the present application can be implemented in hardware. As an example, the speech synthesis device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the speech synthesis method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs) or other electronic components.

[0059] The speech synthesis method provided by the embodiments of the present application will be described below with reference to the accompanying drawings. As previously mentioned, the electronic device implementing the speech synthesis method of the embodiments of the present application can be a terminal 401, a server 200, or a combination of the two. Therefore, the execution entity of each step will not be repeatedly described below.

[0060] The speech synthesis method of the embodiment of the present application is described by taking the execution subject as the server 200 as an example. Figure 3A , Figure 3A This is a flow chart of the speech synthesis method provided in the embodiment of the present application, which will be combined with Figure 3A The steps shown are explained.

[0061] In step 101 , a phoneme sequence of a first text and a second text are determined.

[0062] The second text is used to describe the attributes of the first speech.

[0063] In this embodiment of the present application, the first text describes text content that needs to be converted into natural-sounding human speech. The first text can be in the form of electronic text of a written language. A phoneme is the smallest unit of speech in a language and is the basic unit for distinguishing meaning in a particular language. A phoneme is a concept in phonetics, distinct from characters or letters in philology. The number and characteristics of phonemes vary in different languages.

[0064] Taking the first text being Chinese as an example, the phonemes corresponding to the characters in the first text may be the pinyin, tone and rhythm of the characters, wherein the rhythm indicates the length of pauses between characters or words.

[0065] For example, if the first text is "Hello, may I ask who you are?", the phoneme sequence corresponding to the first text may be "nin2 hao3 8qing3 wen4 nin3 shi4". In this phoneme sequence, the number immediately following the pinyin represents the tone, with 1 representing the first tone, 2 representing the second tone, 3 representing the third tone, 4 representing the fourth tone, 5 representing the light tone, and 6 representing the altered tone. The numbers located between the pinyin represent the rhythm, with 7, 8, and 9 representing the length of the rhythmic pause, with 7 representing a shorter pause, 9 representing a longer pause, and 8 representing a medium pause.

[0066] In this embodiment of the present application, the second text can be used to describe the pronunciation emotion and intonation attributes of the first speech, such as gentleness, severity, or joy. As an example, the second text can be "speaking gently, feeling immersive", which is used to indicate that the intonation of the speech synthesized based on the first text is gentle, making the listener feel immersive.

[0067] In actual implementation, after obtaining the first text, a corresponding phoneme sequence can be generated based on the language used in the first text. The phoneme sequence includes the phonemes corresponding to each character in the first text. Each phoneme represents the smallest, most basic unit of speech that can distinguish meaning. In phonetics, a phoneme is defined as the smallest distinguishing feature of speech, which can be used to distinguish different languages ​​or dialects. Here, each phoneme in the phoneme sequence can also be called a unit phoneme, which can be understood as the smallest unit of phoneme.

[0068] In step 102, feature extraction is performed on the phoneme sequence to obtain phoneme features.

[0069] In step 103, feature extraction is performed on the second text to obtain a first attribute feature.

[0070] In actual implementation, a speech synthesis model for speech synthesis may be constructed. The speech synthesis model is based on the related art FastSpeech2 model and adds a branch for feature processing for the second text.

[0071] As an example, Figure 4A The network structure diagram of the FastSpeech2 model provided in the embodiment of this application is shown in Figure 4A The network structure of the FastSpeech2 model can include a phoneme embedding layer, a positional encoding layer, an encoder, a variation adapter, and a mel-spectrogram decoder. Figure 4A Based on the network structure shown in FIG, a network branch is added at the position of the encoder to process the second text. Figure 4B For a network structure diagram of the speech synthesis model provided in the embodiment of the present application, see Figure 4B The newly added network branch may include a feature extraction layer, a feature enhancement layer, and a feature adjustment layer, wherein the result of the encoder serves as the input of the feature adjustment layer, and the result of the feature adjustment layer and the result of the encoder serve as the input of the variation adapter together.

[0072] In actual implementation, after constructing the speech synthesis model, it is necessary to train the speech synthesis model to obtain a target speech synthesis model that meets the training end conditions. The training end conditions may include but are not limited to: reaching a preset number of training times, or the loss convergence reaching a preset convergence level.

[0073] For example, when training a speech synthesis model, the training samples can include a phoneme sample sequence, descriptive sample text, and a standard spectrum. The speech synthesis model is trained using the phoneme sample sequence as input to the phoneme embedding layer, the descriptive sample text as input to the feature extraction layer, and the predicted spectrum as output. The predicted spectrum and the standard spectrum output by the speech synthesis model are input into the loss function to calculate the loss value. Backpropagation is then used to adjust the parameters of the speech synthesis model until the training termination criteria are met, resulting in the target speech synthesis model.

[0074] In actual implementation, after obtaining the phoneme sequence of the first text and the second text describing the attributes of the first speech, the target speech synthesis model can use the phoneme sequence as input to the phoneme embedding layer to perform feature extraction on the phoneme sequence to obtain phoneme features. The second text can then be used as input to the feature extraction layer of the target speech synthesis model to perform feature extraction on the second text to obtain first attribute features. The first attribute features are obtained by extracting the attribute information of the first speech described by the second text. Therefore, the first attribute features can be used to represent the attribute information of the first speech described by the second text.

[0075] In some embodiments, see Figure 3B , Figure 3A The step 102 “extracting features from the phoneme sequence to obtain phoneme features” may be performed through steps 1021 to 1023 .

[0076] In step 1021, the phoneme sequence is vector-converted to obtain the phoneme vector of the phoneme sequence.

[0077] In actual implementation, the phoneme sequence includes unit phonemes corresponding to each character in the first text. Therefore, when the phoneme sequence is vectorized, each unit phoneme in the phoneme sequence can be vectorized to obtain a unit phoneme vector corresponding to each unit phoneme, and all unit phoneme vectors constitute the phoneme vector of the phoneme sequence.

[0078] As an example, a phoneme sequence can be input into Figure 4B The phoneme embedding layer shown in the figure converts a phoneme sequence into a vector. The phoneme embedding layer maps unit phonemes into fixed-dimensional vectors, specifically unit phoneme vectors, capturing the information contained in the unit phonemes. The phoneme embedding layer generates a phoneme embedding sequence corresponding to the phoneme sequence, also known as a phoneme vector. This phoneme embedding sequence includes the unit phoneme vector corresponding to each unit phoneme.

[0079] In step 1022, position encoding is performed on the phoneme vector to obtain a phoneme encoding feature including position information.

[0080] Positional encoding is the process of adding positional information to elements in sequence data (such as text). In this application, positional information is added to each unit phoneme vector in the phoneme embedding sequence. When processing phoneme embedding sequences, it is necessary to understand the order between each unit phoneme vector, because the order of unit phoneme vectors is usually closely related to their semantic meaning. For example, in a sentence, the order of words affects their meaning and the overall meaning of the sentence.

[0081] In actual implementation, since the phoneme embedding layer and encoder of the FastSpeech2 model do not have the ability to process the position information of sequence data, an additional position encoding layer is added between the phoneme embedding layer and the encoder to provide the position information of the phoneme embedding sequence through the additional position encoding layer.

[0082] The positional encoding layer generates a unique position vector for each position in the phoneme embedding sequence (i.e., phoneme vector), so that words or characters at different positions can be distinguished. This position vector is usually generated by calculating a set of sine and cosine functions based on the position information, and then using these function values ​​as the elements of the position vector.

[0083] In this way, the phoneme vectors corresponding to the phoneme sequence are positionally encoded to obtain the positional information of the phoneme vectors. The phoneme vectors are then converted into phoneme encoding features that include this positional information. Phoneme encoding features are vector-represented phoneme embedding sequences with positional vectors. Positional encoding of phoneme vectors allows for a better understanding of their structure. This positional encoding method can be easily extended to phoneme embedding sequences of arbitrary lengths, and the encoded positional vectors are unique for different positions in the phoneme embedding sequence.

[0084] In step 1023, based on the position information, latent state features are extracted from the phoneme coding features, and the extracted latent state features are used as phoneme features.

[0085] In practice, to learn higher-level textual information between phonemes, such as syntax, the semantic contextual relationships between unit phonemes can be captured by extracting hidden states. A hidden state is a model state that captures the relationship between previous and subsequent phonemes in a sequence. It represents the intrinsic state of a position in the phoneme encoding feature. This state incorporates the influence of the corresponding information of the previous phoneme and can affect subsequent states.

[0086] As an example, Figure 4BThe encoder shown can extract hidden state features of phoneme coding features based on the position information carried in the phoneme coding features, and map the phoneme coding features (a series of unit phoneme vectors with position information) into a phoneme hidden state sequence (a series of hidden states). The phoneme hidden state sequence can be called the hidden state feature of the phoneme sequence, that is, the phoneme feature.

[0087] In some embodiments, see Figure 3C , Figure 3A The step 103 “extracting features from the second text to obtain first attribute features” may be performed through steps 1031 to 1032 .

[0088] In step 1031 , semantic features of the second text are extracted to obtain semantic features of the second text.

[0089] In actual implementation, the second text can be input into Figure 4B The feature extraction layer shown can extract semantic features from the input second text, convert the second text into a semantic feature vector that can be used by the target speech synthesis model, and obtain the semantic features of the second text.

[0090] Here, the feature extraction layer usually pays more attention to the sentence meaning information of the second text, that is, the global feature information. Therefore, the obtained semantic features usually have the global feature information of the second text.

[0091] In step 1032 , feature enhancement processing is performed on the semantic feature to obtain a first attribute feature of the second text.

[0092] In actual implementation, in order to better understand the meaning of the attributes described in the second text, feature enhancement can be performed on the semantic features to extract global and local feature information of the second text.

[0093] In some embodiments, Figure 3C Step 1032 can be implemented through the following process.

[0094] Performing local feature extraction on the semantic feature to obtain a local feature; performing semantic enhancement processing on the local feature to obtain a semantic enhancement feature; and determining a first attribute feature of the second text based on the semantic feature and the semantic enhancement feature.

[0095] In actual implementation, semantic features refer to the overall semantic features of the second text, which have global feature information of the second text. The global feature information focuses more on the sentence meaning information of the second text. Therefore, in order to obtain more detailed feature information of the second text, it is also possible to extract local features from the semantic features to obtain the global and local semantic information of the second text, so as to obtain the first attribute feature with global feature information and local feature information. Local features refer to the features conveyed by a certain word in the second text, and focus more on the semantic information of the second text on characters or words and the association information of adjacent characters. For example, when the second text is a paragraph, the semantic features reflect the semantic information of the entire paragraph, while the local features can only reflect the information of a certain word in the entire paragraph.

[0096] As an example, local feature extraction is performed on the semantic features of the second text to obtain the local features of the second text. Then, semantic enhancement processing can be performed on the local features through any method such as deep learning, transfer learning, semantic embedding, etc. to learn to obtain deeper local feature information and obtain semantic enhancement features. Subsequently, the first attribute feature of the second text having both global feature information and local feature information is obtained by using the semantic features that represent the global feature information and the semantic enhancement features that represent the local feature information. In some feasible methods, the semantic features and the semantic enhancement features can be weighted and summed to obtain the first attribute feature. Alternatively, after the local feature extraction is performed, the local features can be directly embedded into the semantic features, and then the semantic features carrying the local features can be semantically enhanced to obtain the first attribute feature of the second text.

[0097] As an example, Figure 4B The feature enhancement layer shown can be constructed by a convolutional neural network (CNN), see Figure 5 ,Convolutional neural networks can include convolutional layers, normalization layers, activation layers, and fully connected layers.

[0098] For example, semantic features are first fed into a convolutional layer, which uses a series of convolution kernels to perform convolution operations on a single feature dimension. Each convolution kernel generates a feature map, which is the activation value of the semantic feature as the convolution kernel slides. After the convolution operation, the convolution layer applies a nonlinear activation function, such as the ReLU function. The output of the convolution layer then serves as the input to the normalization layer. After the nonlinear activation function, the normalization layer normalizes the semantic features that carry local features. This is done by subtracting the mean from each feature value, dividing it by the standard deviation, and then multiplying it by a learnable coefficient. This helps prevent gradient vanishing or gradient exploding problems during processing, allowing the target language synthesis model to converge faster. The activation layer (Gaussian Error Linear Unit, GELU) then improves on the ReLU function by combining the linear portion of the ReLU function with a Gaussian error function, enabling the target language synthesis model to learn a more complex model of semantic features that carry local features, thereby increasing semantic expression capabilities. Finally, the fully connected layer (Linear layer) uses a learnable transformation matrix to map the input semantic features carrying local features to a new feature space and output a first attribute feature of a fixed dimension.

[0099] In the embodiment of the present application, the convolution layer can perform local feature extraction on the semantic features and enhance the ability to obtain local information. The normalization layer can normalize the data and improve the stability during feature processing. The activation layer can enhance the semantic expression ability. The fully connected layer is a dimensional transformation, which converts the larger-dimensional semantic features carrying local features into feature vectors of smaller dimensions, thereby reducing the number of parameters.

[0100] Through the above embodiment, the first attribute feature of the second text has richer feature information than the semantic feature, and has both global feature information and local feature information. In this way, when the first attribute feature is subsequently adjusted, the second attribute feature can be made to have attributes that are more consistent with the description of the second text.

[0101] Continue reading Figure 3A , continue with the above step 103 for explanation.

[0102] In step 104, attention adjustment is performed on the first attribute feature based on the phoneme feature to obtain a second attribute feature.

[0103] Since the attention mechanism is primarily used to process sequential data, such as in fields like natural language processing and computer vision, its core idea is to mimic how humans focus their attention. This mechanism calculates attention weights for each input position, enabling the model to dynamically assign weights based on different parts of the input data, allowing the model to dynamically focus on key parts of the input data. When processing sequential data, the attention mechanism allows the model to consider information at every position in the sequence, rather than relying solely on input at fixed positions. Furthermore, in sequential data, some information may be separated by large distances. The attention mechanism helps the model better capture these long-range dependencies, improving the model's ability to capture complex patterns in the sequence.

[0104] Therefore, the embodiment of the present application uses the attention mechanism to adjust the attention of the first attribute feature by paying attention to the phoneme feature on the first attribute feature. Through the mutual attention and mutual dependence between the phoneme feature and the first attribute feature, the second attribute feature obtained after the adjustment can simultaneously have the phoneme information represented by the phoneme feature and the attribute information represented by the first attribute feature. The process of adjusting the attention of the first attribute feature based on the phoneme feature can be understood as the process of preliminarily fusing the phoneme feature with the first attribute feature.

[0105] In some embodiments, see Figure 3D , Figure 3A Step 104 can be implemented through steps 1041 to 1042.

[0106] In step 1041, based on the attention mechanism, the attention weight between the phoneme feature and the first attribute feature is determined.

[0107] In some embodiments, step 1041 may be implemented in the following manner.

[0108] Based on the attention mechanism, the phoneme feature is converted into a query feature, and the first attribute feature is converted into a key feature; a dot product operation is performed on the query feature and the key feature to obtain the attention feature; based on the feature dimension of the first attribute feature, the attention feature is scaled to obtain the scaled attention feature; the scaled attention feature is normalized to obtain the attention weight between the phoneme feature and the first attribute feature.

[0109] As an example, Figure 4BThe feature adjustment layer shown can be built using the Scaled DotAttention mechanism. The basic idea of ​​the Scaled DotAttention mechanism is to calculate the dot product similarity between the query vector and a set of key vectors, convert it into a probability distribution through a multi-class classification function (such as the softmax function), and then use this probability distribution to weight the value vector, thereby focusing on important information.

[0110] Since the features are usually expressed as vectors when they are processed through a model, in the embodiment of the present application, the query feature may also be called a query vector, the key feature may be called a key vector, and the value feature may be called a value vector.

[0111] In the embodiment of the present application, the attribute situation reflected by the first attribute feature can be focused on by focusing on the attention situation between the phoneme feature and the first attribute feature, thereby achieving a preliminary fusion of the phoneme feature and the first attribute feature.

[0112] In actual implementation, the first attribute feature output by the feature enhancement layer and the phoneme feature output by the encoder serve as inputs to the feature adjustment layer. Both the first attribute feature and the phoneme feature can be feature vectors represented by vectors. Therefore, the phoneme feature can be used as the query feature in the attention mechanism, that is, the corresponding query vector (Query vector), and the first attribute feature can be used as the key feature in the attention mechanism, that is, the key vector (Key vector). Since multiple attribute features of the second text can be obtained when semantic features are extracted for the second text, that is, multiple attribute feature vectors are extracted, the first attribute feature vector corresponding to the first attribute feature usually includes multiple attribute feature vectors; similarly, when converting the phoneme sequence into phoneme features, each unit phoneme in the phoneme sequence is subjected to feature extraction and other processing. Therefore, the obtained phoneme feature is a phoneme hidden state sequence, and the phoneme hidden state sequence includes the hidden state features of each unit phoneme, that is, the phoneme feature vector corresponding to the phoneme feature also includes multiple hidden state feature vectors. Here, the vectors corresponding to the first attribute feature and the phoneme feature are usually multidimensional vectors. Therefore, the first attribute feature can also be regarded as the first attribute feature matrix corresponding to the first attribute feature vector, and the phoneme feature can also be regarded as the corresponding phoneme feature vector.

[0113] In the Scaled Dot Attention mechanism, the following calculations are included:

[0114] First, the dot product operation is performed on the query vector (Query vector) and the key vector (Key vector) to calculate the similarity score between the query vector and all key vectors, which measures the degree of match between the query vector and the key vector at each position.

[0115] Secondly, as the dimension of the key vector increases, the result of the dot product operation will also increase rapidly, which may cause the gradient of the multi-class classification function to become extremely small. Therefore, the result of the dot product operation can be scaled by dividing it by the dimension of the key vector. This keeps the attention scores at each position at a similar scale before being processed by the multi-class classification function, ensuring the convergence of the speech synthesis model.

[0116] The scaled dot product result is then applied to the multi-class classification function, normalized, and the attention score converted into a probability distribution. This means that the multi-class classification function converts the attention scores at all positions into probabilistic form, so that their sum is 1, thereby determining the relative importance of each position's value vector (value vector, value feature) when calculating the final output. Here, after converting the attention score into a probability distribution, the attention weight corresponding to the attention score is obtained.

[0117] Finally, the dot product operation is performed on the attention weight matrix and the value matrix corresponding to the value vector (value feature) to obtain the weighted value vector, namely the context vector. The vector of each output position of the context vector is synthesized by adjusting the corresponding value vector by the attention weight.

[0118] The essence of the scaled dot product attention mechanism is to quantify the similarity between the query feature (query vector) and the key feature (key vector) through the dot product, and then assign attention weights through a multi-category classification function, and perform weighted summation of the value features (value vectors) according to these weights to form a context-sensitive representation of each position in the input sequence. This mechanism dynamically pays attention to different positions in the input sequence through query features, and can enhance the ability of the speech synthesis model to capture dependencies through weighted integration. In an embodiment of the present application, the adjusted feature obtained by adjusting the attention of the attribute feature is the context vector.

[0119] In actual implementation, based on the above calculation process, when the phoneme feature is used as the query feature (query vector) and the first attribute feature is used as the key feature (key vector), the phoneme feature is dot-producted with the first attribute feature to calculate the attention score between the phoneme feature and each attribute feature in the first attribute feature, and the attention feature between the phoneme feature and the first attribute feature is obtained. The attention feature is an attention feature matrix including the attention scores between the phoneme feature and each attribute feature. In order to ensure convergence, the attention feature is scaled based on the feature dimension of the first attribute feature to obtain the scaled attention feature. Then, the scaled attention feature is normalized by a multi-category classification function to convert the attention score in the attention feature into a probability distribution, and obtain the attention weight between the phoneme feature and the first attribute feature.

[0120] In step 1042, the first attribute feature is adjusted based on the attention weight to obtain a second attribute feature.

[0121] In some embodiments, step 1042 may be implemented in the following manner.

[0122] The first attribute feature is converted into a value feature; a dot product operation is performed on the attention weight and the value feature to obtain the second attribute feature.

[0123] In actual implementation, since this application focuses on the attention between the phoneme feature and the first attribute feature, the first attribute feature can be directly used as the value feature in the attention mechanism, and the attention weight between the first attribute feature and the phoneme feature is used to perform a dot product operation on the first attribute feature to obtain the second attribute feature. The second attribute feature can be understood as achieving the mutual attention and mutual dependence of the phoneme feature and the first attribute feature through the scaled dot product attention mechanism (Scaled Dot Attention), so that based on the attention of the two, the phoneme feature and the first attribute feature are preliminarily fused to obtain the second attribute feature that has both the attribute information reflected by the second text and the phoneme information reflected by the phoneme feature.

[0124] In actual implementation, a dot product operation is performed on the attention weight and the first attribute feature. The first attribute feature can be weighted summed using the single-dimensional attention weight data corresponding to each dimension of the attention weight matrix to obtain the single-dimensional second attribute feature corresponding to each dimension of the attention weight matrix. The single-dimensional second attribute feature constitutes the second attribute feature.

[0125] As an example, assume query feature Q = [q_1, q_2, q_3]; key feature K = [k_1, k_2, k_3]; and value feature V = [v_1, v_2, v_3]. Let the phoneme feature be the query feature Q, the first attribute feature be the key feature K, and the value feature V be the value feature. Then, the value feature V = [v_1, v_2, v_3] = [k_1, k_2, k_3]. Here, we assume that both the phoneme feature and the first attribute feature are three-dimensional vectors, and use the three-dimensional vector as an example for the dot product operation. In actual processing, phoneme characteristics and first attribute features are typically more complex feature vectors.

[0126] The attention weight matrix is ​​obtained by performing a dot product operation on the query feature and the key feature, that is, the dot product operation on the phone feature and the first attribute feature. Here, the dimensionality scaling and normalization steps can be omitted, and the dot product result can be directly used as the attention weight between the phone feature and the first attribute feature.

[0127] The attention weight matrix a after the dot product operation is the weight distribution corresponding to the relative importance between the feature elements of the phoneme feature and the first attribute feature. For example: a = QK^T = \begin{bmatrix}q_1k_1&q_1k_2&q_1k_3\\q_2k_1&q_2k_2&q_2k_3\\q_3k_1&q_3k_2&q_3k_3\end{bmatrix}.

[0128] We then perform a dot product operation on the attention weight matrix a and the first attribute feature V to obtain the final context vector c (also known as the second attribute feature). The result of the dot product operation is a three-dimensional vector, where each dimension is the dot product result of the corresponding element of the phoneme feature Q and the first attribute feature K. For each query q_i, q_i is weighted and summed with the values ​​v_j corresponding to all keys k_j to obtain the components c_i of the context vector. Each component c_i in the context vector is c_i = sum a_{ij}v_j. For the feature example given above, the component c_i of the context vector will be: c_i = a_{i1}v_1 + a_{i2}v_2 + a_{i3}v_3, and the final context vector c is: c = [c_1, c_2, c_3]. Here, c_1, c_2, and c_3 are the three-dimensional vectors of attention weights calculated through the above process, which represent the relative importance of each attribute feature in the first attribute feature to the phoneme feature. This context vector c is the second attribute feature, which carries both the attribute information covered by the second text and the phoneme information covered by the phoneme sequence.

[0129] In the above example, a_{i1} refers to the attention weight of the i-th row and the first column in the attention weight matrix, which represents the strength of the association between the query q_i and the key k_1. The specific calculation is as follows:

[0130] Assume that the attention weight matrix a is as follows:

[0131] a=begin{bmatrix}q_1k_1&q_1k_2&q_1k_3\\q_2k_1&q_2k_2&q_2k_3\\q_3k_1&q_3k_2&q_3k_3\end{bmatrix}, then, a_{i1} is the attention weight of the i-th row and 1st column of the attention weight matrix, calculated as follows: a_{i1}=q_ik_1, Continuing the above example, assume that q_i and k_1 are: q_i=[q_{i1},q_{i2},q_{i3}], k_1=[k_{11},k_{12},k_{13}]; then, the calculation result of a_{i1} is: a_{i1}=q_{i1}k_{11}+q_{i2}k_{12}+q_{i3}k_{13}. This result represents the strength of association between the query q_i and the key k_1. It is obtained by multiplying each component of the query vector by the corresponding component of the key vector and summing the results. In practice, the query features Q and key features K will usually have more dimensions and will be calculated using a more complex network structure, but the basic calculation principle is the same.

[0132] Continue reading Figure 3A , continue with the above step 104 for description.

[0133] In step 105, the second attribute feature and the phoneme feature are fused to obtain a fused feature.

[0134] In some embodiments, the context vector obtained based on the scaled dot product attention mechanism (Scaled Dot Attention) has the same dimension as the query vector (query feature), that is, the dimension of the second attribute feature is consistent with the dimension of the phoneme feature. Therefore, the second attribute feature can be directly added to the phoneme feature to achieve the fusion processing of the second attribute feature and the phoneme feature. The advantage of direct addition is that the feature dimension after addition does not increase, reducing the number of parameters, and the fused feature obtained by direct addition can bring the attribute information of the second text to the phoneme feature.

[0135] In some embodiments, Figure 3A Step 105 can be implemented in the following manner.

[0136] Determine a first weight of the second attribute feature and a second weight of the phoneme feature; and fuse the second attribute feature and the phoneme feature based on the first weight and the second weight to obtain a fused feature.

[0137] In actual implementation, the process of directly adding the second attribute feature and the phoneme feature to achieve the fusion of the two features is equivalent to treating the second attribute feature and the phoneme feature as equally important, and the degree of fusion of the two features is consistent. However, in different fusion requirements, the degree of attention paid to the second attribute feature and the phoneme feature is different, and the degree of fusion of the second attribute feature and the phoneme feature during fusion also needs to be different. Therefore, the embodiment of the present application can also assign weights to the second attribute feature and the phoneme feature respectively.

[0138] As an example, weights can be assigned to the second attribute features and phoneme features using a historical experience weight matrix, or using a weight matrix trained using machine learning methods. The weight elements in the weight matrix determine the degree of fusion between the second attribute features and phoneme features.

[0139] As an example, determine the first weight corresponding to each component in the second attribute feature vector corresponding to the second attribute feature, and the second weight of each component in the phoneme feature vector corresponding to the phoneme feature. For the components of the second attribute feature and the phoneme feature at corresponding positions, determine the first product of the component of the second attribute feature and the corresponding first weight, and the second product of the component of the phoneme feature and the corresponding second weight, and use the sum of the first product and the second product as the fusion value corresponding to the two components. By weighted summation of the second attribute feature and the phoneme feature at each component, the fusion processing of the second attribute feature and the phoneme feature is realized to obtain a fusion feature, which includes the fusion value of the components of the second attribute feature and the phoneme feature at corresponding positions.

[0140] Through the above embodiments, whether the second attribute feature and the phoneme feature are directly added or weighted together, the resulting fused feature is a comprehensive feature vector that combines phoneme information and attribute information. Each component of the comprehensive feature vector is the fusion value of the latent sequence feature and the attribute feature in the corresponding dimension. Because the fused feature carries attribute information, speech corresponding to the first text and conforming to the attribute information represented by the first attribute feature can be generated based on the fused feature. For example, speech that conforms to the emotional information represented by the attribute information can be synthesized. Furthermore, before the fusion process, the process of adjusting the first attribute feature based on the attention mechanism can be considered as a preliminary fusion process of the phoneme feature and the first attribute feature. The attention mechanism can achieve fusion of the phoneme feature and the first attribute feature through mutual attention and mutual dependence between the phoneme feature and the first attribute feature, so that the second attribute feature simultaneously possesses the attribute information described by the second text and the phoneme information represented by the phoneme feature. By fusing the second attribute feature obtained through preliminary fusion with the phoneme feature, the attribute information described by the second text can be better integrated into the phoneme feature. Moreover, since the second attribute feature has phoneme information, it can also be regarded as a phoneme feature carrying attribute information. Therefore, fusing the phoneme feature carrying attribute information with the original phoneme feature can reduce the impact on the phoneme feature, does not change the semantics represented by the phoneme feature, and reduces the negative impact on the phoneme feature.

[0141] After fusing the phoneme features and the second attribute features, the fused features can have the following meanings:

[0142] First, attribute enhancement: Components in the first attribute feature vector may enhance or adjust the values ​​of corresponding components in the phoneme feature vector, thereby producing speech with more attribute characteristics during speech synthesis. For example, if the first attribute feature vector represents the emotion of happiness or excitement, it may increase the components related to happiness or excitement in the phoneme feature vector, thereby reflecting these emotional attribute characteristics in the speech output.

[0143] Secondly, inflection. Attribute information can affect the intonation of speech, i.e., the rhythm of speech. For example, the emotional attribute of sadness may cause a falling intonation, while the emotional attribute of happiness may cause a rising intonation. This intonation variation can be achieved by adjusting the pitch-related components of the phoneme feature vector.

[0144] Third, intensity adjustment: Attribute information can also affect the intensity or loudness of speech. For example, strong emotional attributes may cause speech to be more forceful, thereby increasing the intensity of the speech. This intensity adjustment can be achieved by adjusting the intensity-related components of the phoneme feature vector.

[0145] Fourthly, speed changes. Attribute information may also affect the speed of speech. For example, emotional attributes of tension or excitement may lead to faster speaking, while emotional attributes of relaxation or contemplation may slow down speaking. This change in speed can be achieved by adjusting the components related to the time interval in the phoneme feature vector. The components related to the time interval refer to the features in the phoneme feature vector that are related to the time length or duration between phonemes. For example, in Mandarin, different initials and finals have different pronunciation time lengths, and tones also have different durations. In speech synthesis, if you want to simulate nervous or excited emotions, you can do so by reducing the time interval between phonemes and increasing the speaking speed. On the contrary, if you want to simulate contemplation or relaxation, you can do so by increasing the time interval between phonemes and reducing the speaking speed.

[0146] Finally, the target speech synthesis model can generate a corresponding speech signal based on the fusion features, thereby reading out the first text corresponding to the phoneme sequence, and at the same time giving the first text corresponding attributes such as emotional color.

[0147] In step 106, spectrum prediction is performed based on the fused features to obtain a predicted spectrum, and speech synthesis is performed based on the predicted spectrum to obtain a first speech corresponding to the first text.

[0148] In some embodiments, "predicting the spectrum based on the fused features to obtain the predicted spectrum" in step 106 can be implemented in the following manner.

[0149] Attribute prediction is performed based on the fused features to obtain predicted attribute information; based on the predicted attribute information, spectrum mapping is performed on the fused features to obtain a predicted spectrum.

[0150] Mel spectrum is a graphical representation method frequently used in audio signal processing. It can reflect the spectral characteristics of audio signals and some information related to human auditory perception. Mel spectrum graphs are commonly used in speech recognition, music analysis, sound separation, and other audio processing tasks. Therefore, in the embodiment of the present application, after obtaining the fused features, the fused features can also be spectrally predicted to obtain the predicted spectrum corresponding to the fused features. Subsequently, the predicted spectrum is converted into the first speech through a vocoder.

[0151] See also Figure 4BIn the target speech synthesis model, the Variance Adapter is a component used in speech synthesis to introduce variation in the vocoder, thereby giving the first speech more natural variability and diversity. The Variance Adapter's primary function is to capture and simulate the natural variability in speech signals, which can be caused by factors such as the speaker's personality, emotional state, articulation speed, and vocal richness. By introducing this variability, the first speech sounds more natural and realistic, avoiding the problem of being too mechanical and monotonous.

[0152] In actual implementation, after receiving the fused features, the variational adapter can perform attribute prediction on the fused features and add the predicted attribute information to the fused features. This allows the spectrum decoder to perform spectrum prediction based on the fused features carrying the predicted attribute information, thereby obtaining a predicted spectrum. The predicted attribute information here can represent differential attribute information such as duration, pitch, and energy. This differential attribute information is different from the attribute information represented by the second text and is predicted based on the attribute information described in the second text.

[0153] In the target speech synthesis model, a variational adapter can predict and add predictive attribute information to the fused features. This predictive attribute information helps the target speech synthesis model better understand and generate speech signals with varying durations, pitches, and intensities, improving the naturalness and expressiveness of the speech synthesis and making the first speech closer to realistic human speech. The variational adapter can include multiple predictors, each responsible for predicting a type of differential attribute information. As an example, the variational adapter can include a duration predictor, a pitch predictor, and an energy predictor. The duration predictor is used to predict the duration of the current or next phoneme in the fused feature. This can be the absolute duration or relative duration (e.g., the logarithm of the phoneme length); the pitch predictor is used to predict the pitch value of the phoneme in the fused feature. The pitch value can be a numerical value within a specific pitch range or a fractional value based on a pitch mapping; and the energy predictor is used to predict the energy level of the phoneme in the fused feature, which is generally related to the intensity of the syllable. The predictor in the variational adapter typically generates predictions based on the latent state features of the current phoneme, the attribute information represented by the secondary text, and the correlation between the preceding and following phonemes. Combining this variational information with the fused features allows the target speech synthesis model to more finely control the generated sequence, thereby improving the naturalness and quality of the generated speech.

[0154] In practical implementation, the variation adapter can be a standalone module or part of the target speech synthesis model. Its parameters can be optimized through the backpropagation algorithm during training to learn the correct variation pattern from the data.

[0155] Through the above embodiment, when synthesizing speech for a first text, a phoneme sequence of the first text and a second text used to describe the attributes of the first speech are obtained, and feature extraction is performed on the phoneme sequence and the second text respectively to obtain phoneme features and first attribute features; then, the attention of the first attribute features is adjusted using the phoneme features, and the adjusted second attribute features are fused with the phoneme features to obtain fused features that carry the phoneme information and the attribute information described by the second text; then, the fused features are used to perform spectrum prediction to obtain a predicted spectrum, and the predicted spectrum is used to generate a first speech corresponding to the first text and conforming to the attributes described by the second text. In this way, by utilizing attention adjustment and fusion processing, the attribute information described in the second text can be integrated into the phoneme features to synthesize a first speech conforming to the attributes described by the second text, thereby achieving refined control of the attributes of the first speech through the second text and improving the accuracy of speech synthesis.

[0156] In a specific embodiment, the speech synthesis method of the present application embodiment is described by taking the execution subject as the server 200 as an example. Figure 4B , set up in server 200 Figure 4B The speech synthesis model shown.

[0157] The speech synthesis model is based on Figure 4A Based on the FastSpeech2 model shown in , a network branch for processing the second text is added at the encoder position. Figure 4B The FastSpeech2 model includes a phoneme embedding layer, a positional encoding layer, an encoder, a variation adapter, and a mel-spectrogram decoder; the newly added network branches include a feature extraction layer, a feature enhancement layer, and a feature adjustment layer.

[0158] Among them, the phoneme embedding layer can obtain the text information between the phoneme sequences of the first text through feature extraction; the positional encoding layer is used to obtain the position information in the current input feature; the encoder converts the phoneme embedding sequence into a phoneme hidden state sequence in order to learn the text information between higher-level phonemes, such as syntax; the variation adapter contains many predictors to predict and add variation information (attribute features) such as duration, pitch, and energy to the hidden state sequence; after the variation adapter, the features corresponding to the phoneme sequence will be mapped to the Mel spectrum Figure 1 The input of the mel-spectrogram decoder is the output of the variation adapter. After the mel-spectrogram decoder, the model can learn richer information and predict the predicted spectrum.

[0159] The newly added network branches include feature extraction layer, feature enhancement layer and feature adjustment layer. Among them, the feature extraction layer can extract features from the second text and convert the second text into semantic features that can be used by the speech synthesis model. The semantic features mainly include the global semantics of the second text, that is, to let the speech synthesis model understand the meaning of the second text. The feature extraction layer can be built using a pre-trained NLP Bert network structure, which has excellent performance in text semantic tasks. The feature enhancement layer can use a convolutional neural network (CNN), see Figure 5 ,Convolutional neural networks can include convolutional layers, normalization layers, activation layers, and fully connected layers.

[0160] The main function of the feature enhancement layer is to enhance the semantic features output by the feature extraction layer to obtain the first attribute features with global feature information and local feature information. The feature adjustment layer is built using the scaled dot product attention mechanism. The phoneme features output by the encoder and the first attribute features output by the feature enhancement layer are used as the input of the feature adjustment layer. The phoneme features are used as the query features of the scaled dot product attention mechanism, and the first attribute features are used as the key features and value features of the scaled dot product attention mechanism. The principle of the scaled dot product attention mechanism is to dynamically pay attention to the feature information at different positions in the key features and value features through the query features, and to perform weighted integration of the feature information, thereby increasing the degree of dependency of the feature information. In other words, the phoneme features and the first attribute features are made to pay attention to each other and to be dependent on each other, so that the phoneme features contain the attribute information described by the second text, and the second attribute features are obtained.

[0161] The feature adjustment layer then feeds the output second attribute features back into the FastSpeech2 model for fusion with the phoneme features. This fusion can be performed by directly adding the second attribute features to the phoneme features or by weighting them based on an empirical weight matrix. The resulting fused features are then fed into the subsequent Variance Adapter and Mel-spectrogram Decoder to produce the predicted spectrum.

[0162] In some embodiments, the target speech synthesis model is also connected to a vocoder, which is connected to a spectrum decoder. The predicted spectrum predicted by the spectrum decoder can be input into the vocoder for speech synthesis to obtain a first speech corresponding to the first text and meeting the attributes described by the second text.

[0163] In server 200, build Figure 4B After the speech synthesis model is constructed, the initial speech synthesis model needs to be trained to obtain an applicable target speech synthesis model. The training termination conditions for the initial speech synthesis model may include, but are not limited to, reaching a preset number of training times and reaching a preset convergence level of loss.

[0164] For example, when training a speech synthesis model, the training samples can be <phoneme sample sequence, descriptive sample text, and standard spectrum>. Here, the speech signal restored from the standard spectrum through the vocoder conforms to the attributes described by the descriptive sample text. The speech synthesis model is trained using the phoneme sample sequence as input to the phoneme embedding layer, the descriptive sample text as input to the feature extraction layer, and the predicted spectrum as output. The predicted spectrum and the standard spectrum output by the speech synthesis model are then fed into a loss function to calculate the loss value. Backpropagation is then used to adjust the parameters of the speech synthesis model until the training end conditions are met, resulting in the target speech synthesis model.

[0165] As an example, after a phoneme sample sequence is input into the initial speech synthesis model, it is transformed into discrete features a using a phoneme dictionary. The phoneme sample sequence undergoes feature extraction in the phoneme embedding layer and is converted into a phoneme embedding sequence, which in turn becomes a continuous feature b1. The discrete feature a then passes through the position encoding layer to determine relative position information or absolute position information b2. The continuous feature b1 carrying position information b2 is then input into the encoder, which consists of several layers of Transformer Encoders. Thanks to the Transformer Encoder's excellent feature learning capabilities, the encoder's output feature vector (phoneme feature) contains richer information. Descriptive sample text is input into the pre-trained feature extraction layer to obtain semantic features. Then, the feature enhancement layer extracts local information from the semantic features, resulting in a first attribute feature. This first attribute feature contains both global and local feature information of the descriptive sample text. The first attribute feature serves as the key feature K and value feature V of the scaled dot-product attention mechanism in the feature adjustment layer. The phoneme feature serves as the query feature Q. The second attribute feature, obtained by the interaction of the phoneme feature and the first attribute feature, is added to the encoder output (phoneme feature) and input into the variational adapter for feature learning such as duration and pitch. It is then upsampled to a feature vector with the same dimension as the standard spectrum by the length adjuster in the variational adapter. The positional encoder then learns position information. The spectrum decoder, consisting of several layers of transformer encoders, can learn richer feature information and predict a predicted spectrum. The dimension of the predicted spectrum is now the same as that of the standard spectrum. The predicted spectrum and the standard spectrum are input into the loss function to calculate the loss data. Backpropagation is performed based on the loss data to adjust the parameters of the initial speech synthesis model. After multiple rounds of iterations, when the loss converges, the difference between the predicted spectrum and the standard spectrum will become smaller and smaller. When the speech synthesis model meets the training end conditions, the model is saved to obtain the target speech synthesis model.

[0166] When applying the target speech synthesis model to implement the speech synthesis method of the embodiment of the present application, the following process may be included:

[0167] First, load the saved target speech synthesis model and input the phoneme sequence of the first text and the second text describing the attributes of the first speech. For example, if the first text is "Hello, may I ask who you are?", the corresponding phoneme sequence might be "nin2 hao3 8qing3 wen4 nin3 shi4." The second text might be "The gentle speech makes you feel like you're there."

[0168] See also Figure 6 By processing the phoneme sequence and the second text by the target speech synthesis model, the output of the target speech synthesis model can be obtained as follows Figure 6 The predicted spectrum is shown.

[0169] Then, the predicted spectrum is input into a vocoder and converted into a speech signal by the vocoder to obtain a first speech corresponding to the first text and conforming to the attributes described by the second text.

[0170] The following continues to describe the exemplary structure of the speech synthesis device 455 provided in the embodiment of the present application as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the speech synthesis device 455 of the memory 440 may include:

[0171] The determination module 4551 is used to determine the phoneme sequence of the first text and the second text, where the second text is used to describe the attributes of the first speech.

[0172] The feature extraction module 4552 is used to extract features from the phoneme sequence to obtain phoneme features.

[0173] Feature extraction is performed on the second text to obtain the first attribute feature.

[0174] The adjustment module 4553 is used to adjust the attention of the first attribute feature based on the phoneme feature to obtain the second attribute feature.

[0175] The fusion module 4554 is used to fuse the second attribute feature and the phoneme feature to obtain a fused feature.

[0176] The synthesis module 4555 is configured to perform spectrum prediction based on the fused features to obtain a predicted spectrum, and perform speech synthesis based on the predicted spectrum to obtain a first speech corresponding to the first text.

[0177] In some embodiments, the adjustment module 4553 is further used to determine the attention weight between the phoneme feature and the first attribute feature based on the attention mechanism; and adjust the first attribute feature based on the attention weight to obtain the second attribute feature.

[0178] In some embodiments, the adjustment module 4553 is also used to convert the phoneme feature into a query feature based on the attention mechanism, and convert the first attribute feature into a key feature; perform a dot product operation on the query feature and the key feature to obtain the attention feature; scale the attention feature based on the feature dimension of the first attribute feature to obtain the scaled attention feature; normalize the scaled attention feature to obtain the attention weight between the phoneme feature and the first attribute feature.

[0179] In some embodiments, the adjustment module 4553 is further used to convert the first attribute feature into a value feature; perform a dot product operation on the attention weight and the value feature to obtain a second attribute feature.

[0180] In some embodiments, the fusion module 4554 is further used to determine a first weight of the second attribute feature and a second weight of the phoneme feature; based on the first weight and the second weight, the second attribute feature and the phoneme feature are fused to obtain a fused feature.

[0181] In some embodiments, the feature extraction module 4552 is also used to perform vector conversion on the phoneme sequence to obtain the phoneme vector of the phoneme sequence; perform position encoding on the phoneme vector to obtain the phoneme coding feature including position information; based on the position information, perform latent state feature extraction on the phoneme coding feature, and use the extracted latent state feature as the phoneme feature.

[0182] In some embodiments, the feature extraction module 4552 is further configured to extract semantic features from the second text to obtain semantic features of the second text; and perform feature enhancement processing on the semantic features to obtain first attribute features of the second text.

[0183] In some embodiments, the feature extraction module 4552 is further used to perform local feature extraction on the semantic feature to obtain the local feature; perform semantic enhancement processing on the local feature to obtain the semantic enhancement feature; and determine the first attribute feature of the second text based on the semantic feature and the semantic enhancement feature.

[0184] In some embodiments, the synthesis module 4555 is further configured to perform attribute prediction based on the fused features to obtain predicted attribute information; and perform spectrum mapping on the fused features based on the predicted attribute features to obtain a predicted spectrum.

[0185] The present invention provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the speech synthesis method described in the present invention.

[0186] The embodiment of the present application provides a computer-readable storage medium in which computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the speech synthesis method provided by the embodiment of the present application, for example, Figure 3A The speech synthesis method is shown.

[0187] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0188] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0189] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0190] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0191] In summary, when synthesizing speech for a first text through the embodiment of the present application, the phoneme sequence of the first text and the second text used to describe the attributes of the first speech are obtained, and feature extraction is performed on the phoneme sequence and the second text respectively to obtain phoneme features and first attribute features; then, the attention of the first attribute features is adjusted using the phoneme features, and the adjusted second attribute features are fused with the phoneme features to obtain fused features carrying phoneme information and attribute information described by the second text; then, the fused features are used to perform spectrum prediction to obtain a predicted spectrum, and the first speech corresponding to the first text and conforming to the attributes described by the second text is generated through the predicted spectrum. In this way, by utilizing attention adjustment and fusion processing, the attribute information described in the second text can be integrated into the phoneme features to synthesize the first speech conforming to the attributes described by the second text, thereby achieving refined control of the attributes of the first speech through the second text and improving the accuracy of the first speech.

[0192] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A speech synthesis method, characterized in that: The method comprises: Determining a phoneme sequence of a first text and a second text, wherein the second text is used to describe an attribute of the first speech; Performing feature extraction on the phoneme sequence to obtain phoneme features; extracting local features from the semantic features of the second text, performing semantic enhancement processing on the local features to obtain semantic enhancement features, and determining a first attribute feature based on the semantic features and the semantic enhancement features; Based on the phoneme feature, performing attention adjustment on the first attribute feature to obtain a second attribute feature; fusing the second attribute feature and the phoneme feature to obtain a fused feature; Spectrum prediction is performed based on the fusion features to obtain a predicted spectrum, and speech synthesis is performed based on the predicted spectrum to obtain a first speech corresponding to the first text.

2. The method according to claim 1, characterized in that The step of performing attention adjustment on the first attribute feature based on the phoneme feature to obtain a second attribute feature includes: Determining an attention weight between the phoneme feature and the first attribute feature based on an attention mechanism; Based on the attention weight, the first attribute feature is adjusted to obtain a second attribute feature.

3. The method according to claim 2, characterized in that The determining, based on the attention mechanism, the attention weight between the phoneme feature and the first attribute feature includes: Based on an attention mechanism, the phoneme feature is converted into a query feature, and the first attribute feature is converted into a key feature; Performing a dot product operation on the query feature and the key feature to obtain an attention feature; Scaling the attention feature based on the feature dimension of the first attribute feature to obtain a scaled attention feature; The scaled attention feature is normalized to obtain an attention weight between the phoneme feature and the first attribute feature.

4. The method according to claim 3, characterized in that The adjusting the first attribute feature based on the attention weight to obtain the second attribute feature includes: Converting the first attribute feature into a value feature; Perform a dot product operation on the attention weight and the value feature to obtain a second attribute feature.

5. The method according to claim 1, wherein The fusing the second attribute feature and the phoneme feature to obtain a fused feature includes: determining a first weight of the second attribute feature and a second weight of the phoneme feature; Based on the first weight and the second weight, the second attribute feature and the phoneme feature are fused to obtain a fused feature.

6. The method according to claim 1, characterized in that The extracting features of the phoneme sequence to obtain phoneme features includes: Performing vector conversion on the phoneme sequence to obtain a phoneme vector of the phoneme sequence; Performing position encoding on the phoneme vector to obtain a phoneme encoding feature including position information; Based on the position information, latent state features are extracted from the phoneme coding features, and the extracted latent state features are used as the phoneme features.

7. The method according to claim 1, characterized in that The performing spectrum prediction based on the fusion features to obtain a predicted spectrum includes: Performing attribute prediction based on the fused features to obtain predicted attribute information; Based on the predicted attribute information, spectrum mapping is performed on the fused features to obtain a predicted spectrum.

8. A speech synthesis device, characterized in that: The device comprises: a determination module, configured to determine a phoneme sequence of a first text and a second text, wherein the second text is used to describe an attribute of the first speech; a feature extraction module, configured to perform feature extraction on the phoneme sequence to obtain phoneme features, extract local features from the semantic features of the second text, perform semantic enhancement processing on the local features to obtain semantic enhancement features, and determine a first attribute feature based on the semantic features and the semantic enhancement features; an adjustment module, configured to perform attention adjustment on the first attribute feature based on the phoneme feature to obtain a second attribute feature; a fusion module, configured to fuse the second attribute feature and the phoneme feature to obtain a fusion feature; A synthesis module is used to perform spectrum prediction based on the fusion feature to obtain a predicted spectrum, and perform speech synthesis based on the predicted spectrum to obtain a first speech corresponding to the first text.

9. An electronic device, characterized in that: The electronic device comprises: a memory for storing computer-executable instructions; A processor, configured to implement the method according to any one of claims 1 to 7 when executing the computer-executable instructions stored in the memory.

10. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images