Speech synthesis method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides a speech synthesis method and device, electronic equipment, a computer program product and a computer readable storage medium. The method comprises the steps of determining a phoneme sequence of a first text and a second text, wherein the second text is used for describing an attribute of a first voice; performing feature extraction on the phoneme sequence to obtain phoneme features, and performing feature extraction on the second text to obtain first attribute features; based on the phoneme features, performing attention adjustment on the first attribute features to obtain second attribute features; performing fusion processing on the second attribute feature and the phoneme feature to obtain a fusion feature; and performing spectrum prediction based on the fusion feature to obtain a predicted spectrum, and performing speech synthesis based on the predicted spectrum to obtain a first speech corresponding to the first text. According to the invention, fine control of the first voice attribute can be realized, and the accuracy of voice synthesis is improved.

Description

Technical Field

[0001] The present application relates to the field of speech synthesis technology, and in particular to a speech synthesis method, device, electronic device, computer-readable storage medium and computer program product. Background Art

[0002] Text-to-Speech (TTS), also known as text-to-speech technology, is a technology that converts any text into fluent speech through computer analysis. As one of the core technologies for realizing human-computer language interaction systems, speech synthesis has received increasing attention for its application value. As speech synthesis technology is applied to various industries, in different scenarios of different industries, the synthesized speech needs to have different attributes, such as emotional attributes, pronunciation attributes, etc. Therefore, how to control the attributes of speech synthesis is an important research direction at present. Summary of the invention

[0003] The embodiments of the present application provide a speech synthesis method, device, electronic device, computer-readable storage medium and computer program product, which can integrate the attribute information of a first speech into the first speech, achieve refined control of the attributes of the first speech, and improve the accuracy of speech synthesis.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present invention provides a speech synthesis method, which comprises:

[0006] Determine a phoneme sequence of a first text and a second text, wherein the second text is used to describe a second text of a first speech attribute; perform feature extraction on the phoneme sequence to obtain phoneme features; perform feature extraction on the second text to obtain first attribute features; based on the phoneme features, perform attention adjustment on the first attribute features to obtain second attribute features; perform fusion processing on the second attribute features and the phoneme features to obtain fusion features; perform spectrum prediction based on the fusion features to obtain a predicted spectrum, and perform speech synthesis based on the predicted spectrum to obtain a first speech corresponding to the first text.

[0007] The present invention provides a speech synthesis device, comprising:

[0008] A determination module, used to determine a phoneme sequence of a first text and a second text for describing a first speech attribute;

[0009] A feature extraction module, used to perform feature extraction on the phoneme sequence to obtain phoneme features, and perform feature extraction on the second text to obtain a first attribute feature;

[0010] An adjustment module, configured to perform attention adjustment on the first attribute feature based on the phoneme feature to obtain a second attribute feature;

[0011] A fusion module, used for fusing the second attribute feature and the phoneme feature to obtain a fusion feature;

[0012] A synthesis module is used to perform spectrum prediction based on the fusion feature to obtain a predicted spectrum, and perform speech synthesis based on the predicted spectrum to obtain a first speech corresponding to the first text.

[0013] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0014] A memory for storing computer executable instructions;

[0015] The processor is used to implement the language synthesis method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0016] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the speech synthesis method provided in the embodiment of the present application when executed by a processor.

[0017] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the speech synthesis method provided in the embodiment of the present application is implemented.

[0018] The embodiments of the present application have the following beneficial effects:

[0019] Through the above embodiment, when synthesizing speech for the first text, the phoneme sequence of the first text and the second text used to describe the attributes of the first speech are obtained, and the phoneme sequence and the second text are respectively extracted to obtain phoneme features and first attribute features; then, the attention of the first attribute features is adjusted using the phoneme features, and the adjusted second attribute features are fused with the phoneme features to obtain fusion features including the phoneme information and the attribute information described by the second text, and then, the spectrum is predicted using the fusion features to obtain a predicted spectrum, and the first speech corresponding to the first text and conforming to the attributes described by the second text is generated through the predicted spectrum. In this way, by using attention adjustment and fusion processing, the attribute information described in the second text can be integrated into the phoneme features to synthesize the first speech conforming to the attributes described by the second text, and the refined control of the attributes of the first speech through the second text is achieved, thereby improving the accuracy of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1It is a structural diagram of the speech synthesis system architecture provided in an embodiment of the present application;

[0021] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0022] Figure 3A It is a flowchart of a speech synthesis method provided in an embodiment of the present application;

[0023] Figure 3B It is a schematic diagram of a process of extracting features from a phoneme sequence provided in an embodiment of the present application;

[0024] Figure 3C is a schematic diagram of a process for extracting features from a second text provided in an embodiment of the present application;

[0025] Figure 3D is a flowchart of attention adjustment provided by an embodiment of the present application;

[0026] Figure 4A is a schematic diagram of the network structure of the FastSpeech2 model provided in the embodiment of the present application;

[0027] Figure 4B is a schematic diagram of the network structure of the speech synthesis model provided in the embodiment of the present application;

[0028] Figure 5 is a schematic diagram of the network structure of the feature enhancement layer provided in an embodiment of the present application;

[0029] Figure 6 It is a schematic diagram of the predicted spectrum provided in the embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0031] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0032] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0033] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0034] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0035] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0036] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0037] 1) Phoneme sequence: A phoneme sequence is a collection of phonemes arranged in order. Each phoneme represents the smallest basic unit of speech that can distinguish meaning. In phonetics, a phoneme is defined as the smallest distinguishing feature of speech, which can be used to distinguish different languages ​​or dialects.

[0038] 2) Attention mechanism. Attention mechanism is a technology widely used in the field of deep learning. It allows the model to selectively focus on important parts and ignore unimportant information when processing information. This mechanism imitates the behavior of human attention concentration, that is, when processing tasks, people usually focus on the most critical information and ignore other interference information. In deep neural networks, attention mechanisms are often used in sequence-to-sequence (seq2seq) models, especially in the fields of machine translation, speech recognition, and natural language processing. Its basic principle is to determine which parts are important by calculating the correlation between the query, key, and value.

[0039] 3) Spectrum is a frequency representation method used for audio signal processing, such as Mel spectrum. It performs nonlinear transformation on frequency according to the auditory perception characteristics of the human ear. This transformation makes the frequency axis no longer linear, but presents a logarithmic scale, simulating the difference in the human auditory system's perception of sounds of different frequencies. In speech synthesis, spectrum is often used as an intermediate representation to generate pitch and sound quality; in speech recognition, Mel spectrum can improve the performance of the recognition system because it is closer to the processing method of the human auditory system; in audio signal processing, Mel spectrum can help achieve tasks such as sound source separation and echo cancellation.

[0040] Non-autoregressive speech synthesis models have attracted more and more attention from the industry and academia. The most popular model is the FastSpeech2 model. Its excellent speech synthesis effect and response speed have won the attention of most companies and applied it to online tasks. The input of the FastSpeech2 model is a phoneme sequence, and the output is a Mel spectrum, which is then synthesized into speech by a vocoder. However, the FastSpeech2 model cannot finely control the properties of the synthesized speech (such as pronunciation emotion). For example, in the customer service scenario or the debt collection scenario, the synthesized speech in the customer service scenario should be as gentle and pleasant as possible, while in the debt collection scenario, it should be a little stricter to guide customers to repay as soon as possible. For this reason, when synthesizing the phoneme sequence into speech, it is necessary to incorporate the information used to describe the properties of the synthesized speech into the synthesis process to achieve fine control of the properties of the synthesized speech.

[0041] The embodiments of the present application provide a speech synthesis method, device, electronic device, computer-readable storage medium and computer program product, which can integrate information describing the attributes of a first speech into the first speech, achieve refined control of the attributes of the synthesized speech, and improve the accuracy of speech synthesis.

[0042] See also Figure 1 , Figure 1It is a schematic diagram of the architecture of the speech synthesis system 100 provided in an embodiment of the present application. To support a speech synthesis application, the terminal 401 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0043] The terminal 401 is used to respond to the speech synthesis instruction and send a speech synthesis request to the server 200. The server 200 is used to respond to the speech synthesis request, obtain the phoneme sequence of the first text and the second text used to describe the attribute of the first speech from the terminal 401, and then perform feature extraction on the phoneme sequence to obtain phoneme features, and perform feature extraction on the second text to obtain first attribute features; based on the phoneme features, perform attention adjustment on the first attribute features to obtain second attribute features; perform fusion processing on the second attribute features and the phoneme features to obtain fusion features; perform spectrum prediction based on the fusion features to obtain predicted spectrum, and perform speech synthesis based on the predicted spectrum to obtain the first speech corresponding to the first text.

[0044] In some embodiments, the server 200 may directly send the processing result "first voice" in response to the speech synthesis request to the terminal 401; the terminal 401 may also actively obtain the "first voice" from the server 200.

[0045] In some embodiments, the terminal 401 can be implemented as various types of terminals such as a laptop computer, a tablet computer, a desktop computer, a set-top box, a smart phone, a smart speaker, a smart watch, a smart TV, a vehicle-mounted terminal, etc., and can also be implemented as a server. Below, an exemplary application when the device is implemented as a terminal will be described.

[0046] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0047] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device 400 provided in an embodiment of the present application, Figure 2The electronic device 400 shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the electronic device 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not described in detail. Figure 2 Various buses are labeled as bus system 440 .

[0048] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0049] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0050] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0051] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0052] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0053] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0054] A network communication module 452, used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0055] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., display screen, speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripherals and displaying content and information);

[0056] The input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.

[0057] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 The speech synthesis device 455 stored in the memory 450 is shown, which can be software in the form of a program and a plug-in, etc., and includes the following software modules: a determination module 4551, a feature extraction module 4552, an adjustment module 4553, a fusion module 4554, and a synthesis module 4555. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0058] In other embodiments, the speech synthesis device provided in the embodiments of the present application can be implemented in hardware. As an example, the speech synthesis device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the speech synthesis method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (Application Specific Integrated Circuit, ASIC), digital signal processors (Digital Signal Processor, DSP), programmable logic devices (Programmable Logic Device, PLD), complex programmable logic devices (Complex Programmable Logic Device, CPLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other electronic components.

[0059] The speech synthesis method provided by the embodiment of the present application will be described below in conjunction with the accompanying drawings. As mentioned above, the electronic device implementing the speech synthesis method of the embodiment of the present application can be a terminal 401, a server 200, or a combination of the two. Therefore, the execution subject of each step will not be repeatedly described below.

[0060] The speech synthesis method of the embodiment of the present application is described by taking the execution subject as the server 200 as an example. Figure 3A , Figure 3A is a flowchart of the speech synthesis method provided in the embodiment of the present application, which will be combined with Figure 3A The steps shown are explained.

[0061] In step 101 , a phoneme sequence of a first text and a second text are determined.

[0062] The second text is used to describe the attribute of the first speech.

[0063] In the embodiment of the present application, the first text describes the text content that needs to be converted into natural-sounding human voice, and the first text can be in the form of electronic text of written language. Phoneme is the smallest unit of speech in a language, and it is the basic unit for distinguishing meaning in a specific language; phoneme is a concept in phonetics, which is different from characters or letters in philology. In different languages, the number and characteristics of phonemes vary.

[0064] Taking the first text being Chinese as an example, the phonemes corresponding to the characters in the first text may be the pinyin, tone and rhythm of the characters, wherein the rhythm indicates the length of the pause between characters or words.

[0065] As an example, if the first text is "Hello, who are you?", the phoneme sequence corresponding to the first text may be "nin2 hao3 8qing3 wen4 nin3 shi4". Here, the number following the pinyin in the phoneme sequence represents the tone, 1 represents the first tone, 2 represents the second tone, 3 represents the third tone, 4 represents the fourth tone, 5 represents the light tone, and 6 represents the changed tone; the numbers located between the pinyin represent the rhythm, 7, 8, and 9 represent the rhythm pause duration, 7 represents a shorter pause duration, 9 represents a longer pause duration, and 8 represents a medium pause duration.

[0066] In the embodiment of the present application, the second text can be used to describe the attributes of the first voice in pronunciation emotion and intonation, such as speaking softly, harshly, happily, etc. As an example, the second text can be "speaking softly, feeling personally in the scene", which is used to indicate that the intonation of the voice synthesized for the first text is soft, so that the listener feels personally in the scene.

[0067] In actual implementation, after obtaining the first text, a corresponding phoneme sequence can be generated according to the language used in the first text. The phoneme sequence includes the phonemes corresponding to the characters in the first text, and each phoneme represents the smallest basic unit in speech that can distinguish meaning. In phonetics, a phoneme is defined as the smallest distinguishing feature of speech, which can be used to distinguish different languages ​​or dialects. Here, each phoneme in the phoneme sequence can also be called a unit phoneme, which can be understood as the smallest unit of phoneme.

[0068] In step 102, feature extraction is performed on the phoneme sequence to obtain phoneme features.

[0069] In step 103, feature extraction is performed on the second text to obtain a first attribute feature.

[0070] In actual implementation, a speech synthesis model for speech synthesis may be constructed. The speech synthesis model is based on the FastSpeech2 model of the related technology and adds a branch for feature processing for the second text.

[0071] As an example, Figure 4A The network structure diagram of the FastSpeech2 model provided in the embodiment of the present application is shown in FIG. Figure 4A The network structure of the FastSpeech2 model can include a phoneme embedding layer, a positional encoding layer, an encoder, a variation adapter, and a mel-spectrogram decoder. Figure 4A Based on the network structure shown, a network branch is added at the position of the encoder to process the second text. Figure 4B The network structure diagram of the speech synthesis model provided in the embodiment of the present application is shown in FIG. Figure 4B The newly added network branch may include a feature extraction layer, a feature enhancement layer, and a feature adjustment layer, wherein the result of the encoder is used as the input of the feature adjustment layer, and the result of the feature adjustment layer and the result of the encoder are used together as the input of the variation adapter.

[0072] In actual implementation, after constructing the speech synthesis model, the speech synthesis model needs to be trained to obtain a target speech synthesis model that meets the training end conditions. The training end conditions may include but are not limited to: the number of training times reaches a preset number of times, the loss convergence reaches a preset convergence degree, etc.

[0073] As an example, when training a speech synthesis model, the training samples may include a phoneme sample sequence, a descriptive sample text, and a standard spectrum. The phoneme sample sequence is used as the input of the phoneme embedding layer, the descriptive sample text is used as the input of the feature extraction layer, and the predicted spectrum is used as the output. The speech synthesis model is trained, and the predicted spectrum and the standard spectrum output by the speech synthesis model are input into the loss function, the loss value is calculated, and the parameters of the speech synthesis model are adjusted by back propagation according to the loss value until the training end conditions are met to obtain the target speech synthesis model.

[0074] In actual implementation, after obtaining the phoneme sequence of the first text and the second text describing the attribute of the first speech, the phoneme sequence can be used as the input of the phoneme embedding layer through the target speech synthesis model to extract the features of the phoneme sequence and obtain the phoneme features; the second text can be used as the input of the feature extraction layer of the target language synthesis model to extract the features of the second text and obtain the first attribute features. Among them, the first attribute features are obtained by extracting the attribute information of the first speech described by the second text, so the first attribute features can be used to characterize the attribute information of the first speech described by the second text.

[0075] In some embodiments, see Figure 3B , Figure 3A The step 102 “extracting features from the phoneme sequence to obtain phoneme features” may be performed through steps 1021 to 1023 .

[0076] In step 1021, the phoneme sequence is vector-converted to obtain the phoneme vector of the phoneme sequence.

[0077] In actual implementation, the phoneme sequence includes unit phonemes corresponding to each character in the first text. Therefore, when the phoneme sequence is vectorized, each unit phoneme in the phoneme sequence can be vectorized to obtain a unit phoneme vector corresponding to each unit phoneme, and all unit phoneme vectors constitute the phoneme vector of the phoneme sequence.

[0078] As an example, a phoneme sequence can be input into Figure 4B The phoneme embedding layer shown in FIG. 1 realizes vector conversion of the phoneme sequence through the phoneme embedding layer. The phoneme embedding layer can map the unit phoneme to a vector of fixed dimension, that is, the unit phoneme is mapped to a unit phoneme vector, and the information contained in the unit phoneme is captured through the unit phoneme vector. The phoneme embedding layer can obtain the phoneme embedding sequence corresponding to the phoneme sequence, that is, the phoneme vector, and the phoneme embedding sequence includes the unit phoneme vector corresponding to each unit phoneme.

[0079] In step 1022, the phoneme vector is positionally encoded to obtain a phoneme encoding feature including position information.

[0080] Position encoding is the process of adding position information to elements in sequence data (such as text). In this application, position information is added to each unit phoneme vector in the phoneme embedding sequence. When processing phoneme embedding sequences, it is necessary to understand the order between each unit phoneme vector, because the order of unit phoneme vectors is usually closely related to their semantic meaning. For example, in a sentence, the order of words affects their meaning and the overall meaning of the sentence.

[0081] In actual implementation, since the phoneme embedding layer, encoder, etc. of the FastSpeech2 model themselves do not have the ability to process the position information of sequence data, an additional position encoding layer is added between the phoneme embedding layer and the encoder to provide the position information of the phoneme embedding sequence through the additional position encoding layer.

[0082] The role of the position encoding layer is to generate a unique position vector for each position of the phoneme embedding sequence (i.e., phoneme vector) so that words or characters at different positions can be distinguished. This position vector is usually generated by calculating a set of sine and cosine functions based on the position information, and then using these function values ​​as elements of the position vector.

[0083] In this way, the position information of the phoneme vector can be obtained by positionally encoding the phoneme vector corresponding to the phoneme sequence, and the phoneme vector can be converted into a phoneme coding feature including the position information. The phoneme coding feature is a phoneme embedding sequence with a position vector represented in the form of a vector. By positionally encoding the phoneme vector, the structure of the phoneme vector can be better understood. This position coding method can be easily extended to a phoneme embedding sequence of any length, and for different positions in the phoneme embedding sequence, the encoded position vector is unique.

[0084] In step 1023, based on the position information, latent state features are extracted from the phoneme coding features, and the extracted latent state features are used as phoneme features.

[0085] In actual implementation, in order to learn higher-level text information between phonemes, such as syntax, the semantic contextual relationship between unit phonemes can be captured by extracting hidden states. Hidden states are model states that can capture the relationship between previous and next phonemes in a sequence. They represent the intrinsic state of a position in the phoneme encoding feature. This state contains the influence of the previous phoneme corresponding information and can affect subsequent states.

[0086] As an example, Figure 4BThe encoder shown can extract hidden state features of phoneme coding features based on the position information carried in the phoneme coding features, and map the phoneme coding features (a series of unit phoneme vectors with position information) into a phoneme hidden state sequence (a series of hidden states). The phoneme hidden state sequence can be called the hidden state feature of the phoneme sequence, that is, the phoneme feature.

[0087] In some embodiments, see Figure 3C , Figure 3A The step 103 “extracting features from the second text to obtain first attribute features” shown can be performed through steps 1031 to 1032 .

[0088] In step 1031, semantic features are extracted from the second text to obtain semantic features of the second text.

[0089] In actual implementation, the second text can be input into Figure 4B The feature extraction layer shown can perform semantic feature extraction on the input second text, convert the second text into a semantic feature vector that can be used by the target speech synthesis model, and obtain the semantic features of the second text.

[0090] Here, the feature extraction layer usually pays more attention to the sentence meaning information of the second text, that is, the global feature information. Therefore, the obtained semantic features usually have the global feature information of the second text.

[0091] In step 1032, feature enhancement processing is performed on the semantic feature to obtain the first attribute feature of the second text.

[0092] In actual implementation, in order to better understand the attribute meaning described in the second text, feature enhancement may be performed on the semantic features to extract global and local feature information of the second text through feature enhancement.

[0093] In some embodiments, Figure 3C Step 1032 can be implemented by the following process.

[0094] Performing local feature extraction on the semantic feature to obtain the local feature; performing semantic enhancement processing on the local feature to obtain the semantic enhancement feature; and determining the first attribute feature of the second text based on the semantic feature and the semantic enhancement feature.

[0095] In actual implementation, semantic features refer to the overall semantic features of the second text, which have global feature information of the second text. The global feature information pays more attention to the sentence meaning information of the second text. Therefore, in order to obtain more detailed feature information of the second text, the global and local semantic information of the second text can also be obtained by extracting local features from the semantic features, so as to obtain the first attribute feature with global feature information and local feature information. Local features refer to the features conveyed by a certain word in the second text, and pay more attention to the semantic information of the second text on characters or words and the association information of adjacent characters. For example, when the second text is a paragraph, the semantic features reflect the semantic information of the entire paragraph, while the local features can only reflect the information of a certain word in the entire paragraph.

[0096] As an example, local features are extracted from the semantic features of the second text to obtain the local features of the second text. Then, the local features can be semantically enhanced by any method such as deep learning, transfer learning, semantic embedding, etc., so as to learn deeper local feature information and obtain semantically enhanced features. Then, the first attribute feature of the second text having both global feature information and local feature information is obtained by using the semantic features representing the global feature information and the semantically enhanced features representing the local feature information. In some achievable methods, the first attribute feature can be obtained by weighted summing the semantic features and the semantically enhanced features, or after extracting the local features, the local features can be directly embedded in the semantic features, and then the semantic features carrying the local features can be semantically enhanced to obtain the first attribute feature of the second text.

[0097] As an example, Figure 4B The feature enhancement layer shown can be constructed by a convolutional neural network (CNN), see Figure 5 , a convolutional neural network can include convolutional layers, normalization layers, activation layers, and fully connected layers.

[0098] As an example, the semantic features are first fed into the convolution layer, which can use a series of convolution kernels to perform convolution operations on a single feature dimension; each convolution kernel generates a feature map, which is the activation value of the semantic feature during the sliding process of the convolution kernel; after the convolution operation, the convolution layer applies a nonlinear activation function, such as the ReLU function. Then, the output of the convolution layer is used as the input of the normalization layer. After the nonlinear activation function, the normalization layer normalizes the semantic features carrying local features, that is, the value of each feature is subtracted from its mean and divided by the standard deviation, and then multiplied by a learnable coefficient, which helps prevent the problem of gradient vanishing or gradient exploding during the processing process, so that the target language synthesis model converges faster. Then, the activation layer (Gaussian Error Linear Unit, GELU) is improved on the basis of the ReLU function. The activation layer combines the linear part of the ReLU function with the Gaussian error function, so that the target language synthesis model can learn a more complex model of semantic features carrying local features and increase the semantic expression ability. Finally, the fully connected layer (Linear layer) uses a learnable transformation matrix to map the input semantic features carrying local features to a new feature space and output a first attribute feature of a fixed dimension.

[0099] In the embodiment of the present application, the convolution layer can perform local feature extraction on the semantic features and enhance the ability to obtain local information. The normalization layer can normalize the data and improve the stability during feature processing. The activation layer can enhance the semantic expression ability. The fully connected layer is a dimensional transformation, which converts the larger dimensional semantic features carrying local features into feature vectors of smaller dimensionality, thereby reducing the number of parameters.

[0100] Through the above embodiment, the first attribute feature of the second text has richer feature information than the semantic feature, and has both global feature information and local feature information. In this way, when the first attribute feature is subsequently adjusted, the second attribute feature can have attributes that are more consistent with the description of the second text.

[0101] Continue reading Figure 3A , and continue with the above step 103 for explanation.

[0102] In step 104, attention adjustment is performed on the first attribute feature based on the phoneme feature to obtain a second attribute feature.

[0103] Since the attention mechanism is mainly used to process sequence data, such as natural language processing and computer vision. The core idea of ​​the attention mechanism is to simulate the way humans focus their attention, calculate the attention weight of each input position, and enable the model to dynamically assign weights according to different parts of the input data, so that the model can dynamically focus on the key parts of the input data. When processing sequence data, the attention mechanism allows the model to take into account the information at each position in the sequence, rather than just relying on input at fixed positions. In addition, in sequence data, some information may be far apart, and the attention mechanism can help the model better capture these long-distance dependencies and improve the model's ability to capture complex patterns in the sequence.

[0104] Therefore, the embodiment of the present application uses the attention mechanism to adjust the attention of the first attribute feature by paying attention to the first attribute feature through the phoneme feature, and through the mutual attention and mutual dependence between the phoneme feature and the first attribute feature, the second attribute feature obtained after the adjustment can simultaneously have the phoneme information represented by the phoneme feature and the attribute information represented by the first attribute feature. The process of adjusting the attention of the first attribute feature based on the phoneme feature can be understood as the process of preliminarily fusing the phoneme feature with the first attribute feature.

[0105] In some embodiments, see Figure 3D , Figure 3A Step 104 can be implemented through steps 1041 to 1042.

[0106] In step 1041, based on the attention mechanism, the attention weight between the phoneme feature and the first attribute feature is determined.

[0107] In some embodiments, step 1041 may be implemented in the following manner.

[0108] Based on the attention mechanism, the phoneme feature is converted into a query feature, and the first attribute feature is converted into a key feature; a dot product operation is performed on the query feature and the key feature to obtain the attention feature; based on the feature dimension of the first attribute feature, the attention feature is scaled to obtain the scaled attention feature; the scaled attention feature is normalized to obtain the attention weight between the phoneme feature and the first attribute feature.

[0109] As an example, Figure 4BThe feature adjustment layer shown can be built by the scaled dot product attention mechanism. The basic idea of ​​the scaled dot product attention mechanism is to calculate the dot product similarity between the query vector and a set of key vectors, and convert it into a probability distribution through a multi-category classification function (such as the softmax function), and then use this probability distribution to weight the value vector, so as to focus on important information.

[0110] Since the features are usually expressed as vectors when they are processed through a model, in the embodiments of the present application, the query feature may also be referred to as a query vector, the key feature may be referred to as a key vector, and the value feature may be referred to as a value vector.

[0111] In the embodiment of the present application, the attribute situation reflected by the first attribute feature can be focused on by focusing on the attention situation between the phoneme feature and the first attribute feature, thereby achieving a preliminary fusion of the phoneme feature and the first attribute feature.

[0112] In actual implementation, the first attribute feature output by the feature enhancement layer and the phoneme feature output by the encoder are used as inputs to the feature adjustment layer. Both the first attribute feature and the phoneme feature can be feature vectors represented by vectors. Therefore, the phoneme feature can be used as a query feature in the attention mechanism, that is, the corresponding query vector (Query vector), and the first attribute feature can be used as a key feature in the attention mechanism, that is, the key vector (Key vector). Since multiple attribute features of the second text can be obtained when the semantic features are extracted for the second text, that is, multiple attribute feature vectors are extracted, the first attribute feature vector corresponding to the first attribute feature usually includes multiple attribute feature vectors; similarly, when converting the phoneme sequence into the phoneme feature, each unit phoneme in the phoneme sequence is subjected to feature extraction and other processing. Therefore, the obtained phoneme feature is a phoneme hidden state sequence, and the phoneme hidden state sequence includes the hidden state features of each unit phoneme, that is, the phoneme feature vector corresponding to the phoneme feature also includes multiple hidden state feature vectors. Here, the vectors corresponding to the first attribute feature and the phoneme feature are usually multidimensional vectors. Therefore, the first attribute feature can also be regarded as the first attribute feature matrix corresponding to the first attribute feature vector, and the phoneme feature can also be regarded as the corresponding phoneme feature vector.

[0113] In the Scaled Dot Attention mechanism, the following calculations are included:

[0114] First, the dot product operation is performed on the query vector (Query vector) and the key vector (Key vector) to calculate the similarity score between the query vector and all key vectors, which measures the matching degree between the query vector and the key vector at each position.

[0115] Secondly, as the dimension of the key vector increases, the result of the dot product operation will also increase rapidly, which may cause the gradient of the multi-class classification function to become extremely small. Therefore, the result of the dot product operation can be scaled by dividing it by the dimension of the key vector. In this way, the attention scores at each position can be kept at a similar scale before being processed by the multi-class classification function, ensuring the convergence of the speech synthesis model.

[0116] Then, the result of the scaled dot product operation is applied to the multi-class classification function, and the result of the dot product operation is normalized to convert the attention score into a probability distribution. That is, the multi-class classification function can convert the attention scores at all positions into a probability form so that their sum is 1, thereby determining the relative importance of the value vector (Value vector, value feature) of each position when calculating the final output. Here, after converting the attention score into a probability distribution, the attention weight corresponding to the attention score is obtained.

[0117] Finally, the attention weight matrix is ​​dot-producted with the value matrix corresponding to the value vector (value feature) to obtain a weighted value vector, namely the context vector. The vector of each output position of the context vector is synthesized by adjusting the corresponding value vector by the attention weight.

[0118] The essence of the scaled dot product attention mechanism is to quantify the similarity between the query feature (query vector) and the key feature (key vector) through the dot product, and then assign attention weights through a multi-category classification function, and weighted sum the value features (value vectors) according to these weights to form a context-sensitive representation of each position in the input sequence. This mechanism realizes dynamic attention to different positions in the input sequence through query features, and through weighted integration, it can enhance the ability of the speech synthesis model to capture dependencies. In an embodiment of the present application, the adjusted feature obtained by adjusting the attention to the attribute feature is the context vector.

[0119] In actual implementation, based on the above calculation process, when the phoneme feature is used as the query feature (query vector) and the first attribute feature is used as the key feature (key vector), the phoneme feature is dot-producted with the first attribute feature to calculate the attention score between the phoneme feature and each attribute feature in the first attribute feature, and the attention feature between the phoneme feature and the first attribute feature is obtained. The attention feature is an attention feature matrix including the attention scores between the phoneme feature and each attribute feature. In order to ensure convergence, the attention feature is scaled based on the feature dimension of the first attribute feature to obtain the scaled attention feature. Then, the scaled attention feature is normalized by a multi-category classification function to convert the attention score in the attention feature into a probability distribution to obtain the attention weight between the phoneme feature and the first attribute feature.

[0120] In step 1042, the first attribute feature is adjusted based on the attention weight to obtain the second attribute feature.

[0121] In some embodiments, step 1042 may be implemented in the following manner.

[0122] The first attribute feature is converted into a value feature; a dot product operation is performed on the attention weight and the value feature to obtain the second attribute feature.

[0123] In actual implementation, since the present application focuses on the attention between the phoneme feature and the first attribute feature, the first attribute feature can be directly used as the value feature in the attention mechanism, and the attention weight between the first attribute feature and the phoneme feature is used to perform a dot product operation on the first attribute feature to obtain the second attribute feature. The second attribute feature can be understood as realizing the mutual attention and mutual dependence between the phoneme feature and the first attribute feature through a scaled dot product attention mechanism (Scaled Dot Attention), so that based on the attention of the two, the phoneme feature and the first attribute feature are preliminarily fused to obtain the second attribute feature that has both the attribute information reflected by the second text and the phoneme information reflected by the phoneme feature.

[0124] In actual implementation, a dot product operation is performed on the attention weight and the first attribute feature. The first attribute feature can be weighted summed using the single-dimensional attention weight data corresponding to each dimension of the attention weight matrix to obtain the single-dimensional second attribute feature corresponding to each dimension of the attention weight matrix. The single-dimensional second attribute feature constitutes the second attribute feature.

[0125] As an example, assume that query feature Q = [q_1, q_2, q_3]; key feature K = [k_1, k_2, k_3]; value feature V = [v_1, v_2, v_3]. The phoneme feature is used as the query feature Q, the first attribute feature is used as the key feature K and the value feature V, then the value feature V = [v_1, v_2, v_3] = [k_1, k_2, k_3]. Here, assume that the phoneme feature and the first attribute feature are both three-dimensional vectors, and the dot product operation is explained using the three-dimensional vector as an example. In actual processing, the phoneme feature and the first attribute feature are usually more complex feature vectors.

[0126] The attention weight matrix is ​​obtained by the dot product operation of the query feature and the key feature, that is, by the dot product operation of the phoneme feature and the first attribute feature. Here, the steps of dimension scaling and normalization can be omitted, and the result of the dot product operation is directly used as the attention weight between the phoneme feature and the first attribute feature.

[0127] The attention weight matrix a after the dot product operation is the weight distribution corresponding to the relative importance between the feature elements of the phoneme feature and the first attribute feature. For example: a = QK^T = \begin{bmatrix}q_1k_1&q_1k_2&q_1k_3\\q_2k_1&q_2k_2&q_2k_3\\q_3k_1&q_3k_2&q_3k_3\end{bmatrix}.

[0128] Then, we perform a dot product operation on the attention weight matrix a and the first attribute feature V to obtain the final context vector c (also known as the second attribute feature). The result of the dot product operation is a three-dimensional vector, each dimension is the dot product result of the phoneme feature Q and the first attribute feature K on the corresponding element. For each query q_i, q_i is weighted and summed with the values ​​v_j corresponding to all keys k_j to obtain the components c_i of the context vector. Each component c_i in the context vector is sum a_{ij}v_j. For the feature example given above, the component c_i of the context vector will be: c_i=a_{i1}v_1+a_{i2}v_2+a_{i3}v_3, and the final context vector c is: c=[c_1,c_2,c_3]. Here, c_1,c_2,c_3 are the vectors of the attention weights of the three dimensions calculated by the above process, which represent the relative importance of each attribute feature in the first attribute feature to the phoneme feature. This context vector c, namely the second attribute feature, carries both the attribute information covered by the second text and the phoneme information covered by the phoneme sequence.

[0129] In the above example, a_{i1} refers to the attention weight of the ith row and first column in the attention weight matrix, which represents the strength of association between the query q_i and the key k_1. The specific calculation is as follows:

[0130] Assume that the attention weight matrix a is as follows:

[0131] a=begin{bmatrix}q_1k_1&q_1k_2&q_1k_3\\q_2k_1&q_2k_2&q_2k_3\\q_3k_1&q_3k_2&q_3k_3\end{bmatrix}, then, a_{i1} is the attention weight of the i-th row and 1st column of the attention weight matrix, calculated as follows: a_{i1}=q_ik_1, Continuing the above example, assuming that q_i and k_1 are: q_i=[q_{i1},q_{i2},q_{i3}], k_1=[k_{11},k_{12},k_{13}]; then, the calculation result of a_{i1} is: a_{i1}=q_{i1}k_{11}+q_{i2}k_{12}+q_{i3}k_{13}. This result represents the strength of association between the query q_i and the key k_1, which is obtained by multiplying each component of the query vector with the corresponding component of the key vector and summing them. In practice, the query feature Q and the key feature K usually have more dimensions and are calculated through a complex network structure, but the basic calculation principle is the same.

[0132] Continue reading Figure 3A , and continue with the above step 104 for explanation.

[0133] In step 105, the second attribute feature and the phoneme feature are fused to obtain a fused feature.

[0134] In some embodiments, the context vector obtained based on the scaled dot product attention mechanism (Scaled Dot Attention) is consistent with the dimension of the query vector (query feature), that is, the dimension of the second attribute feature is consistent with the phoneme feature. Therefore, the second attribute feature can be directly added to the phoneme feature to achieve the fusion processing of the second attribute feature and the phoneme feature. The advantage of direct addition is that the feature dimension after addition will not increase, reducing the number of parameters, and the fusion feature obtained by direct addition can bring the attribute information of the second text to the phoneme feature.

[0135] In some embodiments, Figure 3A Step 105 can be implemented in the following manner.

[0136] Determine a first weight of the second attribute feature and a second weight of the phoneme feature; based on the first weight and the second weight, fuse the second attribute feature and the phoneme feature to obtain a fused feature.

[0137] In actual implementation, the second attribute feature and the phoneme feature are directly added to realize the process of fusing the two features, which is equivalent to treating the second attribute feature and the phoneme feature as equally important, and the degree of fusion of the two features is consistent. However, in different fusion requirements, the degree of attention paid to the second attribute feature and the phoneme feature is different, and the degree of fusion of the second attribute feature and the phoneme feature during fusion also needs to be different. Therefore, the embodiment of the present application can also assign weights to the second attribute feature and the phoneme feature respectively.

[0138] As an example, the weights can be assigned to the second attribute features and the phoneme features through a historical experience weight matrix, or through a weight matrix trained by a machine learning method. The weight elements in the weight matrix determine the degree of fusion between the second attribute features and the phoneme features between the corresponding components.

[0139] As an example, determine the first weight corresponding to each component in the second attribute feature vector corresponding to the second attribute feature, and the second weight of each component in the phoneme feature vector corresponding to the phoneme feature. For the components of the second attribute feature and the phoneme feature at corresponding positions, determine the first product of the component of the second attribute feature and the corresponding first weight, and the second product of the component of the phoneme feature and the corresponding second weight, and use the sum of the first product and the second product as the fusion value corresponding to the two components. By weighted summation of the second attribute feature and the phoneme feature at each component, the fusion processing of the second attribute feature and the phoneme feature is realized to obtain a fusion feature, which includes the fusion value of the components of the second attribute feature and the phoneme feature at corresponding positions.

[0140] Through the above embodiments, whether the second attribute feature and the phoneme feature are directly added or the second attribute feature and the phoneme feature are weighted and summed, the obtained fusion feature is a comprehensive feature vector that integrates the phoneme information and the attribute information, and each component of the comprehensive feature vector is the fusion value of the latent sequence feature and the attribute feature on the corresponding dimension. Since the fusion feature carries the attribute information, a speech corresponding to the first text and conforming to the attribute information represented by the first attribute feature can be generated based on the fusion feature, for example, a speech conforming to the emotional information represented by the attribute information can be synthesized. In addition, before the fusion process is performed, the process of adjusting the features of the first attribute feature based on the attention mechanism can be regarded as a preliminary fusion process of the phoneme feature and the first attribute feature. The attention mechanism can achieve the fusion of the two through the mutual attention and mutual dependence between the phoneme feature and the first attribute feature, so that the second attribute feature has both the attribute information described by the second text and the phoneme information represented by the phoneme feature. By fusing the second attribute feature obtained through preliminary fusion with the phoneme feature, the attribute information described by the second text can be better integrated into the phoneme feature. Moreover, since the second attribute feature has phoneme information, it can also be regarded as a phoneme feature carrying attribute information. Therefore, fusing the phoneme feature carrying attribute information with the original phoneme feature can reduce the impact on the phoneme feature, does not change the semantics represented by the phoneme feature, and reduces the negative impact on the phoneme feature.

[0141] After the phoneme feature and the second attribute feature are fused, the fused feature can have the following meanings:

[0142] On the one hand, attribute enhancement, the components in the first attribute feature vector may enhance or adjust the values ​​of the corresponding components in the phoneme feature vector, thereby producing a more attribute-colored speech during speech synthesis. For example, if the first attribute feature vector represents the emotion of happiness or excitement, it may increase the components related to happiness or excitement in the phoneme feature vector, thereby reflecting these emotional attribute characteristics in the speech output.

[0143] Second, intonation changes. Attribute information may affect the intonation of speech, that is, the intonation of speech. For example, the emotional attribute of sadness may cause the intonation to drop, while the emotional attribute of happiness may cause the intonation to rise. This change in intonation can be achieved by adjusting the pitch-related components in the phoneme feature vector.

[0144] Third, intensity adjustment: attribute information may also affect the intensity or loudness of speech. For example, strong emotional attributes may cause speech to be more forceful, thereby increasing the intensity of speech. This intensity adjustment can be achieved by adjusting the intensity-related components in the phoneme feature vector.

[0145] Fourthly, speed changes, attribute information may also affect the speed of speech. For example, emotional attributes of tension or excitement may cause faster speaking, while emotional attributes of relaxation or contemplation may slow down speaking. This change in speed can be achieved by adjusting the components related to the time interval in the phoneme feature vector. Among them, the components related to the time interval refer to the features related to the time length or duration between phonemes in the phoneme feature vector. For example, in Mandarin, different initials and finals have different pronunciation time lengths, and tones also have different durations. In speech synthesis, if you want to simulate nervous or excited emotions, you can reduce the interval time between phonemes and increase the speaking speed. On the contrary, if you want to simulate contemplative or relaxed emotions, you can increase the interval time between phonemes and reduce the speaking speed.

[0146] Finally, the target speech synthesis model can generate a corresponding speech signal based on the fused features, thereby reading out the first text corresponding to the phoneme sequence, and at the same time giving the first text corresponding attributes such as emotional color.

[0147] In step 106, spectrum prediction is performed based on the fusion features to obtain a predicted spectrum, and speech synthesis is performed based on the predicted spectrum to obtain a first speech corresponding to the first text.

[0148] In some embodiments, "predicting the spectrum based on the fusion features to obtain the predicted spectrum" in step 106 can be implemented in the following manner.

[0149] Attribute prediction is performed based on the fused features to obtain predicted attribute information; based on the predicted attribute information, spectrum mapping is performed on the fused features to obtain a predicted spectrum.

[0150] Mel spectrum is a graphical representation method often used in audio signal processing, which can reflect the spectral characteristics of audio signals and some information related to human auditory perception. Mel spectrum graphs are usually used in speech recognition, music analysis, sound separation and other audio processing tasks. Therefore, in the embodiment of the present application, after obtaining the fusion feature, the fusion feature can also be predicted to obtain the predicted spectrum corresponding to the fusion feature, and then the predicted spectrum is converted into the first speech through the vocoder.

[0151] See also Figure 4BIn the target speech synthesis model, the variation adapter is a component in speech synthesis, which is used to introduce changes in the vocoder so that the first speech has more natural variability and diversity. The main function of the variation adapter is to capture and simulate the natural variability in the speech signal, which can be caused by factors such as the speaker's personality, emotional state, pronunciation speed, and the fullness of the voice. By introducing this variability, the first speech sounds more natural and real, avoiding the problem of the first speech being too mechanical and monotonous.

[0152] In actual implementation, after receiving the fused features, the variation adapter can perform attribute prediction on the fused features, and add the predicted attribute information to the fused features, so that the spectrum decoder performs spectrum prediction based on the fused features carrying the predicted attribute information to obtain a predicted spectrum. The predicted attribute information here can represent difference attribute information such as duration, pitch, energy, etc., which is different from the attribute information represented by the second text. The difference attribute information here is predicted based on the attribute information described in the second text.

[0153] In the target speech synthesis model, the variation adapter can predict and add prediction attribute information for the fusion feature, which helps the target speech synthesis model to better understand and generate speech signals with different durations, pitches and intensities, improve the naturalness and expressiveness of speech synthesis, and make the first speech closer to real human speech. The variation adapter may include multiple predictors, each of which is responsible for predicting a type of difference attribute information. As an example, the variation adapter may include a duration predictor, a pitch predictor, and an energy predictor. Among them, the duration predictor is used to predict the duration of the current phoneme or the next phoneme in the fusion feature, which can be the absolute duration of the phoneme or the relative duration (such as the logarithm of the phoneme length); the pitch predictor is used to predict the pitch value of the phoneme in the fusion feature, which can be a value within a specific pitch range, or a fractional value based on pitch mapping; the energy predictor is used to predict the energy level of the phoneme in the fusion feature, which is usually related to the intensity of the syllable. The predictor in the variational adapter usually generates predictions based on the hidden state features of the current phoneme, the attribute information represented by the second text, and the correlation between the previous and next phonemes. Combining this variation information with the fusion features allows the target speech synthesis model to more finely control the generated sequence, thereby improving the naturalness and quality of the generated speech.

[0154] In actual implementation, the variation adapter can be an independent module or part of the target speech synthesis model. Its parameters can be optimized by the back-propagation algorithm during training so that the correct variation pattern can be learned from the data.

[0155] Through the above embodiment, when synthesizing speech for the first text, the phoneme sequence of the first text and the second text used to describe the attribute of the first speech are obtained, and the feature extraction of the phoneme sequence and the second text is performed respectively to obtain the phoneme feature and the first attribute feature; then, the attention of the first attribute feature is adjusted by using the phoneme feature, and the adjusted second attribute feature is fused with the phoneme feature to obtain a fusion feature carrying the phoneme information and the attribute information described by the second text, and then, the spectrum is predicted by using the fusion feature to obtain a predicted spectrum, and the first speech corresponding to the first text and conforming to the attribute described by the second text is generated by the predicted spectrum. In this way, by using the attention adjustment and fusion processing, the attribute information described in the second text can be integrated into the phoneme feature to synthesize the first speech conforming to the attribute described by the second text, and the refined control of the attribute of the first speech by the second text is realized, thereby improving the accuracy of speech synthesis.

[0156] In a specific embodiment, the speech synthesis method of the present application embodiment is described by taking the execution subject as the server 200 as an example. Figure 4B , build the following in server 200 Figure 4B The speech synthesis model shown.

[0157] The speech synthesis model is based on Figure 4A Based on the FastSpeech2 model shown in , a network branch for processing the second text is added at the encoder position. Figure 4B The FastSpeech2 model includes a phoneme embedding layer, a positional encoding layer, an encoder, a variation adaptor, and a mel-spectrogram decoder; the newly added network branches include a feature extraction layer, a feature enhancement layer, and a feature adjustment layer.

[0158] Among them, the phoneme embedding layer can obtain the text information between the phoneme sequences of the first text through feature extraction; the positional encoding layer is used to obtain the position information in the current input feature; the encoder converts the phoneme embedding sequence into a phoneme hidden state sequence in order to learn the text information between higher-level phonemes, such as syntax; the variation adapter contains many predictors to predict and add variation information (attribute features) such as duration, pitch, and energy to the hidden state sequence; after the variation adapter, the features corresponding to the phoneme sequence will be mapped to the same as the Mel spectrum Figure 1 The input of the mel-spectrogram decoder is the output of the variation adapter. After the mel-spectrogram decoder, the model can learn richer information and predict the predicted spectrum.

[0159] The newly added network branches include feature extraction layer, feature enhancement layer and feature adjustment layer. Among them, the feature extraction layer can extract features from the second text and convert the second text into semantic features that can be used by the speech synthesis model. The semantic features mainly include the global semantics of the second text, that is, to let the speech synthesis model understand the meaning of the second text. The feature extraction layer can be built using a pre-trained NLP Bert network structure, which has excellent performance in text semantic tasks. The feature enhancement layer can use a convolutional neural network (CNN), see Figure 5 , a convolutional neural network can include convolutional layers, normalization layers, activation layers, and fully connected layers.

[0160] The main function of the feature enhancement layer is to enhance the semantic features output by the feature extraction layer, and obtain the first attribute features with global feature information and local feature information. The feature adjustment layer is built using the scaled dot attention mechanism. The phoneme features output by the encoder and the first attribute features output by the feature enhancement layer are used as the input of the feature adjustment layer. The phoneme features are used as the query features of the scaled dot attention mechanism, and the first attribute features are used as the key features and value features of the scaled dot attention mechanism. The principle of the scaled dot attention mechanism is to dynamically pay attention to the feature information at different positions in the key features and value features through the query features, and to perform weighted integration of the feature information, thereby increasing the degree of dependency of the feature information. In other words, the phoneme features and the first attribute features are made to pay attention to each other and depend on each other, so that the phoneme features contain the attribute information described by the second text, and the second attribute features are obtained.

[0161] Then, the feature adjustment layer transmits the output second attribute feature back to the FastSpeech2 model to fuse the second attribute feature with the phoneme feature. The fusion process here can be to directly add the second attribute feature and the phoneme feature, or to weight the second attribute feature and the phoneme feature based on the experience weight matrix. The fusion feature obtained after the fusion process is then input into the subsequent variation adapter (Variance Adaptor) and spectrum decoder (Mel-spectrogram Decoder) to obtain the predicted spectrum.

[0162] In some embodiments, the target speech synthesis model is also connected to a vocoder, which is connected to a spectrum decoder. The predicted spectrum predicted by the spectrum decoder can be input into the vocoder for speech synthesis to obtain a first speech corresponding to the first text and conforming to the attributes described by the second text.

[0163] In server 200, build Figure 4B After the speech synthesis model is shown, it is necessary to perform model training on the constructed initial speech synthesis model to obtain an applicable target speech synthesis model. Among them, the training end conditions for the initial speech synthesis model to perform model training may include but are not limited to: the number of training times reaches a preset number of times, the loss convergence reaches a preset convergence degree, etc.

[0164] As an example, when training a speech synthesis model, the training sample can be <phoneme sample sequence, descriptive sample text, standard spectrum>, where the speech signal restored by the standard spectrum through the vocoder conforms to the attribute meaning described by the descriptive sample text. The speech synthesis model is trained with the phoneme sample sequence as the input of the phoneme embedding layer, the descriptive sample text as the input of the feature extraction layer, and the predicted spectrum as the output. The predicted spectrum and the standard spectrum output by the speech synthesis model are input into the loss function, the loss value is calculated, and the parameters of the speech synthesis model are adjusted by backpropagation according to the loss value until the training end conditions are met to obtain the target speech synthesis model.

[0165] As an example, after the phoneme sample sequence is input into the initial speech synthesis model, the phoneme sample sequence is transformed into a discrete feature a through the phoneme dictionary; the phoneme sample sequence is subjected to feature extraction through the phoneme embedding layer and converted into a phoneme embedding sequence, which is a continuous feature b1; the discrete feature a can determine the relative position information or the absolute position information b2 after passing through the position encoding layer; the continuous feature b1 carrying the position information b2 is input into the encoder, which is composed of several layers of Transformer Encoder. Thanks to the excellent feature learning ability of Transformer Encoder, the output feature vector (phoneme feature) of the encoder contains richer information. The descriptive sample text is input into the pre-trained feature extraction layer to obtain the semantic feature, and then the local information of the semantic feature is obtained through the feature enhancement layer to obtain the first attribute feature, which has both the global feature information of the descriptive sample text and the local information of the descriptive sample text. The first attribute feature is used as the key feature K and value feature V of the scaled dot product attention mechanism in the feature adjustment layer, and the phoneme feature is used as the query feature Q. The second attribute feature obtained by the interaction between the phoneme feature and the first attribute feature is added to the output (phoneme feature) of the encoder and input into the variation adapter for feature learning of duration, pitch, etc., and is upsampled to a feature vector with the same dimension as the standard spectrum by the length adjuster in the variation adapter, and then the position information is learned by the position encoder. The spectrum decoder is composed of several layers of transformer encoder layers, which can learn richer feature information and predict the predicted spectrum. At this time, the dimension of the predicted spectrum is consistent with that of the standard spectrum. The predicted spectrum and the standard spectrum are input into the loss function, the loss data is calculated, and the parameters of the initial language synthesis model are adjusted according to the back propagation of the loss data. After multiple rounds of iterations, when the loss converges, the difference between the predicted spectrum and the standard spectrum will become smaller and smaller. When the speech synthesis model meets the training end conditions, the model is saved to obtain the target speech synthesis model.

[0166] When the target speech synthesis model is applied to implement the speech synthesis method of the embodiment of the present application, the following process may be included:

[0167] First, load the saved target speech synthesis model, and input the phoneme sequence of the first text and the second text describing the attributes of the first speech. The first text is "Hello, may I ask who you are?", then the phoneme sequence corresponding to the first text can be "nin2 hao3 8qing3 wen4 nin3 shi4". The second text can be "Speaking gently, feeling immersive."

[0168] See also Figure 6 , through the processing of the phoneme sequence and the second text by the target speech synthesis model, the output of the target speech synthesis model can be obtained as follows Figure 6 The predicted spectrum is shown.

[0169] Then, the predicted spectrum is input into a vocoder and converted into a speech signal by the vocoder to obtain a first speech corresponding to the first text and conforming to the attributes described by the second text.

[0170] The following is a description of an exemplary structure of the speech synthesis device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the speech synthesis device 455 of the memory 440 may include:

[0171] The determination module 4551 is used to determine the phoneme sequence of the first text and the second text, where the second text is used to describe the attributes of the first speech.

[0172] The feature extraction module 4552 is used to extract features from the phoneme sequence to obtain phoneme features.

[0173] Feature extraction is performed on the second text to obtain the first attribute feature.

[0174] The adjustment module 4553 is used to adjust the attention of the first attribute feature based on the phoneme feature to obtain the second attribute feature.

[0175] The fusion module 4554 is used to fuse the second attribute feature and the phoneme feature to obtain a fusion feature.

[0176] The synthesis module 4555 is used to perform spectrum prediction based on the fused features to obtain a predicted spectrum, and perform speech synthesis based on the predicted spectrum to obtain a first speech corresponding to the first text.

[0177] In some embodiments, the adjustment module 4553 is further used to determine the attention weight between the phoneme feature and the first attribute feature based on the attention mechanism; based on the attention weight, the first attribute feature is adjusted to obtain the second attribute feature.

[0178] In some embodiments, the adjustment module 4553 is also used to convert the phoneme feature into a query feature based on the attention mechanism, and convert the first attribute feature into a key feature; perform a dot product operation on the query feature and the key feature to obtain the attention feature; scale the attention feature based on the feature dimension of the first attribute feature to obtain the scaled attention feature; normalize the scaled attention feature to obtain the attention weight between the phoneme feature and the first attribute feature.

[0179] In some embodiments, the adjustment module 4553 is further used to convert the first attribute feature into a value feature; perform a dot product operation on the attention weight and the value feature to obtain a second attribute feature.

[0180] In some embodiments, the fusion module 4554 is further used to determine a first weight of the second attribute feature and a second weight of the phoneme feature; based on the first weight and the second weight, the second attribute feature and the phoneme feature are fused to obtain a fused feature.

[0181] In some embodiments, the feature extraction module 4552 is also used to perform vector conversion on the phoneme sequence to obtain the phoneme vector of the phoneme sequence; perform position encoding on the phoneme vector to obtain the phoneme coding feature including the position information; based on the position information, perform latent state feature extraction on the phoneme coding feature, and use the extracted latent state feature as the phoneme feature.

[0182] In some embodiments, the feature extraction module 4552 is further used to extract semantic features from the second text to obtain semantic features of the second text; and perform feature enhancement processing on the semantic features to obtain first attribute features of the second text.

[0183] In some embodiments, the feature extraction module 4552 is further used to perform local feature extraction on the semantic feature to obtain the local feature; perform semantic enhancement processing on the local feature to obtain the semantic enhancement feature; and determine the first attribute feature of the second text based on the semantic feature and the semantic enhancement feature.

[0184] In some embodiments, the synthesis module 4555 is further used to perform attribute prediction based on the fused features to obtain predicted attribute information; and perform spectrum mapping on the fused features based on the predicted attribute features to obtain a predicted spectrum.

[0185] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer readable storage medium, and the processor executes the computer executable instruction, so that the electronic device performs the speech synthesis method described in the embodiment of the present application.

[0186] The present application embodiment provides a computer-readable storage medium in which computer executable instructions or computer programs are stored. When the computer executable instructions or computer programs are executed by a processor, the processor will execute the speech synthesis method provided by the present application embodiment, for example, Figure 3A The speech synthesis method is shown.

[0187] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0188] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0189] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0190] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0191] In summary, when synthesizing speech for a first text through the embodiment of the present application, the phoneme sequence of the first text and the second text used to describe the attributes of the first speech are obtained, and the phoneme sequence and the second text are respectively extracted to obtain phoneme features and first attribute features; then, the attention of the first attribute features is adjusted using the phoneme features, and the adjusted second attribute features are fused with the phoneme features to obtain fused features carrying phoneme information and attribute information described by the second text, and then, the fused features are used to predict the spectrum to obtain a predicted spectrum, and the first speech corresponding to the first text and conforming to the attributes described by the second text is generated through the predicted spectrum. In this way, by using attention adjustment and fusion processing, it is possible to integrate the attribute information described in the second text into the phoneme features to synthesize the first speech conforming to the attributes described by the second text, and to achieve refined control of the attributes of the first speech through the second text, thereby improving the accuracy of the first speech.

[0192] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A speech synthesis method, characterized in that: The method comprises: Determining a phoneme sequence of a first text and a second text, wherein the second text is used to describe a property of the first speech; Extracting features from the phoneme sequence to obtain phoneme features; Performing feature extraction on the second text to obtain a first attribute feature; Based on the phoneme feature, attention adjustment is performed on the first attribute feature to obtain a second attribute feature; fusing the second attribute feature and the phoneme feature to obtain a fused feature; Spectrum prediction is performed based on the fusion features to obtain a predicted spectrum, and speech synthesis is performed based on the predicted spectrum to obtain a first speech corresponding to the first text.

2. The method according to claim 1, characterized in that The step of adjusting the attention of the first attribute feature based on the phoneme feature to obtain the second attribute feature includes: Based on the attention mechanism, determining the attention weight between the phoneme feature and the first attribute feature; Based on the attention weight, the first attribute feature is adjusted to obtain a second attribute feature.

3. The method according to claim 2, characterized in that The determining, based on the attention mechanism, the attention weight between the phoneme feature and the first attribute feature comprises: Based on an attention mechanism, the phoneme feature is converted into a query feature, and the first attribute feature is converted into a key feature; Performing a dot product operation on the query feature and the key feature to obtain an attention feature; Based on the feature dimension of the first attribute feature, scaling the attention feature to obtain a scaled attention feature; The scaled attention feature is normalized to obtain an attention weight between the phoneme feature and the first attribute feature.

4. The method according to claim 3, characterized in that The adjusting the first attribute feature based on the attention weight to obtain the second attribute feature includes: Converting the first attribute feature into a value feature; A dot product operation is performed on the attention weight and the value feature to obtain a second attribute feature.

5. The method according to claim 1, characterized in that The fusing the second attribute feature and the phoneme feature to obtain a fused feature includes: determining a first weight of the second attribute feature and a second weight of the phoneme feature; Based on the first weight and the second weight, the second attribute feature and the phoneme feature are fused to obtain a fused feature.

6. The method according to claim 1, characterized in that The step of extracting features from the phoneme sequence to obtain phoneme features includes: Performing vector conversion on the phoneme sequence to obtain a phoneme vector of the phoneme sequence; Performing position encoding on the phoneme vector to obtain a phoneme encoding feature including position information; Based on the position information, latent state features are extracted from the phoneme coding features, and the extracted latent state features are used as the phoneme features.

7. The method according to claim 1, characterized in that The extracting features of the second text to obtain the first attribute features includes: Extracting semantic features from the second text to obtain semantic features of the second text; Perform feature enhancement processing on the semantic feature to obtain a first attribute feature of the second text.

8. The method according to claim 7, characterized in that The performing feature enhancement processing on the semantic feature to obtain the first attribute feature of the second text includes: Performing local feature extraction on the semantic feature to obtain a local feature; Performing semantic enhancement processing on the local features to obtain semantic enhancement features; Based on the semantic feature and the semantic enhancement feature, a first attribute feature of the second text is determined.

9. The method according to claim 1, characterized in that: The performing spectrum prediction based on the fusion feature to obtain a predicted spectrum includes: Perform attribute prediction based on the fused features to obtain predicted attribute information; Based on the predicted attribute information, spectrum mapping is performed on the fused features to obtain a predicted spectrum.

10. A speech synthesis device, characterized in that: The device comprises: A determination module, used to determine a phoneme sequence of a first text and a second text, wherein the second text is used to describe an attribute of the first speech; A feature extraction module, configured to perform feature extraction on the phoneme sequence to obtain phoneme features, and perform feature extraction on the second text to obtain first attribute features; an adjustment module, configured to perform attention adjustment on the first attribute feature based on the phoneme feature to obtain a second attribute feature; A fusion module, used for fusing the second attribute feature and the phoneme feature to obtain a fusion feature; A synthesis module is used to perform spectrum prediction based on the fusion feature to obtain a predicted spectrum, and perform speech synthesis based on the predicted spectrum to obtain a first speech corresponding to the first text.

11. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions; A processor, configured to implement the method according to any one of claims 1 to 9 when executing the computer executable instructions stored in the memory.

12. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

13. A computer program product comprising computer executable instructions or a computer program, characterized in that: When the computer executable instructions or computer programs are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images