Speech synthesis method and device, computer device and storage medium

Abstract

The application relates to the technical field of financial technology, and discloses a speech synthesis method and device, computer equipment and a storage medium. The method comprises the following steps: obtaining a phoneme vector of a target text; performing mask processing on the phoneme vector of the target text based on a pre-trained Attention mask matrix to obtain a masked phoneme vector; performing duration prediction based on the masked phoneme vector to obtain a predicted phoneme duration; performing inverse mapping processing on the masked phoneme vector based on the predicted phoneme duration through a reverse flow model to obtain hidden variables corresponding to each masked phoneme vector; and decoding the hidden variables through a pre-trained decoder to obtain a target speech signal. The phoneme vector is subjected to mask processing through the Attention mask matrix to obtain the masked phoneme vector, so that the random duration predictor is more inclined to sample the phoneme vector which has not been subjected to mask processing in the masked phoneme vector, thereby achieving the purpose of controlling the sampling process.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of financial technology, and in particular to a speech synthesis method and device, computer equipment and a storage medium. BACKGROUND

[0002] With the development of science and technology, intelligent speech synthesis technology (such as VITS technology) has been applied in many technical fields, such as the technical field of financial technology, the technical field of intelligent reading, etc.

[0003] For example, in the financial field, when a user understands the related information of life insurance through a mobile phone installed financial software, the user can identify the related text information of life insurance through online reading function and synthesize speech for reading.

[0004] However, in actual application, the VITS technology is to randomly sample in the phoneme duration distribution fitted to achieve the purpose of generating different speech of multiple sentences, but due to the randomness of sampling, it is difficult to control the sampling process, resulting in the inability to control the generated speech effect. SUMMARY

[0005] The embodiments of the present application provide a speech synthesis method, device, computer equipment and storage medium to solve the problem that the VITS model cannot control the generated speech effect due to the control of the sampling process when generating speech.

[0006] A speech synthesis method, the method comprising:

[0007] obtaining a phoneme vector of a target text through a text encoder;

[0008] masking at least part of the phoneme vector of the target text based on a pre-trained Attention mask matrix to obtain a masked phoneme vector;

[0009] predicting a phoneme duration based on the masked phoneme vector through a random duration predictor to obtain a predicted phoneme duration;

[0010] performing inverse mapping processing on the masked phoneme vector based on the predicted phoneme duration through a reverse flow model to obtain an implicit variable corresponding to each of the masked phoneme vectors;

[0011] decoding the implicit variable through a pre-trained decoder to obtain a target speech signal.

[0012] The above method, optionally, the Attention mask matrix is obtained by training in the following way:

[0013] obtaining a training text sample and a training speech sample;

[0014] inputting the training text sample and the training speech sample into the VITS model for model training, and training an Attention mask matrix based on the training text sample and a preset target vocabulary set;

[0015] calculating the total loss value of the VITS model to determine whether the VITS model is sufficiently trained;

[0016] if the VITS model is sufficiently trained, determining that the Attention mask matrix is trained.

[0017] Optionally, the vocabulary in the target vocabulary set is any one of a noun, a verb, and an interjection.

[0018] Optionally, the vocabulary in the target vocabulary set is a plurality of vocabularies of a noun, a verb, and an interjection, and each of the vocabularies carries a corresponding part-of-speech label.

[0019] Optionally, the mask processing unit is configured to perform mask processing on the phoneme vector based on the pre-trained Attention mask matrix by using a random duration predictor to obtain a masked phoneme vector, and the mask processing includes:

[0020] point-multiplying the Attention value in the Attention mask matrix with the corresponding phoneme vector to obtain a point-multiplication result;

[0021] splicing the point-multiplication result and the phoneme vector to obtain the masked phoneme vector.

[0022] A speech synthesis device, the device comprising:

[0023] a vector acquisition unit configured to acquire a phoneme vector of a target text by using a text encoder;

[0024] a mask processing unit configured to perform mask processing on at least part of the phoneme vector of the target text based on a pre-trained Attention mask matrix to obtain a masked phoneme vector;

[0025] a duration prediction unit configured to perform duration prediction based on the masked phoneme vector by using a random duration predictor to obtain a predicted phoneme duration;

[0026] a variable mapping unit configured to perform inverse mapping processing on the masked phoneme vector based on the predicted phoneme duration by using a reverse flow model to obtain an implicit variable corresponding to each of the masked phoneme vectors;

[0027] A voice synthesis unit is configured to decode the latent variable by using a pre-trained decoder to obtain a target voice signal.

[0028] Optionally, the Attention mask matrix is trained by the following method:

[0029] Obtain a training text sample and a training voice sample.

[0030] Input the training text sample and the training voice sample into a VITS model for model training, and train an Attention mask matrix based on the training text sample and a preset target vocabulary set.

[0031] Calculate the total loss value of the VITS model to determine whether the VITS model is sufficiently trained.

[0032] If the VITS model is sufficiently trained, it is determined that the trained Attention mask matrix is obtained.

[0033] Optionally, the vocabulary in the target vocabulary set is any one of a noun, a verb, and an interjection.

[0034] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the above voice synthesis method when executing the computer program.

[0035] A computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the above voice synthesis method.

[0036] The voice synthesis method, device, computer device, and storage medium perform mask processing on at least part of the phoneme vectors of the target text by using the pre-trained Attention mask matrix to obtain masked phoneme vectors, then perform duration prediction on the masked phoneme vectors by using a random duration predictor to obtain predicted phoneme durations, and obtain a target voice signal based on the predicted phoneme durations and the masked phoneme vectors. As can be seen, the voice synthesis method performs mask processing on at least part of the phoneme vectors of the target text by using the pre-trained Attention mask matrix to obtain masked phoneme vectors, so that the random duration predictor is more inclined to sample the phoneme vectors that are not masked in the masked phoneme vectors to obtain predicted phoneme durations, and finally obtain a target voice signal. Therefore, the voice synthesis method can control the sampling process of the VITS model by performing mask processing on at least part of the phoneme vectors of the target text by using the Attention mask matrix, so as to control the generation effect of the target voice signal. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0038] Figure 1 is a flow diagram of a speech synthesis method disclosed by an embodiment of the present application;

[0039] Figure 2 is a partial flow diagram of a speech synthesis method disclosed by an embodiment of the present application;

[0040] Figure 3 is a partial flow diagram of a speech synthesis method disclosed by an embodiment of the present application;

[0041] Figure 4 is a structural diagram of a speech synthesis device disclosed by an embodiment of the present application;

[0042] Figure 5 is a structural diagram of a computer device disclosed by an embodiment of the present application. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0044] It should be understood that when used in the specification and the appended claims of the present application, the term "comprising" indicates the presence of the described features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0045] It should also be understood that the term "and / or" used in the specification and the appended claims of the present application means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0046] As used in the description of the application and the appended claims, the term "if' can be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [a described condition or event] is detected" can be interpreted to mean "upon determining" or "in response to determining" or "upon detecting [the described condition or event]" or "in response to detecting [the described condition or event]," depending on the context.

[0047] In addition, the terms "first", "second", "third", etc. as used in the description of the application and the appended claims are merely used to differentiate descriptions and cannot be understood as indicating or implying relative importance.

[0048] Reference in the specification to "one embodiment" or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase "in one embodiment" or "in some embodiments" in various places in the specification are not necessarily all referring to the same embodiment, although it can. The terms "including", "containing", "having" and variations thereof mean "including but not limited to", unless expressly specified otherwise.

[0049] The application discloses a speech synthesis method and device, computer equipment and storage medium. At least part of the phoneme vectors of the target text is masked by the pre-trained Attention mask matrix to obtain the masked phoneme vector, and then the random duration predictor is used to predict the duration of the phoneme vector to obtain the predicted phoneme duration. Based on the predicted phoneme duration and the masked phoneme vector, the target speech signal is obtained. It can be seen that the at least part of the phoneme vectors of the target text is masked by the pre-trained Attention mask matrix to obtain the masked phoneme vector, so that the random duration predictor is more inclined to sample the phoneme vector which is not masked in the masked phoneme vector, and the predicted phoneme duration is obtained, and finally the target speech signal is obtained. Therefore, the sampling process of the VITS model can be controlled by masking at least part of the phoneme vectors of the target text by the Attention mask matrix, so as to achieve the purpose of controlling the generation effect of the target speech signal. The following will be described by specific embodiments.

[0050] As Figure 1As shown, an implementation flowchart of a speech synthesis method disclosed in an embodiment of the application is shown. The method is suitable for electronic devices with natural language processing capabilities, such as mobile phones, tablet computers, personal computers, servers, and the like. The method in this embodiment can include the following steps:

[0051] S101: Obtain the phoneme vector of the target text through the text encoder.

[0052] After obtaining the target text, the target text is converted into target text phonemes, and then the target text phonemes are encoded by the text encoder to obtain the phoneme vector.

[0053] In a specific implementation, the text encoder in this embodiment can be a transformer model encoder. The target text phonemes are extracted from the target text and input into the text encoder to obtain the phoneme vector of each target text phoneme.

[0054] S102: Perform mask processing on at least part of the phoneme vector of the target text based on the pre-trained Attention mask matrix to obtain a masked phoneme vector.

[0055] Through pre-training, the Attention value in the Attention mask matrix is obtained, and the phoneme vector of the target text is masked processed according to the Attention value to obtain a masked phoneme vector.

[0056] In a specific implementation, the Attention value in the Attention mask matrix will be different according to different training samples used for pre-training of the Attention mask matrix, and different mask processing is performed on the phoneme vector of the target text according to the different Attention values in the Attention mask matrix to obtain different masked phoneme vectors. That is, by controlling the training samples used for pre-training of the Attention mask matrix, the Attention value in the Attention mask matrix can be controlled, and the phoneme vector of the target text is correspondingly masked processed to obtain the desired masked phoneme vector. The masked phoneme vector includes the masked phoneme vector and the unmasked phoneme vector, that is, the masked phoneme vector refers to the set of masked phoneme vectors and unmasked phoneme vectors.

[0057] S103: Perform duration prediction based on the masked phoneme vector through the random duration predictor to obtain a predicted phoneme duration.

[0058] The mask phoneme vector is input into a stochastic duration predictor to perform time prediction according to the mask phoneme vector to obtain a predicted phoneme duration.

[0059] In a specific implementation, the stochastic duration predictor is a flow-based generative model. After the mask phoneme vector is input into the stochastic duration predictor, the stochastic duration predictor is more inclined to sample the phoneme vector that is not masked in the mask phoneme vector, so that the synthesized speech signal is more obviously different in the text part corresponding to the phoneme vector that is not masked, and the purpose of controllable random sampling process of the stochastic duration predictor is achieved.

[0060] S104: The mask phoneme vector is inversely mapped based on the predicted phoneme duration by using a reverse flow model to obtain an implicit variable corresponding to each mask phoneme vector.

[0061] In a specific implementation, the reverse flow model in this embodiment can be obtained by training based on a flow model. The reverse flow model has learned the corresponding relationship between the implicit variable and the phoneme vector in advance, and then obtains the implicit variable corresponding to each mask phoneme vector according to the mapping relationship between the mask phoneme vector and the implicit variable.

[0062] S105: The implicit variable is decoded by using a pre-trained decoder to obtain a target speech signal.

[0063] In a specific implementation, the decoder in this embodiment can be a vocoder. The implicit variable is input into the vocoder to obtain the target speech signal. The implicit variable is decoded by using the vocoder to obtain the target speech signal. Accordingly, the target speech signal can be generated based on the target text. The decoder is a pre-trained vocoder. The decoder has learned the conversion relationship between the implicit variable and the speech signal in advance.

[0064] To sum up, the application discloses a speech synthesis method, which performs mask processing on at least part of the phoneme vectors of the target text through a pre-trained Attention mask matrix to obtain masked phoneme vectors, then performs duration prediction on the masked phoneme vectors through a random duration predictor to obtain predicted phoneme durations, and obtains a target speech signal based on the predicted phoneme durations and the masked phoneme vectors. As can be seen, the application performs mask processing on at least part of the phoneme vectors of the target text through a pre-trained Attention mask matrix to obtain masked phoneme vectors, so that the random duration predictor is more inclined to sample the phoneme vectors that have not been masked in the masked phoneme vectors, and the predicted phoneme durations are obtained, and finally the target speech signal is obtained. Therefore, mask processing on at least part of the phoneme vectors of the target text through the Attention mask matrix can control the sampling process of the VITS model, so as to control the generation effect of the target speech signal.

[0065] Based on Figure 1 In a specific implementation, the Attention mask matrix can be trained in the following manner, as shown in the following formula: Figure 2

[0066] Specifically, the method in the embodiment is implemented based on a VITS model. When the VITS model is trained, the Attention mask matrix is trained synchronously to obtain the Attention values in the Attention mask matrix. When the training of the VITS model is completed, the trained Attention mask matrix is obtained. The training process is specifically as follows:

[0067] S201: Obtain a training text sample and a training speech sample.

[0068] In a specific implementation, when the VITS model in the embodiment is trained, the training text sample and the training speech sample need to be input simultaneously. The training text sample and the training speech sample correspond to each other. For example, the text content of the training text sample is in the form of text, that is, “I love China”. The speech content of the training language sample is in the form of speech, that is, “I love China”. Based on this, after the training text sample and the training speech sample are input into the VITS model, the VITS model learns the corresponding relationship between the training text phonemes of the training text sample and the speech signal of the training speech sample, so as to generate a target speech signal according to a target text.

[0069] S202: Input the training text sample and the training speech sample into the VITS model for model training, and train the Attention mask matrix based on the training text sample and a preset target vocabulary set.

[0070] ​After the training text sample and the training speech sample are input into the VITS model, the training text sample is converted into a training text phoneme, the training speech sample is converted into a training hidden variable, and then model training is performed based on the training text phoneme and the training hidden variable. When training the VITS model, Attention mask matrix training is performed based on the training text phoneme and a preset target vocabulary set to obtain an Attention value in the Attention mask matrix, until the VITS model training is completed, and the required Attention mask matrix is determined.

[0071] S203: Calculate the total loss value of the VITS model to determine whether the VITS model is sufficiently trained.

[0072] If the VITS model is sufficiently trained, step S204 is performed, that is, the trained Attention mask matrix is determined, and if the VITS model is not sufficiently trained, iterative training is continued based on the training text sample and the training speech sample until the VITS model is sufficiently trained.

[0073] In a specific implementation, the total loss value of the VITS model includes a reconstruction loss value, an adversarial loss value, a feature discriminator loss value, and the like, and the total loss value of the VITS model is calculated based on these loss values. During the training of the VITS model, when the calculated total loss value no longer decreases, it is determined whether the VITS model is sufficiently trained, step S204 is performed, and when the calculated total loss value is still decreasing, iterative training is continued based on the training text sample and the training speech sample. In the iterative training process, the model parameters of the VITS model are constantly updated, and the corresponding total loss value is calculated, until the total loss value no longer decreases, and step S204 is performed.

[0074] S204: Determine the trained Attention mask matrix.

[0075] After obtaining the trained Attention mask matrix, the target speech signal corresponding to the target text can be generated according to the Attention mask, and then speech playback is performed based on the target speech signal.

[0076] In summary, in the embodiment, the Attention mask matrix is trained based on the training process of the VITS model, which can control the speech effect generated by the VITS model while not significantly increasing the training cost of the VITS model.

[0077] Based on Figure 2 In a specific implementation, the vocabulary in the target vocabulary set in the embodiment is any one of a noun, a verb, and an interjection.

[0078] In a specific implementation, the parts of speech of the words in the target word set in the embodiment can be controlled as needed, for example, the words in the target word set can all be nouns, or the words in the target word set can all be interjections, etc. According to the phonemes of the words with a single part of speech in the target word set and the training text phonemes of the training text sample, an Attention mask matrix is trained through an attention mechanism.

[0079] According to the generated Attention mask matrix, the phoneme vectors of other parts of speech in the phoneme vector of the target text can be masked, so that the sampling of the phoneme vector of the part of speech by the random duration predictor is more aggressive, thereby controlling the sampling process of the random duration predictor.

[0080] Based on Figure 2 In a specific implementation, the words in the target word set in the embodiment are any one or more of nouns, verbs, and interjections, and each word carries a corresponding part of speech label.

[0081] The words in the target word set can include words of multiple parts of speech, and in order to ensure the training effect of the Attention mask matrix, the words of different parts of speech need to be labeled with magnetic labels, so that the Attention mask matrix trained can make the random duration predictor tend to sample the phoneme vectors of multiple parts of speech of the target text, thereby controlling the sampling process of the random duration predictor.

[0082] Based on Figure 1 In a specific implementation, step S103 in the embodiment can be implemented through the following steps, as shown in Figure 3

[0083] S301: Multiply the Attention values in the Attention mask matrix with the corresponding phoneme vectors respectively to obtain a point multiplication result.

[0084] S302: Concatenate the point multiplication result with the phoneme vector to obtain a masked phoneme vector.

[0085] Wherein, the Attention values in the Attention mask matrix and the phoneme vectors of the target text both conform to the encoder-decoder structure, and then the point multiplication is performed according to the matrix structure, that is, each phoneme vector of the target text is multiplied with the Attention values in the Attention mask matrix, and then the result after the point multiplication is concatenated with the original phoneme vector, thereby realizing the masking processing of the phoneme vector to obtain a masked phoneme vector. ​

[0086] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the application.

[0087] As shown in Figure 4 The structure diagram of a speech synthesis device disclosed by the embodiment of the application, which is suitable for electronic devices with natural language processing capabilities, such as mobile phones, tablet computers, personal computers, servers, and the like.

[0088] Specifically, the speech synthesis device in the embodiment can include the following units:

[0089] The vector acquisition unit 401 is configured to acquire the phoneme vector of the target text through a text encoder;

[0090] The mask processing unit 402 is configured to perform mask processing on at least part of the phoneme vector of the target text based on a pre-trained Attention mask matrix to obtain a masked phoneme vector;

[0091] The duration prediction unit 403 is configured to perform duration prediction on the masked phoneme vector based on a random duration predictor to obtain a predicted phoneme duration;

[0092] The variable mapping unit 404 is configured to perform inverse mapping processing on the masked phoneme vector based on the predicted phoneme duration through a reverse flow model to obtain the hidden variable corresponding to each of the masked phoneme vectors;

[0093] The speech synthesis unit 405 is configured to decode the hidden variable through a pre-trained decoder to obtain a target speech signal.

[0094] In summary, the application discloses a speech synthesis device, which performs mask processing on at least part of the phoneme vector of the target text through a pre-trained Attention mask matrix to obtain a masked phoneme vector, then performs duration prediction on the masked phoneme vector through a random duration predictor to obtain a predicted phoneme duration, and obtains a target speech signal based on the predicted phoneme duration and the masked phoneme vector. As can be seen, the application performs mask processing on at least part of the phoneme vector of the target text through a pre-trained Attention mask matrix to obtain a masked phoneme vector, so that the random duration predictor is more inclined to sample the phoneme vector that has not been masked in the masked phoneme vector, and a predicted phoneme duration is obtained, and finally a target speech signal is obtained. Therefore, mask processing on at least part of the phoneme vector of the target text through the Attention mask matrix can control the sampling process of the VITS model, so as to achieve the purpose of controlling the generation effect of the target speech signal.

[0095] In an implementation manner, the Attention mask matrix is trained by the following manner:

[0096] obtain a training text sample and a training speech sample;

[0097] input the training text sample and the training speech sample into the VITS model for model training, and train the Attention mask matrix based on the training text sample and a preset target vocabulary set;

[0098] calculate the total loss value of the VITS model to determine whether the VITS model is sufficiently trained;

[0099] if the VITS model is sufficiently trained, it is determined that the Attention mask matrix is trained.

[0100] In an implementation manner, the vocabulary in the target vocabulary set is any one of a noun, a verb, and an interjection.

[0101] In an implementation manner, the vocabulary in the target vocabulary set is a plurality of vocabularies of a noun, a verb, and an interjection, and each of the vocabularies carries a corresponding part-of-speech label.

[0102] In an implementation manner, the mask processing unit 402 can be used for:

[0103] multiply the Attention value in the Attention mask matrix by the corresponding phoneme vector respectively to obtain a stationary point multiplication result;

[0104] splicing the stationary point multiplication result and the phoneme vector to obtain a mask phoneme vector.

[0105] For specific limitations of the speech synthesis device, refer to the relevant limitations of the speech synthesis method in the above, which will not be repeated here. Each module in the above speech synthesis device can be realized by software, hardware, and combinations thereof, in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to call and execute the operations corresponding to each module by the processor.

[0106] In one embodiment, a computer device is disclosed, which can be a server, and its internal structure diagram can be as shown in Figure 5As shown in the figure. The computer device includes a processor, a memory, a network interface and a database connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium, an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with the external terminal through the network connection. The computer program is executed by the processor to implement a speech synthesis method.

[0107] In one embodiment, a computer device is provided, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor executing the computer program to implement the following steps:

[0108] Through the text encoder, the phoneme vector of the target text is obtained;

[0109] Based on the pre-trained Attention mask matrix, at least part of the phoneme vectors of the target text are masked to obtain masked phoneme vectors;

[0110] Through the random duration predictor, duration prediction is performed based on the masked phoneme vectors to obtain predicted phoneme durations;

[0111] Through the reverse flow model, the masked phoneme vectors are inversely mapped based on the predicted phoneme durations to obtain the corresponding latent variables of each masked phoneme vector;

[0112] The latent variables are decoded by the pre-trained decoder to obtain the target speech signal.

[0113] In summary, the present application discloses a computer device, which masks at least part of the phoneme vectors of the target text through the pre-trained Attention mask matrix to obtain masked phoneme vectors, then performs duration prediction on the masked phoneme vectors through the random duration predictor to obtain predicted phoneme durations, and obtains the target speech signal based on the predicted phoneme durations and the masked phoneme vectors. It can be seen that the present application masks at least part of the phoneme vectors of the target text through the pre-trained Attention mask matrix to obtain masked phoneme vectors, so that the random duration predictor is more inclined to sample the phoneme vectors that have not been masked in the masked phoneme vectors, and the predicted phoneme durations are obtained, and finally the target speech signal is obtained. Therefore, by masking at least part of the phoneme vectors of the target text through the Attention mask matrix, the sampling process of the VITS model can be controlled to achieve the purpose of controlling the generation effect of the target speech signal.

[0114] In an implementation manner, the Attention mask matrix is trained by the following manner:

[0115] obtain a training text sample and a training speech sample;

[0116] input the training text sample and the training speech sample into a VITS model for model training, and train an Attention mask matrix based on the training text sample and a preset target vocabulary set;

[0117] calculate the total loss value of the VITS model to determine whether the VITS model is sufficiently trained;

[0118] if the VITS model is sufficiently trained, it is determined that the trained Attention mask matrix is obtained.

[0119] In an implementation manner, the vocabulary in the target vocabulary set is any one of a noun, a verb, and an interjection.

[0120] In an implementation manner, the vocabulary in the target vocabulary set is a plurality of vocabularies of a noun, a verb, and an interjection, and each of the vocabularies carries a corresponding part-of-speech label.

[0121] In an implementation manner, the Attention mask matrix is trained by the following manner:

[0122] multiply the Attention value in the Attention mask matrix by the corresponding phoneme vector respectively to obtain a stationary point multiplication result;

[0123] splicing the stationary point multiplication result and the phoneme vector to obtain a masked phoneme vector.

[0124] In one embodiment, a computer readable storage medium is disclosed, when the instructions in the computer readable storage medium are executed by the processor in the computer device, the computer device can execute the steps of any embodiment of the speech synthesis method disclosed in the present application. The computer readable storage medium can be non-volatile or volatile.

[0125] In one embodiment, a computer readable storage medium is provided, and a computer program is stored on the computer readable storage medium. The computer program is executed by the processor to implement the following steps:

[0126] obtain a phoneme vector of a target text through a text encoder;

[0127] mask the at least part of the phoneme vectors of the target text based on the pre-trained Attention mask matrix to obtain masked phoneme vectors;

[0128] perform length prediction on the masked phoneme vectors based on the random length predictor to obtain predicted phoneme lengths;

[0129] perform inverse mapping on the masked phoneme vectors based on the predicted phoneme lengths by the inverse flow model to obtain hidden variables corresponding to each of the masked phoneme vectors;

[0130] decode the hidden variables by the pre-trained decoder to obtain a target speech signal.

[0131] In an implementation manner, the Attention mask matrix is obtained by the following manner:

[0132] obtain training text samples and training speech samples;

[0133] input the training text samples and the training speech samples into a VITS model for model training, and train an Attention mask matrix based on the training text samples and a preset target vocabulary set;

[0134] calculate a total loss value of the VITS model to determine whether the VITS model is sufficiently trained;

[0135] if the VITS model is sufficiently trained, determine that the trained Attention mask matrix is obtained.

[0136] In an implementation manner, the vocabulary in the target vocabulary set is any one of a noun, a verb, and an interjection.

[0137] In an implementation manner, the vocabulary in the target vocabulary set is a plurality of vocabularies of a noun, a verb, and an interjection, and each of the vocabularies carries a corresponding part-of-speech label.

[0138] In an implementation manner, the mask the phoneme vectors based on the pre-trained Attention mask matrix to obtain masked phoneme vectors includes:

[0139] multiply the Attention values in the Attention mask matrix with corresponding phoneme vectors respectively to obtain a stationary point multiplication result;

[0140] splice the stationary point multiplication result with the phoneme vectors to obtain masked phoneme vectors.

[0141] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments of each method. Any reference to memory, storage, database or other medium used in each embodiment provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0142] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0143] The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A speech synthesis method, characterized in that, The method includes: The phoneme vector of the target text is obtained through a text encoder; The target text's phoneme vectors are masked based on a pre-trained Attention mask matrix to obtain masked phoneme vectors. The predicted phoneme duration is obtained by using a random duration predictor based on the mask phoneme vector. By using the inverse flow model, the mask phoneme vector is inversely mapped based on the predicted phoneme duration to obtain the latent variable corresponding to each mask phoneme vector; The latent variables are decoded using a pre-trained decoder to obtain the target speech signal; The Attention mask matrix is ​​trained in the following way: Obtain training text samples and training speech samples; The training text samples and the training speech samples are input into the VITS model for model training, and the Attention mask matrix is ​​trained based on the training text samples and the preset target vocabulary set. Calculate the total loss value of the VITS model to determine whether the VITS model has been sufficiently trained; If the VITS model is sufficiently trained, the trained Attention mask matrix is ​​determined.

2. The speech synthesis method as described in claim 1, characterized in that, The target vocabulary set includes any one of the following: nouns, verbs, and interjections.

3. The speech synthesis method as described in claim 1, characterized in that, The target vocabulary set contains a variety of words including nouns, verbs, and interjections, and each word carries a corresponding part-of-speech tag.

4. The speech synthesis method as described in claim 1, characterized in that, The step of masking the phoneme vector using a random duration predictor based on a pre-trained attention mask matrix to obtain a masked phoneme vector includes: The Attention values ​​in the Attention mask matrix are multiplied at stationary points with the corresponding phoneme vectors to obtain the stationary multiplication results. The result of multiplying the stationary points is concatenated with the phoneme vector to obtain the mask phoneme vector.

5. A speech synthesis device, characterized in that, The device includes: The vector acquisition unit is used to acquire the phoneme vector of the target text through the text encoder; The masking unit is used to perform masking processing on at least a portion of the phoneme vectors of the target text based on a pre-trained Attention mask matrix to obtain a masked phoneme vector. The duration prediction unit is used to perform duration prediction based on the mask phoneme vector using a random duration predictor to obtain the predicted phoneme duration. The variable mapping unit is used to perform inverse mapping processing on the mask phoneme vector based on the predicted phoneme duration through the inverse flow model to obtain the latent variable corresponding to each mask phoneme vector; A speech synthesis unit is used to decode the latent variables through a pre-trained decoder to obtain the target speech signal; The Attention mask matrix is ​​trained in the following way: Obtain training text samples and training speech samples; The training text samples and the training speech samples are input into the VITS model for model training, and the Attention mask matrix is ​​trained based on the training text samples and the preset target vocabulary set. Calculate the total loss value of the VITS model to determine whether the VITS model has been sufficiently trained; If the VITS model is sufficiently trained, the trained Attention mask matrix is ​​determined.

6. The speech synthesis device as described in claim 5, characterized in that, The target vocabulary set includes any one of the following: nouns, verbs, and interjections.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the speech synthesis method as described in any one of claims 1 to 4.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the speech synthesis method as described in any one of claims 1 to 4.

Images