Speech synthesis method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of financial science and technology, and discloses a speech synthesis method and device, computer equipment and a storage medium. Comprising the steps of obtaining a phoneme vector of a target text; performing mask processing on the phoneme vector of the target text based on a pre-trained Attention mask matrix to obtain a mask phoneme vector; performing duration prediction based on the mask phoneme vector to obtain predicted phoneme duration; performing inverse mapping processing on the mask phoneme vectors based on the predicted phoneme duration through a countercurrent model to obtain a hidden variable corresponding to each mask phoneme vector; and decoding the hidden variable through a pre-trained decoder to obtain a target voice signal. According to the method, the phoneme vectors are masked through the Attention mask matrix to obtain the masked phoneme vectors, so that the random duration predictor is more likely to sample the phoneme vectors which are not masked in the masked phoneme vectors, and the purpose of controlling the sampling process is achieved.

Description

Technical Field

[0001] The present invention relates to the field of financial technology, and in particular to a speech synthesis method, apparatus, computer equipment and storage medium. Background Art

[0002] With the development of science and technology, intelligent speech synthesis technology (such as VITS technology) has been applied in multiple technical fields, such as financial technology, intelligent reading technology, and so on.

[0003] For example, in the financial field, when a user learns about life insurance through financial software installed on a mobile phone, the user can use the online reading function to identify the relevant text information about life insurance and synthesize speech for reading.

[0004] However, in practical applications, VITS technology generates multiple sentences of different speech by randomly sampling the fitted phoneme duration distribution. However, due to the randomness of sampling, it is difficult to control the sampling process, resulting in the inability to control the generated speech effect. Summary of the invention

[0005] The embodiments of the present invention provide a speech synthesis method, apparatus, computer equipment and storage medium to solve the problem that when a VITS model generates speech, the generated speech effect cannot be controlled due to the control of the sampling process.

[0006] A speech synthesis method, the method comprising: Obtain the phoneme vector of the target text through the text encoder; Masking at least part of the phoneme vector of the target text based on a pre-trained Attention mask matrix to obtain a masked phoneme vector; Performing duration prediction based on the masked phoneme vector by a random duration predictor to obtain a predicted phoneme duration; Performing inverse mapping processing on the masked phoneme vector based on the predicted phoneme duration through an inverse flow model to obtain a hidden variable corresponding to each masked phoneme vector; The latent variable is decoded by a pre-trained decoder to obtain a target speech signal.

[0007] In the above method, optionally, the Attention mask matrix is ​​trained in the following way: Obtain training text samples and training voice samples; Inputting the training text sample and the training speech sample into the VITS model for model training, and training the Attention mask matrix based on the training text sample and a preset target vocabulary set; Calculating the total loss value of the VITS model to determine whether the VITS model is fully trained; If the VITS model is fully trained, it is determined that the trained Attention mask matrix is ​​obtained.

[0008] In the above method, optionally, the vocabulary in the target vocabulary set is any one of nouns, verbs, and interjections.

[0009] In the above method, optionally, the words in the target vocabulary set are multiple words among nouns, verbs, and interjections, and each of the words carries a corresponding part-of-speech tag.

[0010] In the above method, optionally, the step of performing mask processing on the phoneme vector based on a pre-trained Attention mask matrix by using a random duration predictor to obtain a masked phoneme vector includes: Performing stationary point multiplication on the attention values ​​in the attention mask matrix and the corresponding phoneme vectors respectively to obtain stationary point multiplication results; The stationary point multiplication result is concatenated with the phoneme vector to obtain a masked phoneme vector.

[0011] A speech synthesis device, comprising: A vector acquisition unit, used to acquire the phoneme vector of the target text through a text encoder; A mask processing unit, configured to perform mask processing on at least part of the phoneme vectors of the target text based on a pre-trained Attention mask matrix to obtain a masked phoneme vector; A duration prediction unit, configured to perform duration prediction based on the masked phoneme vector by means of a random duration predictor to obtain a predicted phoneme duration; A variable mapping unit, configured to perform inverse mapping processing on the masked phoneme vector based on the predicted phoneme duration through an inverse flow model to obtain a hidden variable corresponding to each masked phoneme vector; The speech synthesis unit is used to decode the latent variable through a pre-trained decoder to obtain a target speech signal.

[0012] In the above device, optionally, the Attention mask matrix is ​​trained in the following manner: Obtain training text samples and training voice samples; Inputting the training text sample and the training speech sample into the VITS model for model training, and training the Attention mask matrix based on the training text sample and a preset target vocabulary set; Calculating the total loss value of the VITS model to determine whether the VITS model is fully trained; If the VITS model is fully trained, it is determined that the trained Attention mask matrix is ​​obtained.

[0013] In the above device, optionally, the vocabulary in the target vocabulary set is any one of nouns, verbs, and interjections.

[0014] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned speech synthesis method when executing the computer program.

[0015] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the speech synthesis method described above is implemented.

[0016] The above-mentioned speech synthesis method, device, computer equipment and storage medium mask at least part of the phoneme vectors of the target text through a pre-trained Attention mask matrix to obtain a masked phoneme vector, and then predict the duration of the masked phoneme vector through a random duration predictor to obtain a predicted phoneme duration, and obtain a target speech signal based on the predicted phoneme duration and the masked phoneme vector. It can be seen that the present invention masks at least part of the phoneme vectors of the target text through a pre-trained Attention mask matrix to obtain a masked phoneme vector, so that the random duration predictor is more inclined to sample the phoneme vectors that are not masked in the masked phoneme vector, and obtain the predicted phoneme duration, and finally obtain the target speech signal. Therefore, masking at least part of the phoneme vectors of the target text through the Attention mask matrix can realize the control of the sampling process of the VITS model, so as to achieve the purpose of controlling the generation effect of the target speech signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0018] Figure 1 It is a flowchart of a speech synthesis method disclosed in one embodiment of the present invention; Figure 2 It is a partial flow chart of a speech synthesis method disclosed in one embodiment of the present invention; Figure 3 It is a partial flow chart of a speech synthesis method disclosed in one embodiment of the present invention; Figure 4 is a structural schematic diagram of a speech synthesis device disclosed in an embodiment of the present invention; Figure 5 It is a structural schematic diagram of a computer device disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0021] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0022] As used in the present specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]", depending on the context.

[0023] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0024] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0025] The present invention discloses a speech synthesis method, device, computer equipment and storage medium, which performs masking processing on at least part of the phoneme vectors of the target text through a pre-trained Attention mask matrix to obtain a masked phoneme vector, and then performs duration prediction on the masked phoneme vector through a random duration predictor to obtain a predicted phoneme duration, and obtains a target speech signal based on the predicted phoneme duration and the masked phoneme vector. It can be seen that the present invention performs masking processing on at least part of the phoneme vectors of the target text through a pre-trained Attention mask matrix to obtain a masked phoneme vector, so that the random duration predictor is more inclined to sample the phoneme vectors that are not masked in the masked phoneme vector, and obtain the predicted phoneme duration, and finally obtain the target speech signal. Therefore, by performing masking processing on at least part of the phoneme vectors of the target text through the Attention mask matrix, the sampling process of the VITS model can be controlled to achieve the purpose of controlling the generation effect of the target speech signal. The following is an explanation through a specific embodiment.

[0026] like Figure 1 FIG. 1 is a flowchart of a speech synthesis method disclosed in an embodiment of the present invention. The method is applicable to electronic devices with natural language processing capabilities, such as mobile phones, tablet computers, personal computers, servers, and other devices. The method in this embodiment may include the following steps: S101: Obtain the phoneme vector of the target text through the text encoder.

[0027] The target text is obtained, converted into target text phonemes, and then the target text phonemes are encoded through a text encoder to obtain a phoneme vector.

[0028] In a specific implementation, the text encoder in this embodiment may be an encoder of a transformer model, which extracts target text phonemes from the target text, inputs the target text phonemes into the text encoder, and obtains the phoneme vector of each target text phoneme.

[0029] S102: Mask at least part of the phoneme vector of the target text based on the pre-trained Attention mask matrix to obtain a masked phoneme vector.

[0030] Through pre-training, the Attention value in the Attention mask matrix is ​​obtained, and the phoneme vector of the target text is masked according to the Attention value to obtain the masked phoneme vector.

[0031] In a specific implementation, the Attention values ​​in the Attention mask matrix will be different according to the different training samples used for the pre-training of the Attention mask matrix. Then, according to the different Attention values ​​in the Attention mask matrix, the phoneme vectors of the target text are subjected to different masking processes to obtain different masked phoneme vectors. In other words, by controlling the training samples used for the pre-training of the Attention mask matrix, the purpose of controlling the Attention value in the Attentionmask matrix can be achieved, and then the phoneme vectors of the target text are subjected to corresponding masking processes to obtain the desired masked phoneme vectors. Among them, the masked phoneme vector includes a masked phoneme vector and an unmasked phoneme vector, that is, the masked phoneme vector refers to a collection of masked phoneme vectors and unmasked phoneme vectors.

[0032] S103: Perform duration prediction based on the masked phoneme vector using a random duration predictor to obtain a predicted phoneme duration.

[0033] The masked phoneme vector is input into a stochastic duration predictor to perform time prediction based on the masked phoneme vector to obtain a predicted phoneme duration.

[0034] In the specific implementation, the random duration predictor is a flow-based generation model. After the masked phoneme vector is input into the random duration predictor, the random duration predictor is more inclined to sample the unmasked phoneme vectors in the masked phoneme vector, which ultimately makes the difference in the text part of the synthesized speech signal corresponding to the unmasked phoneme vector more obvious, thereby achieving the purpose of controllable random sampling process of the random duration predictor.

[0035] S104: Perform inverse mapping processing on the masked phoneme vector based on the predicted phoneme duration through the inverse flow model to obtain a hidden variable corresponding to each masked phoneme vector.

[0036] In a specific implementation, the countercurrent model in this embodiment can be obtained based on flow model training. The countercurrent model pre-learns the correspondence between latent variables and phoneme vectors, and then obtains the latent variables corresponding to each masked phoneme vector based on the mapping relationship between the masked phoneme vector and the latent variables.

[0037] S105: Decode the latent variable through the pre-trained decoder to obtain the target speech signal.

[0038] In a specific implementation, the decoder in this embodiment can be a vocoder, and the latent variable is input into the vocoder to obtain the target speech signal. The latent variable is decoded by the vocoder to obtain the target speech signal. Accordingly, the purpose of generating the target speech signal can be achieved based on the target text. Among them, the decoder is a pre-trained vocoder, and the decoder has pre-learned the conversion relationship between the latent variable and the speech signal.

[0039] In summary, the present invention discloses a speech synthesis method, which performs masking processing on at least part of the phoneme vectors of the target text through a pre-trained Attention mask matrix to obtain a masked phoneme vector, and then performs duration prediction on the masked phoneme vector through a random duration predictor to obtain a predicted phoneme duration, and obtains a target speech signal based on the predicted phoneme duration and the masked phoneme vector. It can be seen that the present invention performs masking processing on at least part of the phoneme vectors of the target text through a pre-trained Attention mask matrix to obtain a masked phoneme vector, so that the random duration predictor is more inclined to sample the phoneme vectors that are not masked in the masked phoneme vector, and obtain the predicted phoneme duration, and finally obtain the target speech signal. Therefore, by performing masking processing on at least part of the phoneme vectors of the target text through the Attention mask matrix, the sampling process of the VITS model can be controlled to achieve the purpose of controlling the generation effect of the target speech signal.

[0040] based on Figure 1 In the specific implementation, the Attention mask matrix can be trained in the following ways, such as Figure 2 As shown: Specifically, the method in this embodiment is implemented based on the VITS model. When the VITS model is trained, the Attention mask matrix is ​​trained synchronously to obtain the Attention value in the Attention mask matrix. When the training of the VITS model is completed, the trained Attention mask matrix is ​​obtained. The training process is specifically as follows: S201: Acquire training text samples and training speech samples.

[0041] In a specific implementation, when the VITS model in this embodiment is being trained, it is necessary to input training text samples and training voice samples at the same time. The training text samples and the training voice samples correspond to each other. For example, if the text content of the training text sample is "I love China" in text form, then the voice content of the training language sample is "I love China" in voice form. Based on this, after the training text sample and the training voice sample are input into the VITS model, the VITS model learns the corresponding relationship between the training text phonemes of the training text sample and the voice signals of the training voice sample, thereby realizing the generation of the target voice signal according to the target text.

[0042] S202: Input the training text samples and the training speech samples into the VITS model for model training, and train the Attention mask matrix based on the training text samples and the preset target vocabulary set.

[0043] After the training text samples and training speech samples are input into the VITS model, the training text samples are converted into training text phonemes, and the training speech samples are converted into training latent variables, and then the model is trained based on the training text phonemes and training latent variables. When training the VITS model, the Attentionmask matrix is ​​trained based on the training text phonemes and the preset target vocabulary set, and the Attention values ​​in the Attentionmask matrix are obtained until the VITS model training is completed and the required Attentionmask matrix is ​​determined to be obtained.

[0044] S203: Calculate the total loss value of the VITS model to determine whether the VITS model is fully trained.

[0045] If the VITS model is fully trained, step S204 is executed to determine whether a trained Attentionmask matrix is ​​obtained. If the VITS model is not fully trained, iterative training is continued based on training text samples and training speech samples until the VITS model is fully trained.

[0046] In a specific implementation, the total loss value of the VITS model includes the reconstruction loss value, the adversarial loss value, the feature discriminator loss value, etc., and the total loss value of the VITS model is calculated based on these losses. During the VITS model training process, when the calculated total loss value no longer decreases, it is determined whether the VITS model is fully trained, and step S204 is executed. When the calculated total loss value is still decreasing, iterative training is continued based on the training text samples and the training speech samples. During the iterative training process, the model parameters of the VITS model are continuously updated, and the corresponding total loss value is calculated until the total loss value no longer decreases, and step S204 is executed.

[0047] S204: Determine that a trained Attention mask matrix is ​​obtained.

[0048] After obtaining the trained Attention mask matrix, the target speech signal corresponding to the target text can be generated according to the Attention mask, and then speech playback can be performed based on the target speech signal.

[0049] In summary, in this embodiment, the Attention mask matrix is ​​trained based on the VITS model training process, which can control the speech effect generated by the VITS model without significantly increasing the training cost of the VITS model.

[0050] based on Figure 2 In a specific implementation, the vocabulary in the target vocabulary set in this embodiment is any one of nouns, verbs, and interjections.

[0051] In a specific implementation, the parts of speech of the words in the target vocabulary set in this embodiment can be controlled as needed. For example, all the words in the target vocabulary set can be nouns, or all the words in the target vocabulary set can be interjections, etc. According to the phonemes of the words of a single part of speech in the target vocabulary set and the training text phonemes of the training text samples, the Attention mask matrix is ​​obtained through attention mechanism training.

[0052] According to the generated Attention mask matrix, the phoneme vectors of the target text other than the part of speech can be masked, so that the random duration predictor can sample the phoneme vector of the part of speech more aggressively, thereby controlling the sampling process of the random duration predictor.

[0053] based on Figure 2 In a specific implementation, the words in the target vocabulary set in this embodiment are any one or more words among nouns, verbs, and interjections, and each word carries a corresponding part-of-speech tag.

[0054] The target vocabulary set may include words of multiple parts of speech. To ensure the training effect of the Attention mask matrix, words of different parts of speech need to be marked with magnetic labels. The Attention mask matrix obtained by training can make the random duration predictor tend to sample the phoneme vectors of multiple parts of speech of the target text, thereby realizing the control of the sampling process of the random duration predictor.

[0055] based on Figure 1 In the specific implementation, step S103 in this embodiment can be implemented by the following steps: Figure 3 As shown: S301: Perform stationary point multiplication on the attention values ​​in the attention mask matrix and the corresponding phoneme vectors to obtain stationary point multiplication results.

[0056] S302: Concatenate the stationary point multiplication result with the phoneme vector to obtain a masked phoneme vector.

[0057] Among them, the attention value in the attention mask matrix and the phoneme vector of the target text both conform to the encoder-decoder structure, and then stationary multiplication is performed according to the matrix structure, that is, each phoneme vector of the target text is multiplied by the attention value in the attention mask matrix, and then the result of the stationary multiplication is concatenated with the original phoneme vector, thereby realizing the mask processing of the phoneme vector and obtaining the masked phoneme vector.

[0058] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0059] like Figure 4 As shown, it is a structural schematic diagram of a speech synthesis device disclosed in an embodiment of the present invention. The device is suitable for electronic devices with natural language processing capabilities, such as mobile phones, tablet computers, personal computers, servers and other devices.

[0060] Specifically, the speech synthesis device in this embodiment may include the following units: A vector acquisition unit 401 is used to acquire a phoneme vector of a target text through a text encoder; A mask processing unit 402 is used to perform mask processing on at least part of the phoneme vector of the target text based on a pre-trained Attention mask matrix to obtain a masked phoneme vector; The duration prediction unit 403 is used to perform duration prediction based on the masked phoneme vector by means of a random duration predictor to obtain a predicted phoneme duration; A variable mapping unit 404 is used to perform inverse mapping processing on the masked phoneme vector based on the predicted phoneme duration through an inverse flow model to obtain a hidden variable corresponding to each masked phoneme vector; The speech synthesis unit 405 is used to decode the latent variable through a pre-trained decoder to obtain a target speech signal.

[0061] In summary, the present invention discloses a speech synthesis device, which masks at least part of the phoneme vectors of the target text through a pre-trained Attention mask matrix to obtain a masked phoneme vector, and then predicts the duration of the masked phoneme vector through a random duration predictor to obtain a predicted phoneme duration, and obtains a target speech signal based on the predicted phoneme duration and the masked phoneme vector. It can be seen that the present invention masks at least part of the phoneme vectors of the target text through a pre-trained Attention mask matrix to obtain a masked phoneme vector, so that the random duration predictor is more inclined to sample the phoneme vectors that are not masked in the masked phoneme vector, and obtain the predicted phoneme duration, and finally obtain the target speech signal. Therefore, by masking at least part of the phoneme vectors of the target text through the Attention mask matrix, the sampling process of the VITS model can be controlled to achieve the purpose of controlling the generation effect of the target speech signal.

[0062] In one implementation, the Attention mask matrix is ​​trained in the following manner: Obtain training text samples and training voice samples; Inputting the training text sample and the training speech sample into the VITS model for model training, and training the Attention mask matrix based on the training text sample and a preset target vocabulary set; Calculating the total loss value of the VITS model to determine whether the VITS model is fully trained; If the VITS model is fully trained, it is determined that the trained Attention mask matrix is ​​obtained.

[0063] In one implementation, the vocabulary in the target vocabulary set is any one of nouns, verbs, and interjections.

[0064] In one implementation, the words in the target vocabulary set are multiple words among nouns, verbs, and interjections, and each of the words carries a corresponding part-of-speech tag.

[0065] In one implementation, the mask processing unit 402 may be configured to: Performing stationary point multiplication on the attention values ​​in the attention mask matrix and the corresponding phoneme vectors respectively to obtain stationary point multiplication results; The stationary point multiplication result is concatenated with the phoneme vector to obtain a masked phoneme vector.

[0066] For the specific definition of the speech synthesis device, please refer to the relevant definition of the speech synthesis method above, which will not be repeated here. Each module in the above-mentioned speech synthesis device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0067] In one embodiment, a computer device is disclosed. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a speech synthesis method is implemented.

[0068] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: Obtain the phoneme vector of the target text through the text encoder; Masking at least part of the phoneme vector of the target text based on a pre-trained Attention mask matrix to obtain a masked phoneme vector; Performing duration prediction based on the masked phoneme vector by a random duration predictor to obtain a predicted phoneme duration; Performing inverse mapping processing on the masked phoneme vector based on the predicted phoneme duration through an inverse flow model to obtain a hidden variable corresponding to each masked phoneme vector; The latent variable is decoded by a pre-trained decoder to obtain a target speech signal.

[0069] In summary, the present invention discloses a computer device, which masks at least part of the phoneme vectors of the target text through a pre-trained Attention mask matrix to obtain a masked phoneme vector, and then predicts the duration of the masked phoneme vector through a random duration predictor to obtain a predicted phoneme duration, and obtains a target speech signal based on the predicted phoneme duration and the masked phoneme vector. It can be seen that the present invention masks at least part of the phoneme vectors of the target text through a pre-trained Attention mask matrix to obtain a masked phoneme vector, so that the random duration predictor is more inclined to sample the phoneme vectors that are not masked in the masked phoneme vector, and obtain the predicted phoneme duration, and finally obtain the target speech signal. Therefore, by masking at least part of the phoneme vectors of the target text through the Attention mask matrix, the sampling process of the VITS model can be controlled to achieve the purpose of controlling the generation effect of the target speech signal.

[0070] In one implementation, the Attention mask matrix is ​​trained in the following manner: Obtain training text samples and training voice samples; Inputting the training text sample and the training speech sample into the VITS model for model training, and training the Attention mask matrix based on the training text sample and a preset target vocabulary set; Calculating the total loss value of the VITS model to determine whether the VITS model is fully trained; If the VITS model is fully trained, it is determined that the trained Attention mask matrix is ​​obtained.

[0071] In one implementation, the vocabulary in the target vocabulary set is any one of nouns, verbs, and interjections.

[0072] In one implementation, the words in the target vocabulary set are multiple words among nouns, verbs, and interjections, and each of the words carries a corresponding part-of-speech tag.

[0073] In one implementation, the step of performing mask processing on the phoneme vector based on a pre-trained Attention mask matrix by using a random duration predictor to obtain a masked phoneme vector includes: Performing stationary point multiplication on the attention values ​​in the attention mask matrix and the corresponding phoneme vectors respectively to obtain stationary point multiplication results; The stationary point multiplication result is concatenated with the phoneme vector to obtain a masked phoneme vector.

[0074] In one embodiment, a computer-readable storage medium is disclosed. When instructions in the computer-readable storage medium are executed by a processor in a computer device, the computer device is enabled to perform the steps of any embodiment of a speech synthesis method disclosed in the present invention. The computer-readable storage medium may be non-volatile or volatile.

[0075] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: Obtain the phoneme vector of the target text through the text encoder; Masking at least part of the phoneme vector of the target text based on a pre-trained Attention mask matrix to obtain a masked phoneme vector; Performing duration prediction based on the masked phoneme vector by a random duration predictor to obtain a predicted phoneme duration; Performing inverse mapping processing on the masked phoneme vector based on the predicted phoneme duration through an inverse flow model to obtain a hidden variable corresponding to each masked phoneme vector; The latent variable is decoded by a pre-trained decoder to obtain a target speech signal.

[0076] In one implementation, the Attention mask matrix is ​​trained in the following manner: Obtain training text samples and training voice samples; Inputting the training text sample and the training speech sample into the VITS model for model training, and training the Attention mask matrix based on the training text sample and a preset target vocabulary set; Calculating the total loss value of the VITS model to determine whether the VITS model is fully trained; If the VITS model is fully trained, it is determined that the trained Attention mask matrix is ​​obtained.

[0077] In one implementation, the vocabulary in the target vocabulary set is any one of nouns, verbs, and interjections.

[0078] In one implementation, the words in the target vocabulary set are multiple words among nouns, verbs, and interjections, and each of the words carries a corresponding part-of-speech tag.

[0079] In one implementation, the step of performing mask processing on the phoneme vector based on a pre-trained Attention mask matrix by using a random duration predictor to obtain a masked phoneme vector includes: Performing stationary point multiplication on the attention values ​​in the attention mask matrix and the corresponding phoneme vectors respectively to obtain stationary point multiplication results; The stationary point multiplication result is concatenated with the phoneme vector to obtain a masked phoneme vector.

[0080] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0081] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0082] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A speech synthesis method, characterized in that: The method comprises: Obtain the phoneme vector of the target text through the text encoder; Masking at least part of the phoneme vector of the target text based on a pre-trained Attention mask matrix to obtain a masked phoneme vector; Performing duration prediction based on the masked phoneme vector by a random duration predictor to obtain a predicted phoneme duration; Performing inverse mapping processing on the masked phoneme vector based on the predicted phoneme duration through an inverse flow model to obtain a hidden variable corresponding to each masked phoneme vector; The latent variable is decoded by a pre-trained decoder to obtain a target speech signal.

2. The speech synthesis method according to claim 1, wherein: The Attention mask matrix is ​​trained in the following way: Obtain training text samples and training voice samples; Inputting the training text sample and the training speech sample into the VITS model for model training, and training the Attention mask matrix based on the training text sample and a preset target vocabulary set; Calculating the total loss value of the VITS model to determine whether the VITS model is fully trained; If the VITS model is fully trained, it is determined that the trained Attention mask matrix is ​​obtained.

3. The speech synthesis method according to claim 2, characterized in that: The words in the target vocabulary set are any of nouns, verbs, and interjections.

4. The speech synthesis method according to claim 2, characterized in that: The words in the target vocabulary set are a variety of words among nouns, verbs, and interjections, and each of the words carries a corresponding part-of-speech tag.

5. The speech synthesis method according to claim 1, wherein: The method of performing mask processing on the phoneme vector based on the pre-trained Attention mask matrix by using the random duration predictor to obtain the masked phoneme vector includes: Performing stationary point multiplication on the attention values ​​in the attention mask matrix and the corresponding phoneme vectors respectively to obtain stationary point multiplication results; The stationary point multiplication result is concatenated with the phoneme vector to obtain a masked phoneme vector.

6. A speech synthesis device, characterized in that: The device comprises: A vector acquisition unit, used to acquire the phoneme vector of the target text through a text encoder; A mask processing unit, configured to perform mask processing on at least part of the phoneme vectors of the target text based on a pre-trained Attention mask matrix to obtain a masked phoneme vector; A duration prediction unit, configured to perform duration prediction based on the masked phoneme vector by means of a random duration predictor to obtain a predicted phoneme duration; A variable mapping unit, configured to perform inverse mapping processing on the masked phoneme vector based on the predicted phoneme duration through an inverse flow model to obtain a hidden variable corresponding to each masked phoneme vector; The speech synthesis unit is used to decode the latent variable through a pre-trained decoder to obtain a target speech signal.

7. The speech synthesis device according to claim 6, wherein: The Attention mask matrix is ​​trained in the following way: Obtain training text samples and training voice samples; Inputting the training text sample and the training speech sample into the VITS model for model training, and training the Attention mask matrix based on the training text sample and a preset target vocabulary set; Calculating the total loss value of the VITS model to determine whether the VITS model is fully trained; If the VITS model is fully trained, it is determined that the trained Attention mask matrix is ​​obtained.

8. The speech synthesis device according to claim 6, characterized in that: The words in the target vocabulary set are any of nouns, verbs, and interjections.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the speech synthesis method according to any one of claims 1 to 5 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the speech synthesis method according to any one of claims 1 to 5 is implemented.

Images