A method, device and electronic device for N-ary non-autoregressive speech synthesis

Abstract

The present invention belongs to the technical field of data information processing, and provides a method, device, electronic device and recording medium based on N-ary non-autoregressive speech synthesis method, the method comprising: obtaining text content to be synthesized, and performing standardization processing on the text content to be synthesized; inputting the standardized text content to be synthesized into a speech model based on N-ary non-autoregressive, and outputting a refined Mel spectrum, where N is a natural integer; and converting the refined Mel spectrum into a speech synthesis file. The present invention reduces the prediction task of a conventional non-autoregressive network to 1 / n of the original, greatly reducing the working requirements of the Mel feature decoder, and also improving the robustness of the model.

Description

Technical Field

[0001] The present invention belongs to the technical field of data information processing, and is particularly suitable for data information processing in online education services. More specifically, it relates to a method, device and electronic equipment based on N-ary non-autoregressive speech synthesis. Background Art

[0002] Speech synthesis is a technology that converts text information generated by a computer or input from an external source into human-understandable speech and outputs it. In recent years, speech synthesis technology has developed rapidly. Speech synthesis technology based on neural networks makes the synthesized audio very close to real human voices. There are two main types of speech synthesis solutions based on neural networks: one is based on an autoregressive model, and the other is based on a non-autoregressive model.

[0003] At present, the solution based on the autoregressive model generates speech audio based on the generated speech, which can effectively learn the temporal changes of acoustic features and generate high-quality speech. However, due to the nature of speech sequence generation, the speech inference speed is slow during the generation process, which makes the speech synthesis speed slow. Compared with the former, the solution based on the non-autoregressive model has a faster inference speed and generates more robust speech, which can effectively avoid the problems of missing and repeated synthesis of some texts to be synthesized.

[0004] However, the current solutions based on non-autoregressive models have high requirements on the complexity and fitting ability of the Mel spectrum decoder. When the Mel spectrum decoder is not fully trained or its ability is not strong enough, the overall audio synthesis quality will be greatly reduced or even errors will occur. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] The present invention at least aims to solve the problem that existing solutions based on non-autoregressive models have high requirements on the complexity and fitting ability of the Mel spectrum decoder.

[0007] (II) Technical solution

[0008] In order to solve the above technical problems, one aspect of the present invention proposes a method for speech synthesis based on N-ary non-autoregressive speech, where N is the mel spectrum granularity, and the method comprises:

[0009] Acquire text content to be synthesized, and perform standardization processing on the text content to be synthesized;

[0010] Inputting the text content to be synthesized after the standardization processing into a speech model based on N-ary non-autoregression, and outputting a refined Mel spectrum, wherein N is a natural integer;

[0011] The refined Mel-spectrogram is converted into a speech synthesis file.

[0012] In an exemplary embodiment of the present invention, the standardization process of the text content to be synthesized further includes:

[0013] Cutting the text content to be synthesized into short sentences;

[0014] Performing text regularization processing on the short sentence;

[0015] The short sentence after the text regularization processing is processed by decomposing characters, and a pronunciation sequence consisting of the smallest pronunciation units corresponding to the short sentence is output.

[0016] In an exemplary embodiment of the present invention, the N-gram non-autoregressive based speech model includes an N-gram coarse Mel feature spectrum prediction model and an N-gram fine Mel feature spectrum prediction model.

[0017] In an exemplary embodiment of the present invention, the N-ary coarse Mel-characteristic spectrum prediction model receives the pronunciation sequence and outputs a 1 / N Mel coarse spectrum.

[0018] In an exemplary embodiment of the present invention, the N-ary refined Mel-spectrum feature spectrum prediction model receives the 1 / N Mel-spectrum coarse spectrum and outputs a refined Mel-spectrum.

[0019] In an exemplary embodiment of the present invention, converting the refined Mel spectrum into a speech synthesis file specifically includes:

[0020] Receiving the refined Mel spectrum;

[0021] Select the vocoder model to output the speech synthesis file according to business requirements.

[0022] In an exemplary embodiment of the present invention, if the service requirement is non-real-time synthesis, the WaveRNN vocoder model is selected;

[0023] If the business requirement is real-time synthesis, select the MultiBand_Melgan vocoder model.

[0024] The second aspect of the present invention provides an N-ary non-autoregressive speech synthesis device, where N is the Mel spectrum granularity, and the device includes:

[0025] A text processing module, used for obtaining text content to be synthesized and performing standardization processing on the text content to be synthesized;

[0026] A mel spectrum processing module, used for inputting the text content to be synthesized after the standardization processing into a speech model based on N-ary non-autoregression, and outputting a refined mel spectrum, wherein N is a natural integer;

[0027] The speech synthesis module is used to convert the refined Mel spectrum into a speech synthesis file.

[0028] A third aspect of the present invention provides an electronic device, comprising:

[0029] one or more processors;

[0030] A storage device for storing one or more programs;

[0031] When the one or more programs are executed by the one or more processors, the one or more processors implement the described method.

[0032] A fourth aspect of the present invention provides a computer-readable medium having a computer program stored thereon, wherein the program implements the method described when executed by a processor.

[0033] (III) Beneficial effects

[0034] The present invention modifies the conventional non-autoregressive model prediction task to predict only the 1 / n part of the Mel feature spectrogram corresponding to each text symbol, and then inputs the "1 / n rough spectrum" into a spectrogram restoration network to restore the 1 / n feature spectrogram to n, that is, the conventional non-autoregressive network is used to predict the "1 / n rough spectrum", and then a restoration network is used to restore the Mel feature spectrogram to a normal spectrogram. Since the prediction task of the conventional non-autoregressive network is reduced to the original 1 / n, the working requirements of the Mel feature decoder are greatly reduced, and the robustness of the model is also improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flowchart of a method for synthesizing speech based on N-gram non-autoregressive speech in one embodiment of the present invention;

[0036] Figure 2 is a schematic diagram of a process for standardizing text content in one embodiment of the present invention;

[0037] Figure 3 1 is a schematic diagram of a processing flow of an N-ary coarse Mel-signature spectrum prediction model in one embodiment of the present invention;

[0038] Figure 4 It is a schematic diagram of the structure of a non-autoregressive speech synthesis device based on N-ary in one embodiment of the present invention;

[0039] Figure 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention;

[0040] Figure 6 is a schematic diagram of a computer-readable recording medium according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In the process of introducing specific embodiments, the detailed description of the structure, performance, effect or other characteristics is to enable those skilled in the art to fully understand the embodiments. However, it does not exclude that those skilled in the art can implement the present invention with a technical solution that does not contain the above-mentioned structure, performance, effect or other characteristics under certain circumstances.

[0042] The flowchart in the accompanying drawings is only an exemplary process demonstration, and does not mean that the solution of the present invention must include all the contents, operations and steps in the flowchart, nor does it mean that it must be executed in the order shown in the figure. For example, some operations / steps in the flowchart can be decomposed, some operations / steps can be combined or partially combined, etc. The execution order shown in the flowchart can be changed according to actual conditions without departing from the main purpose of the present invention.

[0043] Frames in the attached figure Figure 1 Generally, it refers to functional entities, which do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processing unit devices and / or microcontroller devices.

[0044] The same reference numerals in the drawings represent the same or similar elements, components or parts, and thus the repeated description of the same or similar elements, components or parts may be omitted below. It should also be understood that although the first, second, third and other attributives representing the numbers may be used herein to describe various devices, elements, components or parts, these devices, elements, components or parts should not be limited by these attributives. In other words, these attributives are only used to distinguish one from another. For example, the first device may also be called the second device, but it does not deviate from the essential technical solution of the present invention. In addition, the terms "and / or" and "and / or" refer to all combinations including any one or more of the listed items.

[0045] In order to solve the above technical problems, the present invention proposes a method and device for speech synthesis based on N-ary non-autoregressive, which modifies the conventional non-autoregressive model prediction task to only predict the 1 / n part of the Mel feature spectrum corresponding to each text symbol, and then inputs the "1 / n rough spectrum" into a spectrogram restoration network to restore the 1 / n feature spectrum to n, which greatly reduces the working requirements of the Mel feature decoder and improves the robustness of the model.

[0046] Figure 1 It is a flowchart of a method based on N-ary non-autoregressive speech synthesis, such as Figure 1 As shown, the method includes:

[0047] S101: Acquire text content to be synthesized, and perform standardization processing on the text content to be synthesized.

[0048] In this embodiment, since the input text content contains a lot of non-standard text content, such as punctuation, numbers, symbols and other words or words, the N-gram non-autoregressive speech model cannot convert these non-standard text information into mel-spectrograms, so the text content needs to be standardized.

[0049] On the basis of the above technical solution, further, the text content to be synthesized is subjected to standardization processing. Figure 2 As shown, specifically including:

[0050] S1011, dividing the text content to be synthesized into short sentences.

[0051] In this embodiment, the input text content is usually a whole paragraph, including multiple long sentences or short sentences. In order to facilitate the model to process, the text content needs to be segmented, and the paragraph is converted into a long sentence, and the long sentence is converted into a short sentence. In this embodiment, the sentence is segmented according to punctuation marks, such as commas, semicolons, periods, exclamation marks, etc.

[0052] S1012: Perform text regularization processing on the short sentence.

[0053] The purpose of text regularization is to convert text into a more convenient and standard form of expression. The texts used in daily life contain a large number of non-standard words, which cannot be found in dictionaries and their pronunciation cannot be obtained through normal phonetic rules.

[0054] In this embodiment, this application is mainly aimed at Chinese text. In Chinese text, non-standard words refer to words containing non-Chinese characters, such as Arabic numerals, English characters, etc. This conversion process is text regularization. Text regularization is a key link in speech synthesis, and non-standard words are often the focus of user attention, such as price, date, time, etc. Text regularization directly affects the quality of speech synthesis. Non-standard words correspond to different standard words in different application scenarios. For example, 2021 is read as "two thousand and twenty-one" by number and "two thousand and twenty-one" by year.

[0055] S1013, performing character decomposition processing on the short sentence after the text regularization processing, and outputting a pronunciation sequence composed of the smallest pronunciation units corresponding to the short sentence.

[0056] In this embodiment, the regularized sentence segments need to be split into words. The split words are predicted to determine the part of speech of each word, which includes noun, verb, adjective, etc. Accurately determining the part of speech of a word can improve the accuracy of the final pronunciation sequence.

[0057] After the part-of-speech prediction, it is necessary to correct the pronunciation of polyphonic characters in the words. Here, the part-of-speech predicted previously is required to determine the pronunciation of the polyphonic characters. For example, in the sentence "I understand", "understand" is a verb, so the first "le" is pronounced as "liǎo", and the second "le" is a particle, so it is pronounced as "le".

[0058] After the pronunciation correction of polyphonic characters, the smallest pronunciation unit corresponding to each character in the short sentence can be determined. In this embodiment, the text is in Chinese, so the smallest pronunciation unit is the initial and final consonants. In other embodiments, if the text is in English, the smallest pronunciation unit is the phoneme. The smallest pronunciation units corresponding to all characters in the short sentence constitute the pronunciation sequence.

[0059] S102. Input the text content of the to-be-synthesized text after the normalization process into a speech model based on N-gram non-autoregressive, and output a refined Mel spectrogram, where N is a natural integer.

[0060] On the basis of the above technical solution, further, the speech model based on N-gram non-autoregressive includes an N-gram rough Mel feature spectrogram prediction model and an N-gram fine Mel feature spectrogram prediction model.

[0061] In this embodiment, the speech model based on N-gram non-autoregressive includes an N-gram rough Mel feature spectrogram prediction model and an N-gram fine Mel feature spectrogram prediction model.

[0062] On the basis of the above technical solution, further, the N-gram rough Mel feature spectrogram prediction model receives the pronunciation sequence and outputs a rough Mel spectrogram.

[0063] In this embodiment, the N-gram rough Mel feature spectrogram prediction model includes a text encoding feature module, a frame number prediction module, a text encoding feature extension module, and a Mel spectrogram decoding module. The process is as Figure 3 shown, and the steps are as follows:

[0064] S1021. The text encoding feature module receives the pronunciation sequence and outputs the text encoding feature corresponding to the pronunciation sequence.

[0065] S1022. The frame number prediction module predicts the number of frames corresponding to the continuous pronunciation duration of each pronunciation unit in the pronunciation sequence according to the text encoding feature. This number of frames is the number of frames of the Mel feature spectrogram corresponding to the pronunciation unit in the finally synthesized audio

[0066] S1023. The text encoding feature extension module extends the text encoding feature according to the number of frames, and finally inputs the extended text encoding feature into the Mel spectrogram decoding module to output a rough Mel spectrogram.

[0067] On the basis of the above technical solution, further, the N-ary fine Mel feature spectrum prediction model receives the Mel rough spectrum, output refined Mel spectrum.

[0068] In this embodiment, the N-ary fine Mel feature spectrum prediction model is mainly responsible for converting the received The Mel rough spectrum is refined. The N-ary refined Mel feature spectrum prediction model uses the CNN network to upsample the Mel rough spectrum into a refined Mel spectrum. That is, since the number of frames corresponding to each pronunciation unit in the Mel rough spectrum is only a fraction of the actual number of frames in the refined Mel spectrum, After the N-ary fine Mel feature spectrum prediction model is used, in addition to being able to correspond each pronunciation unit to The spectrogram data of the frame number is expanded to the spectrogram data of the real frame number, and the rough Mel spectrum can be further refined to finally generate an effective refined Mel spectrum.

[0069] S103: Convert the refined Mel-spectrogram into a speech synthesis file.

[0070] In this embodiment, the refined Mel-spectrogram is synthesized into a final speech synthesis file through a vocoder model.

[0071] On the basis of the above technical solution, further, converting the refined Mel spectrum into a speech synthesis file specifically includes:

[0072] Receiving the refined Mel spectrum;

[0073] Select the vocoder model to output the speech synthesis file according to business requirements.

[0074] On the basis of the above technical solution, further, if the business requirement is non-real-time synthesis, the WaveRNN vocoder model is selected;

[0075] If the business requirement is real-time synthesis, select the MultiBand_Melgan vocoder model.

[0076] In this embodiment, the corresponding voice coder model is selected according to the requirements of the final business. For example, real-time synthesis is selected in the scenario where the business is to interact with the user. The voice is synthesized while communicating with the user, reducing the user's waiting time. When the scenario is to read aloud large sections of text, the timeliness requirement is lower, and non-real-time synthesis can be selected.

[0077] The N-gram non-autoregressive speech synthesis provided by the present invention is explained below through Example 1.

[0078] Embodiment 1:

[0079] S201. Input the text "ZYB".

[0080] S202. Standardize “ZYB”.

[0081] Since this is an example, it is assumed that the input text content "ZYB" remains unchanged before and after normalization, so the pronunciation sequence after normalization is "ZYB".

[0082] S203: The text feature encoding module receives the pronunciation sequence and outputs the text encoding feature "Z', Y', B'".

[0083] S204: The frame number prediction module predicts the frame number value of the pronunciation sequence.

[0084] In this embodiment, it is assumed that the precise frame numbers (pfn, precise frame number) of the spectrograms corresponding to the Mel-signal spectrum of the final synthesized audio of the input text "ZYB" are: pfn1=2, pfn2=4, pfn3=4.

[0085] Assuming that N is 2, when the frame number prediction module predicts, the predicted frame numbers (rfn, root frame number) corresponding to the ZYB pronunciation sequences are:

[0086]

[0087]

[0088]

[0089] S204. The text coding feature expansion module expands the text coding feature using the predicted frame number results 1, 2, 2, and the expansion result is "Z', Y', Y', B', B'".

[0090] S205, the expanded result is input into the Mel spectrum decoding module, and the output Rough Mel spectrum (r_mel, roughmel), "Z_r_mel_1, Y_r_mel_1, Y_r_mel_2, B_r_mel_1, B_r_mel_2".

[0091] S206. The N-ary precise Mel feature spectrum prediction model outputs a refined Mel spectrum (p_mel, precisemel). Since N is 2, the refined Mel spectrum is “Z_p_mel_1, Z_p_mel_2, Y_p_mel_1, Y_p_mel_2, Y_p_mel_3, Y_p_mel_4, B_p_mel_1, B_p_mel_2, B_p_mel_3, B_p_mel_4”.

[0092] S207. Convert the refined Mel-spectrogram into an audio file through a vocoder model.

[0093] Figure 4 4 is a schematic diagram of the structure of an N-ary non-autoregressive speech synthesis device in one embodiment of the present invention. The N-ary non-autoregressive speech synthesis device 400 includes:

[0094] The text processing module 401 is used to obtain the text content to be synthesized and perform standardization processing on the text content to be synthesized.

[0095] In this embodiment, since the input text content contains a lot of non-standard text content, such as punctuation, numbers, symbols and other words or words, the N-gram non-autoregressive speech model cannot convert these non-standard text information into mel-spectrograms, so the text content needs to be standardized.

[0096] On the basis of the above technical solution, further, the text content to be synthesized is subjected to standardization processing. Figure 2 As shown, specifically including:

[0097] S1011, dividing the text content to be synthesized into short sentences.

[0098] In this embodiment, the input text content is usually a whole paragraph, including multiple long sentences or short sentences. In order to facilitate the model to process, the text content needs to be segmented, and the paragraph is converted into a long sentence, and the long sentence is converted into a short sentence. In this embodiment, the sentence is segmented according to punctuation marks, such as commas, semicolons, periods, exclamation marks, etc.

[0099] S1012: Perform text regularization processing on the short sentence.

[0100] The purpose of text regularization is to convert text into a more convenient and standard form of expression. The texts used in daily life contain a large number of non-standard words, which cannot be found in dictionaries and their pronunciation cannot be obtained through normal phonetic rules.

[0101] In this embodiment, this application is mainly aimed at Chinese text. In Chinese text, non-standard words refer to words containing non-Chinese characters, such as Arabic numerals, English characters, etc. This conversion process is text regularization. Text regularization is a key link in speech synthesis, and non-standard words are often the focus of user attention, such as price, date, time, etc. Text regularization directly affects the quality of speech synthesis. Non-standard words correspond to different standard words in different application scenarios. For example, 2021 is read as "two thousand and twenty-one" by number and "two thousand and twenty-one" by year.

[0102] S1013. Perform character splitting on the short sentence after text regularization processing, and output a pronunciation sequence composed of the smallest pronunciation units corresponding to the short sentence.

[0103] In this embodiment, the segmented sentences after regularization processing need to be split into words. Perform part-of-speech prediction on the split words to determine the part-of-speech of each word. The part-of-speech includes nouns, verbs, adjectives, etc. Accurately determining the part-of-speech of words can improve the accuracy of the final pronunciation sequence.

[0104] After part-of-speech prediction, it is necessary to correct the pronunciation of polyphonic characters in the words. Here, the previously predicted part-of-speech is required to determine exactly what pronunciation the polyphonic character should take. For example, in the sentence "I understand", "understand" is a verb, so the first "le" is pronounced as "liǎo", and the second "le" is a particle, so it is pronounced as "le".

[0105] After correcting the pronunciation of polyphonic characters, the smallest pronunciation unit corresponding to each character in the short sentence can be determined. In this embodiment, the text is in Chinese, so the smallest pronunciation unit is the initial and final consonants. In other embodiments, if the text is in English, the smallest pronunciation unit is the phoneme. The smallest pronunciation units corresponding to all characters in the short sentence form a pronunciation sequence.

[0106] The mel-spectrum processing module 402 is configured to input the text content to be synthesized after normalization processing into a speech model based on N-gram non-autoregressive, and output a refined mel-spectrum, where N is a natural integer.

[0107] On the basis of the above technical solution, further, the speech model based on N-gram non-autoregressive includes an N-gram rough mel-spectrum feature prediction model and an N-gram fine mel-spectrum feature prediction model.

[0108] In this embodiment, the speech model based on N-gram non-autoregressive includes an N-gram rough mel-spectrum feature prediction model and an N-gram fine mel-spectrum feature prediction model.

[0109] On the basis of the above technical solution, further, the N-gram rough mel-spectrum feature prediction model receives the pronunciation sequence and outputs A rough mel-spectrum.

[0110] In this embodiment, the N-gram rough mel-spectrum feature prediction model includes a text encoding feature module, a frame number prediction module, a text encoding feature extension module, and a mel-spectrum decoding module. The process is as Figure 3 shown, and the steps are as follows:

[0111] S1021. The text encoding feature module receives the pronunciation sequence and outputs the text encoding features corresponding to the pronunciation sequence.

[0112] S1022, the frame number prediction module predicts the frame number value corresponding to the continuous pronunciation duration of each pronunciation unit in the pronunciation sequence according to the text encoding feature, and the frame number is the number of Mel feature spectrum frames corresponding to the pronunciation unit of the final synthesized audio.

[0113] S1023, the text coding feature expansion module expands the text coding feature according to the frame number value, and finally inputs the expanded text coding feature into the Mel spectrum decoding module, and outputs Mel rough spectrum.

[0114] On the basis of the above technical solution, further, the N-ary fine Mel feature spectrum prediction model receives the Mel rough spectrum, output refined Mel spectrum.

[0115] In this embodiment, the N-ary fine Mel feature spectrum prediction model is mainly responsible for converting the received The Mel rough spectrum is refined. The N-ary refined Mel feature spectrum prediction model uses the CNN network to upsample the Mel rough spectrum into a refined Mel spectrum. That is, since the number of frames corresponding to each pronunciation unit in the Mel rough spectrum is only a fraction of the actual number of frames in the refined Mel spectrum, After the N-ary fine Mel feature spectrum prediction model is used, in addition to being able to correspond each pronunciation unit to The spectrogram data of the frame number is expanded to the spectrogram data of the real frame number, and the rough Mel spectrum can be further refined to finally generate an effective refined Mel spectrum.

[0116] The speech synthesis module 403 is used to convert the refined Mel spectrum into a speech synthesis file.

[0117] In this embodiment, the refined Mel-spectrogram is synthesized into a final speech synthesis file through a vocoder model.

[0118] On the basis of the above technical solution, further, converting the refined Mel spectrum into a speech synthesis file specifically includes:

[0119] Receiving the refined Mel spectrum;

[0120] Select the vocoder model to output the speech synthesis file according to business requirements.

[0121] On the basis of the above technical solution, further, if the business requirement is non-real-time synthesis, the WaveRNN vocoder model is selected;

[0122] If the business requirement is real-time synthesis, select the MultiBand_Melgan vocoder model.

[0123] In this embodiment, the corresponding voice coder model is selected according to the requirements of the final business. For example, real-time synthesis is selected in the scenario where the business is to interact with the user. The voice is synthesized while communicating with the user, reducing the user's waiting time. When the scenario is to read aloud large sections of text, the timeliness requirement is lower, and non-real-time synthesis can be selected.

[0124] Figure 5 It is a block diagram of an electronic device according to an exemplary embodiment.

[0125] like Figure 5 As shown, the electronic device is presented in the form of a general computing device. The processor may be one or more and work in coordination. The present invention does not exclude distributed processing, that is, the processor may be dispersed in different physical devices. The electronic device of the present invention is not limited to a single entity, but may also be the sum of multiple physical devices.

[0126] The memory stores a computer executable program, which is usually a machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least part of the steps in the method.

[0127] The memory includes a volatile memory, such as a random access memory unit (RAM) and / or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).

[0128] Optionally, in this embodiment, the electronic device further includes an I / O interface, which is used for the electronic device to exchange data with an external device. The I / O interface can represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0129] It should be understood that Figure 5 The electronic device shown is only an example of the present invention, and the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as display screens, and some electronic devices also include human-computer interaction elements such as buttons, keyboards, etc. As long as the electronic device can execute the computer-readable program in the memory to implement the method of the present invention or at least part of the steps of the method, it can be considered as an electronic device covered by the present invention.

[0130] Through the above description of the implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by combining software with necessary hardware. Figure 6As shown, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the embodiment of the present invention.

[0131] The software product may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0132] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0133] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0134] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by a device, the computer-readable medium implements the following functions: starting the virtual interaction function based on the user's start instruction; obtaining the real-time video of the user based on the virtual interaction function; inputting the real-time video into the action recognition model to generate an action recognition label, and the action recognition model is implemented by a deep learning model; generating a virtual object according to the action recognition label; drawing the target virtual object in the real-time video of the user for display.

[0135] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.

[0136] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiment of the present invention.

[0137] The exemplary embodiments of the present invention are specifically shown and described above. It should be understood that the present invention is not limited to the detailed structure, configuration or implementation method described herein; on the contrary, the present invention is intended to cover various modifications and equivalent configurations included in the spirit and scope of the appended claims.

Claims

1. A method for speech synthesis based on N-gram non-autoregressive speech, characterized in that: include: Acquire text content to be synthesized, and perform standardization processing on the text content to be synthesized, including: dividing the text content to be synthesized into short sentences, performing text regularization processing on the short sentences, performing character decomposition processing on the short sentences after the text regularization processing, and outputting a pronunciation sequence composed of the smallest pronunciation units corresponding to the short sentences; The pronunciation sequence output after the normalization process is input into a speech model based on N-ary non-autoregression, and a refined Mel spectrum is output, wherein N is the Mel spectrum granularity and N is a natural integer; wherein the speech model based on N-ary non-autoregression includes an N-ary coarse Mel feature spectrum prediction model and an N-ary refined Mel feature spectrum prediction model, and the N-ary coarse Mel feature spectrum prediction model receives the pronunciation sequence and outputs Mel rough spectrum, the N-ary fine Mel feature spectrum prediction model receives the Mel rough spectrum and output refined Mel spectrum; The refined Mel-spectrogram is converted into a speech synthesis file.

2. The method according to claim 1, characterized in that The short sentence after the text regularization processing is processed by decomposing characters, and outputting a pronunciation sequence consisting of the smallest pronunciation unit corresponding to the short sentence, further comprising: The short sentences after text regularization are split into words, the part of speech of the split words is predicted to determine the part of speech of each word, and the pronunciation of polyphones in the words is corrected to determine the minimum pronunciation unit corresponding to each character in the short sentence.

3. The method according to claim 2, characterized in that The N-ary coarse Mel feature spectrum prediction model includes a text coding feature module, a frame number prediction module, a text coding feature expansion module and a Mel spectrum decoding module; the N-ary fine Mel feature spectrum prediction model includes using a CNN network to upsample the coarse Mel spectrum into a fine Mel spectrum.

4. The method according to claim 3, characterized in that The text encoding feature module receives the pronunciation sequence and outputs the text encoding features corresponding to the pronunciation sequence; the frame number prediction module predicts the frame number corresponding to the continuous pronunciation duration of each pronunciation unit in the pronunciation sequence based on the text encoding features. The frame number is the number of Mel feature spectrum frames corresponding to the pronunciation unit of the final synthesized audio. The text encoding feature expansion module expands the text encoding feature according to the frame number value, and finally inputs the expanded text encoding feature into the Mel spectrum decoding module to output Mel rough spectrum.

5. The method according to claim 1, characterized in that The N-ary refined Mel feature spectrum prediction model includes: using a CNN network to upsample the Mel coarse spectrum into a refined Mel spectrum.

6. The method according to claim 1, characterized in that Converting the refined Mel spectrum into a speech synthesis file specifically includes: Receiving the refined Mel spectrum; Select the vocoder model to output the speech synthesis file according to business requirements.

7. The method according to claim 6, characterized in that: If the business requirement is non-real-time synthesis, select the WaveRNN vocoder model; If the business requirement is real-time synthesis, select the MultiBand_Melgan vocoder model.

8. A non-autoregressive speech synthesis device based on N-ary, characterized in that: The device includes: A text processing module, used for obtaining text content to be synthesized, and performing standardization processing on the text content to be synthesized, including: dividing the text content to be synthesized into short sentences, performing text regularization processing on the short sentences, performing character decomposition processing on the short sentences after the text regularization processing, and outputting a pronunciation sequence composed of the smallest pronunciation units corresponding to the short sentences; The mel spectrum processing module is used to input the pronunciation sequence output after the standardization processing into the speech model based on N-ary non-autoregression, and output the refined mel spectrum, wherein N is the mel spectrum granularity and N is a natural integer; wherein the speech model based on N-ary non-autoregression includes and N-ary fine mel feature spectrum prediction model, and the N-ary coarse mel feature spectrum prediction model receives the pronunciation sequence and outputs Mel rough spectrum, the N-ary fine Mel feature spectrum prediction model receives the Mel rough spectrum and output refined Mel spectrum; The speech synthesis module is used to convert the refined Mel spectrum into a speech synthesis file.

9. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images