A method, device and terminal device for audio synthesis based on duration information

Abstract

The present application is applicable to the field of audio synthesis technology, and provides an audio synthesis method, device and terminal device based on duration information, the method comprising: obtaining text data, inputting the text data into a pre-trained duration model for processing, obtaining a mapping relationship between the text data and acoustic feature information, inputting the text data and the mapping relationship into a pre-trained acoustic model for processing, and obtaining audio data. The text data is feature extracted through the duration model, the duration information of the text data is determined, and the mapping relationship between the text data and the corresponding acoustic feature information is determined based on the duration information, so as to obtain audio data corresponding to the text data based on the acoustic model, and provide a strong supervision signal for the alignment processing of the text data and the speech data, significantly reduce the alignment problem in speech synthesis, and output audio data that is completely close to the human voice and has a better rhythm.

Description

Technical Field

[0001] The present application belongs to the technical field of audio synthesis, and in particular, relates to an audio synthesis method, apparatus and terminal device based on duration information. Background Art

[0002] In recent years, with the development of human society and the advancement of science and technology, more and more life scenarios require the support of speech synthesis technology, such as AI voice navigation, intelligent interaction of home robots, and extensive electronic customer service. How to synthesize speech signals that are closer to real human voices has become a constant goal in this field. The main problems that lead to the difference between mechanical speech and human natural speech include sound noise, pronunciation accuracy, and the rhythm of the sound. Among them, the rhythm of the sound can be quantified as the duration of each pronunciation unit. Whether the rhythm is natural plays a decisive role in the quality of the sound, greatly affecting the difficulty of understanding and the user experience of the listener.

[0003] The mainstream speech synthesis acoustic method is a weakly supervised prosody learning method using an attention mechanism, which calculates the similarity between the currently decoded speech unit and the features of each text unit obtained during encoding, and obtains a one-to-many alignment between the text unit and the speech unit. The above method is prone to abnormal situations such as missing words and extra words, which affects the efficiency and accuracy of speech synthesis. Summary of the invention

[0004] The embodiments of the present application provide an audio synthesis method, apparatus, terminal device and readable storage medium based on duration information, which can solve the problems that related methods are prone to abnormal situations such as missing words and extra words, and the efficiency and accuracy of speech synthesis are low.

[0005] In a first aspect, an embodiment of the present application provides an audio synthesis method based on duration information, comprising:

[0006] Acquire data to be processed; wherein the data to be processed includes text data;

[0007] Input the text data into the pre-trained duration model for processing to obtain the mapping relationship between the text data and the acoustic feature information;

[0008] The text data and the mapping relationship are input into a pre-trained acoustic model for processing to obtain audio data.

[0009] In one embodiment, the inputting of text data into a pre-trained duration model for processing to obtain a mapping relationship between the text data and the acoustic feature information includes:

[0010] Inputting the text data into the pre-trained duration model for processing, and predicting the duration information of the text data;

[0011] Based on the duration information, a mapping relationship between the text data and the acoustic feature information is determined.

[0012] In one embodiment, determining the mapping relationship between the text data and the acoustic feature information based on the duration information includes:

[0013] Processing the text data according to the duration information to obtain a text unit sequence;

[0014] The text unit sequence is subjected to feature concatenation processing to obtain a mapping relationship between text data and acoustic feature information.

[0015] In one embodiment, the feature concatenation process is performed on the text unit sequence to obtain a mapping relationship between text data and acoustic feature information, including:

[0016] The text unit sequence and the mapping relationship are input into the pre-trained acoustic model, and the text unit sequence and the acoustic feature sequence are aligned to obtain audio data.

[0017] In one embodiment, the duration model includes a hierarchical convolutional neural network model or a bidirectional long short-term memory network model.

[0018] In one embodiment, the method further comprises:

[0019] Acquire a first training data set; wherein the first training data set includes a plurality of first text training data and duration information corresponding to each first text training data;

[0020] A corresponding label is added to the first text training data according to the duration information, and the first text training data is input into a duration model for pre-training to obtain a pre-trained duration model.

[0021] In one embodiment, the method further comprises:

[0022] Acquire a second training data set; wherein the second training data set includes a plurality of second text training data and audio training data corresponding to each second text training data;

[0023] Extracting acoustic feature information of the audio training data;

[0024] According to the acoustic feature information of the audio training data, corresponding labels are added to the second text training data, and the second text training data is input into the duration model and the acoustic model for pre-training to obtain a pre-trained duration model and a pre-trained acoustic model.

[0025] In a second aspect, an embodiment of the present application provides an audio synthesis device based on duration information, including:

[0026] A data acquisition module, used to acquire data to be processed; wherein the data to be processed includes text data;

[0027] A text data processing module is used to input text data into a pre-trained duration model for processing to obtain a mapping relationship between text data and acoustic feature information;

[0028] The audio data generation module is used to input the text data and the mapping relationship into a pre-trained acoustic model for processing to obtain audio data.

[0029] In one embodiment, the text data processing module includes:

[0030] A prediction submodule, used for inputting the text data into the pre-trained duration model for processing, and predicting the duration information of the text data;

[0031] A determination submodule is used to determine a mapping relationship between text data and acoustic feature information based on the duration information.

[0032] In one embodiment, the determining submodule includes:

[0033] A data processing unit, used for processing the text data according to the duration information to obtain a text unit sequence;

[0034] The splicing processing unit is used to perform feature splicing processing on the text unit sequence to obtain a mapping relationship between text data and acoustic feature information.

[0035] In one embodiment, the splicing processing unit includes:

[0036] The synthesis subunit is used to input the text unit sequence and the mapping relationship into the pre-trained acoustic model, align the text unit sequence with the acoustic feature sequence, and obtain audio data.

[0037] In one embodiment, the duration model includes a hierarchical convolutional neural network model or a bidirectional long short-term memory network model.

[0038] In one embodiment, the device further comprises:

[0039] A first training data acquisition module, used to acquire a first training data set; wherein the first training data set includes a plurality of first text training data and duration information corresponding to each first text training data;

[0040] The first pre-training module is used to add corresponding labels to the first text training data according to the duration information, and input the first text training data into the duration model for pre-training to obtain a pre-trained duration model.

[0041] In one embodiment, the device further includes:

[0042] A second training data acquisition module, used to acquire a second training data set; wherein the second training data set includes a plurality of second text training data and audio training data corresponding to each second text training data;

[0043] An extraction module, used to extract acoustic feature information of the audio training data;

[0044] The second pre-training module is used to add corresponding labels to the second text training data according to the acoustic feature information of the audio training data, and input the second text training data into the duration model and the acoustic model for pre-training to obtain a pre-trained duration model and a pre-trained acoustic model.

[0045] In a third aspect, an embodiment of the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements an audio synthesis method based on duration information as described in any one of the first aspects above.

[0046] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the audio synthesis method based on duration information as described in any one of the above-mentioned first aspects is implemented.

[0047] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes the audio synthesis method based on duration information as described in any one of the first aspects above.

[0048] Compared with the prior art, the embodiments of the present application have the following beneficial effects: feature extraction is performed on text data through a duration model to determine the duration information of the text data, and a mapping relationship between the text data and the corresponding acoustic feature information is determined based on the duration information, thereby obtaining audio data corresponding to the text data based on the acoustic model, providing a strong supervision signal for alignment processing of the text data and speech data, significantly reducing alignment problems in speech synthesis, and outputting audio data that is completely close to the human voice and has a better rhythm.

[0049] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 It is a flowchart of an audio synthesis method based on duration information provided in an embodiment of the present application;

[0052] Figure 2 It is a flowchart of step S102 of the audio synthesis method based on duration information provided in an embodiment of the present application;

[0053] Figure 3 It is a schematic diagram of the application process of the audio synthesis method based on duration information provided in an embodiment of the present application;

[0054] Figure 4 is a structural schematic diagram of an audio synthesis device based on duration information provided in an embodiment of the present application;

[0055] Figure 5 It is a schematic diagram of the structure of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0057] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0058] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0059] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0060] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0061] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0062] The audio synthesis method based on duration information provided in the embodiment of the present application can be applied to terminal devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, laptops, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc. The embodiment of the present application does not impose any restrictions on the specific type of terminal devices.

[0063] Figure 1 A schematic flow chart of an audio synthesis method based on duration information provided in the present application is shown. As an example but not a limitation, the method can be applied to the above-mentioned laptop computer.

[0064] S101. Acquire data to be processed; wherein the data to be processed includes text data.

[0065] Specifically, the data to be processed that needs to be synthesized and input by the user is obtained. The data to be processed includes text data that needs to be synthesized into voice data.

[0066] S102: Input the text data into the pre-trained duration model for processing to obtain a mapping relationship between the text data and the acoustic feature information.

[0067] Specifically, the text data is input into the pre-trained duration model for processing to obtain a mapping relationship between the text data and the acoustic feature information, wherein the feature information of the text data is specifically the duration information of the text data.

[0068] In order to improve the stability of audio synthesis, the text data needs to be divided into multiple smaller text units (such as dividing the text data into multiple phonemes) to synthesize the corresponding audio data according to the duration information corresponding to each text unit.

[0069] S103: Input the text data and the mapping relationship into a pre-trained acoustic model for processing to obtain audio data.

[0070] Specifically, the text data and the mapping relationship are input into a pre-trained acoustic model for processing, and an acoustic feature sequence corresponding to the duration information of the text data is determined, thereby synthesizing the audio data.

[0071] In one embodiment, the inputting of text data into a pre-trained duration model for processing to obtain a mapping relationship between the text data and the acoustic feature information includes:

[0072] Inputting the text data into the pre-trained duration model for processing, and predicting the duration information of the text data;

[0073] Based on the duration information, a mapping relationship between the text data and the acoustic feature information is determined.

[0074] Specifically, the text data is split into multiple text units to obtain a sequence of original text units, and the original text unit sequence is input into a pre-trained duration model for processing. The duration information corresponding to each original text unit is predicted and determined as the duration information of the text data. According to the duration information corresponding to each original text unit and the pre-set frame window size and offset time of the Mel spectrum, the number of Mel spectrum frames corresponding to each original text unit is calculated and determined, thereby determining a one-to-many mapping relationship between the text data and the acoustic feature information (i.e., the Mel spectrum).

[0075] There is a one-to-many relationship between specific text units (such as phonemes) and acoustic feature information (i.e., Mel spectrograms) in the time dimension. For example, if the duration of a phoneme a is 0.2 seconds, it may correspond to 64 frames of Mel spectrograms.

[0076] like Figure 2 As shown, in one embodiment, determining the mapping relationship between text data and acoustic feature information based on the duration information includes:

[0077] S1021, processing the text data according to the duration information to obtain a text unit sequence;

[0078] S1022: Perform feature concatenation processing on the text unit sequence to obtain a mapping relationship between text data and acoustic feature information.

[0079] Specifically, the number of Mel spectrum frames corresponding to each original text unit can be determined according to the duration information of each original text unit, and the original text units are copied and spliced ​​according to the number of Mel spectrum frames corresponding to each original text unit to obtain a new text unit sequence corresponding to the number of Mel spectrum frames, and the new text unit sequence is feature spliced ​​with the Mel spectrum to obtain the prediction result of the acoustic feature (Mel spectrum frame) output by the pre-trained duration model.

[0080] After copying and splicing the text units, the obtained text unit sequence can be expressed as:

[0081]

[0082] in, Represents the Mel spectrum of the first phoneme, and the corresponding x0,x0,x0 represents the first phoneme after copying and concatenation.

[0083] Figure 3 An exemplary application flow chart of an audio synthesis method based on duration information is provided.

[0084] like Figure 3 As shown, the processing of text data is as follows:

[0085] Take {x0,x1,.....xn} as the text unit sequence input to the acoustic model. Taking the subsequence {x0,x1} as an example, first input the subsequence into the encoder (where the encoder still performs conventional calculations, inputs the information of the original text unit, and uses a hierarchical convolutional neural network and a bidirectional long short-term memory network to capture contextual information) to obtain the original text unit feature information (i.e., the duration information of the original text unit). At the same time, the original text unit sequence is input into the duration model for processing, and the duration information of each original text unit in the text data is predicted. Then, based on the predicted duration information of each original text unit, it is copied and spliced ​​to obtain a new text unit sequence aligned with the acoustic feature sequence.

[0086] Then, yes Unit and x i-1 Perform the corresponding feature concatenation processing and input it into the LSTM network. The LSTM output is hi; hi and x i-1The corresponding coded features are concatenated and then input into the fully connected layer, which outputs the prediction result of the acoustic feature information (Mel spectrum) at time i.

[0087] The reasoning process of the LSTM network is shown in Formula 1:

[0088] c t ,h t =LSTM(x t ,c t-1 ,h t-1 )

[0089] f t =σ g (W f x t +U f h t-1 +b f )

[0090] i t =σ g (W i x t +U i h t-1 +b i )

[0091] o t =σ g (W o x t +U0h t-1 +b o )

[0092]

[0093]

[0094] Among them, x represents the input vector; W, U,, represent matrices; b represents the bias vector, W, U, b are the parameters of the LSTM model (which can be determined through the training process of the neural network); σ represents the activation function (sigmoid); ○ represents vector multiplication (element wise multiply).

[0095] The input and output definitions of the LSTM network are shown in Formula 2:

[0096]

[0097] c t ,h t =LSTM(x t ,c t-1 ,h t-1 )

[0098]

[0099] Represents a splicing operation;

[0100] Among them, I t For text units Figure 3 The convolutional features obtained by the convolutional layer in M t-1 and M t are the text unit feature information at time t-1 and time t obtained by the encoder respectively.

[0101] In one embodiment, the feature concatenation process is performed on the text unit sequence to obtain a mapping relationship between text data and acoustic feature information, including:

[0102] The text unit sequence and the mapping relationship are input into the pre-trained acoustic model, and the text unit sequence and the acoustic feature sequence are aligned to obtain audio data.

[0103] Specifically, the text unit sequence and the mapping relationship between the text unit sequence and the acoustic feature sequence are input into the pre-trained acoustic model, and the text unit sequence and the acoustic feature sequence are aligned based on the mapping relationship to obtain audio data.

[0104] In one embodiment, the duration model includes a hierarchical convolutional neural network model or a bidirectional long short-term memory network model.

[0105] Specifically, the duration model includes but is not limited to a hierarchical convolutional neural network model or a bidirectional long short-term memory network model.

[0106] In one embodiment, the method further comprises:

[0107] Acquire a first training data set; wherein the first training data set includes a plurality of first text training data and duration information corresponding to each first text training data;

[0108] A corresponding label is added to the first text training data according to the duration information, and the first text training data is input into a duration model for pre-training to obtain a pre-trained duration model.

[0109] Specifically, multiple first text training data and duration information corresponding to each first text training data are obtained to obtain a first training data set, and corresponding labels are added to the corresponding first text training data based on each duration information (specifically, corresponding duration information labels are added to each phoneme in the first text training data), and the first text training data are input into the duration model for pre-training to obtain a pre-trained duration model.

[0110] In one embodiment, the method further comprises:

[0111] Acquire a second training data set; wherein the second training data set includes a plurality of second text training data and audio training data corresponding to each second text training data;

[0112] Extracting acoustic feature information of the audio training data;

[0113] According to the acoustic feature information of the audio training data, corresponding labels are added to the second text training data, and the second text training data is input into the duration model and the acoustic model for pre-training to obtain a pre-trained duration model and a pre-trained acoustic model.

[0114] Specifically, a plurality of second text training data and audio training data corresponding to each second text training data are obtained to obtain a second training data set, the acoustic feature information (i.e., Mel spectrum) of the audio training data is extracted, labels are added to the corresponding second text training data according to the acoustic feature information of each audio training data (specifically, corresponding labels are added to the start and end points of the moment in the audio data corresponding to each phoneme in the second text training data), and the second text training data is input into the duration model and the acoustic model for pre-training. The training objectives include: simultaneous duration prediction (i.e., prediction of the duration information of each second text training data) and Mel spectrum generation (generating audio data corresponding to the second text training data). During the training process, the real duration is used to supervise the learning of the decoder, and during the inference process, the input data of the duration model is used as a guide, thereby obtaining a pre-trained duration model and a pre-trained acoustic model.

[0115] This embodiment extracts features from text data through a duration model, determines the duration information of the text data, and determines the mapping relationship between the text data and the corresponding acoustic feature information based on the duration information, thereby obtaining audio data corresponding to the text data based on the acoustic model, providing a strong supervision signal for the alignment processing of the text data and speech data, significantly reducing the alignment problem in speech synthesis, and outputting audio data that is completely close to the human voice and has a better rhythm.

[0116] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0117] Corresponding to the audio synthesis method based on duration information described in the above embodiment, Figure 4 A structural block diagram of an audio synthesis device based on duration information provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0118] Reference Figure 4 The audio synthesis device 100 based on duration information comprises:

[0119] The data acquisition module 101 is used to acquire data to be processed; wherein the data to be processed includes text data;

[0120] A text data processing module 102, used to input text data into a pre-trained duration model for processing, and obtain a mapping relationship between text data and acoustic feature information;

[0121] The audio data generating module 103 is used to input the text data and the mapping relationship into a pre-trained acoustic model for processing to obtain audio data.

[0122] In one embodiment, the text data processing module includes:

[0123] A prediction submodule, used for inputting the text data into the pre-trained duration model for processing, and predicting the duration information of the text data;

[0124] A determination submodule is used to determine a mapping relationship between text data and acoustic feature information based on the duration information.

[0125] In one embodiment, the determining submodule includes:

[0126] A data processing unit, used for processing the text data according to the duration information to obtain a text unit sequence;

[0127] The splicing processing unit is used to perform feature splicing processing on the text unit sequence to obtain a mapping relationship between text data and acoustic feature information.

[0128] In one embodiment, the splicing processing unit includes:

[0129] The synthesis subunit is used to input the text unit sequence and the mapping relationship into the pre-trained acoustic model, align the text unit sequence with the acoustic feature sequence, and obtain audio data.

[0130] In one embodiment, the duration model includes a hierarchical convolutional neural network model or a bidirectional long short-term memory network model.

[0131] In one embodiment, the device further comprises:

[0132] A first training data acquisition module, used to acquire a first training data set; wherein the first training data set includes a plurality of first text training data and duration information corresponding to each first text training data;

[0133] The first pre-training module is used to add corresponding labels to the first text training data according to the duration information, and input the first text training data into the duration model for pre-training to obtain a pre-trained duration model.

[0134] In one embodiment, the device further includes:

[0135] A second training data acquisition module, used to acquire a second training data set; wherein the second training data set includes a plurality of second text training data and audio training data corresponding to each second text training data;

[0136] An extraction module, used to extract acoustic feature information of the audio training data;

[0137] The second pre-training module is used to add corresponding labels to the second text training data according to the acoustic feature information of the audio training data, and input the second text training data into the duration model and the acoustic model for pre-training to obtain a pre-trained duration model and a pre-trained acoustic model.

[0138] This embodiment extracts features from text data through a duration model, determines the duration information of the text data, and determines the mapping relationship between the text data and the corresponding acoustic feature information based on the duration information, thereby obtaining audio data corresponding to the text data based on the acoustic model, providing a strong supervision signal for the alignment processing of the text data and speech data, significantly reducing the alignment problem in speech synthesis, and outputting audio data that is completely close to the human voice and has a better rhythm.

[0139] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0140] Figure 5 This is a schematic diagram of the structure of the terminal device provided in this embodiment. Figure 5 As shown, the terminal device 5 of this embodiment includes: at least one processor 50 ( Figure 5 Only one is shown), a memory 51 and a computer program 52 stored in the memory 51 and executable on the at least one processor 50, wherein the processor 50 implements the steps in any of the above-mentioned embodiments of the audio synthesis method based on duration information when executing the computer program 52.

[0141] The terminal device 5 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The terminal device may include, but is not limited to, a processor 50 and a memory 51. Those skilled in the art will appreciate that Figure 5It is only an example of the terminal device 5 and does not constitute a limitation on the terminal device 5. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0142] The processor 50 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0143] In some embodiments, the memory 51 may be an internal storage unit of the terminal device 5, such as a hard disk or memory of the terminal device 5. In other embodiments, the memory 51 may also be an external storage device of the terminal device 5, such as a plug-in hard disk, a smart media card (SMC), a secure digital card (SD), a flash card, etc. equipped on the terminal device 5. Further, the memory 51 may also include both an internal storage unit and an external storage device of the terminal device 5. The memory 51 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 51 may also be used to temporarily store data that has been output or is to be output.

[0144] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0145] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0146] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0147] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0148] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0149] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0150] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0151] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0152] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An audio synthesis method based on duration information, characterized in that: include: Acquire data to be processed; wherein the data to be processed includes text data; Inputting the text data into a pre-trained duration model for processing to obtain a mapping relationship between the text data and the acoustic feature information, including: inputting the text data into the pre-trained duration model for processing to predict the duration information of the text data; processing the text data according to the duration information to obtain a text unit sequence; performing feature splicing processing on the text unit sequence to obtain a mapping relationship between the text data and the acoustic feature information; The text data and the mapping relationship are input into a pre-trained acoustic model for processing, an acoustic feature sequence corresponding to the duration information of the text data is determined, and audio data is obtained based on the acoustic feature sequence.

2. The audio synthesis method based on duration information as claimed in claim 1, characterized in that: The step of performing feature concatenation processing on the text unit sequence to obtain a mapping relationship between text data and acoustic feature information includes: The text unit sequence and the mapping relationship are input into the pre-trained acoustic model, and the text unit sequence and the acoustic feature sequence are aligned to obtain audio data.

3. The audio synthesis method based on duration information as claimed in claim 1, characterized in that: The duration model includes a hierarchical convolutional neural network model or a bidirectional long short-term memory network model.

4. The audio synthesis method based on duration information according to any one of claims 1 to 3, characterized in that: The method further comprises: Acquire a first training data set; wherein the first training data set includes a plurality of first text training data and duration information corresponding to each first text training data; A corresponding label is added to the first text training data according to the duration information, and the first text training data is input into a duration model for pre-training to obtain a pre-trained duration model.

5. The audio synthesis method based on duration information according to any one of claims 1 to 3, characterized in that: The method further comprises: Acquire a second training data set; wherein the second training data set includes a plurality of second text training data and audio training data corresponding to each second text training data; Extracting acoustic feature information of the audio training data; According to the acoustic feature information of the audio training data, corresponding labels are added to the second text training data, and the second text training data is input into the duration model and the acoustic model for pre-training to obtain a pre-trained duration model and a pre-trained acoustic model.

6. An audio synthesis device based on duration information, characterized in that: include: A data acquisition module, used to acquire data to be processed; wherein the data to be processed includes text data; A text data processing module, used for inputting text data into a pre-trained duration model for processing, and obtaining a mapping relationship between text data and acoustic feature information; the text data processing module includes: a prediction submodule, used for inputting the text data into the pre-trained duration model for processing, and predicting the duration information of the text data; a determination submodule, used for determining the mapping relationship between text data and acoustic feature information based on the duration information; the determination submodule includes: a data processing unit, used for processing the text data according to the duration information, and obtaining a text unit sequence; a splicing processing unit, used for performing feature splicing processing on the text unit sequence, and obtaining a mapping relationship between text data and acoustic feature information; The audio data generation module is used to input the text data and the mapping relationship into a pre-trained acoustic model for processing, determine the acoustic feature sequence corresponding to the duration information of the text data, and obtain audio data based on the acoustic feature sequence.

7. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images