A pronunciation reverse derivation method and device of a fusion multi-task learning variational autoencoder

Abstract

The application relates to the technical field of speech recognition, and discloses a pronunciation reverse deduction method and device of a variational autoencoder fused with multi-task learning, which comprises the following steps: collecting synchronous speech signals and pronunciation movement data; training an acoustic domain variational autoencoder based on the speech signals, wherein the acoustic domain variational autoencoder inputs first speaker features into a speaker recognizer to identify the speaker; finally, the first content features and the first speaker features are input into a first decoder to reconstruct acoustic features; training a pronunciation domain variational autoencoder based on the pronunciation movement data, wherein the pronunciation domain variational autoencoder inputs second content features into a second speech recognizer to identify the speech content; finally, the second content features and second speaker features are input into a second decoder to reconstruct pronunciation features; and constructing a pronunciation reverse deduction system to synthesize a pronunciation movement track.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech recognition, and particularly relates to a pronunciation inverse deduction method and device of a variational autoencoder fusing multi-task learning. BACKGROUND

[0002] It has been found in research that part of speaker information is contained in the content features obtained based on the variational autoencoder decoupling, and part of content information is contained in the speaker information, the speaker information from the content features and the content information from the speaker features conflict after being mapped by a neural network, resulting in that the performance of downstream tasks based on the variational autoencoder is adversely affected. Therefore, a variational autoencoder based on mutual information and a variational autoencoder based on weighting are respectively proposed to try to eliminate the noise information in the content features and the speaker features and improve the performance of downstream tasks. However, both of the two methods are indirect constraints on the content features and the speaker features, and there are some difficult problems that have not been completely solved, such as the estimation of mutual information in the variational autoencoder based on mutual information is very difficult and cannot be evaluated.

[0003] The content features obtained based on the variational autoencoder contain part of speaker information, and the speaker features contain part of content information, which brings difficulty to subsequent pronunciation inverse deduction, resulting in that the accuracy of pronunciation inverse deduction is greatly affected. Therefore, an effective method is needed to reduce the noise information in the content features and the speaker features and further improve the performance of speaker-independent pronunciation inverse deduction. SUMMARY

[0004] To achieve the object of the present application, the present application provides a pronunciation inverse deduction method of a variational autoencoder fusing multi-task learning, comprising:

[0005] Step S1: collecting synchronous speech signals and pronunciation movement data;

[0006] Step S2: training an acoustic domain variational autoencoder based on the speech signals, wherein the acoustic domain variational autoencoder comprises a first content encoder, a speaker encoder, a first speech recognizer, a speaker recognizer and a first decoder, the first content encoder and the speaker encoder are respectively used to extract first content features and first speaker features from the speech signals, the first content features are input into the first speech recognizer to recognize speech content, and the first speaker features are input into the speaker recognizer to recognize a speaker; finally, the first content features and the first speaker features are input into the first decoder to reconstruct acoustic features;

[0007] Step S3: training an articulatory domain variational autoencoder based on the articulatory movement data, wherein the articulatory domain variational autoencoder comprises the speaker encoder, a second content encoder, a second speech recognizer and a second decoder, the speaker encoder is configured to extract second speaker features from the speech signal, the second content encoder is configured to extract second content features from the articulatory movement data, and the second content features are input into the second speech recognizer to recognize speech content; and finally the second content features and the second speaker features are input into the second decoder to reconstruct articulatory features;

[0008] Step S4: constructing a content inverse network, and constructing an articulatory inverse system based on the first content encoder, the first speaker encoder, the content inverse network and the second decoder to synthesize articulatory movement trajectories.

[0009] In some embodiments, the training loss function of the acoustic domain variational autoencoder is constructed based on the reconstruction loss, the distribution loss, the speech recognition task loss and the speaker recognition task loss.

[0010] In some embodiments, the training loss function of the articulatory domain variational autoencoder is constructed based on the reconstruction loss, the distribution loss and the speech recognition task loss.

[0011] In some embodiments, the content inverse network comprises a three-layer bidirectional long short-term memory network and a two-layer fully connected neural network; and the loss function of the content inverse network comprises a content feature loss and an articulatory movement reconstruction loss to optimize the parameters of the content inverse network and the second decoder.

[0012] In some embodiments, the loss function of the content inverse network is constructed based on the content feature loss and the articulatory movement reconstruction loss.

[0013] In some embodiments, the content inverse network comprises a three-layer bidirectional long short-term memory network and a two-layer fully connected neural network.

[0014] In some embodiments, the acoustic domain variational autoencoder further comprises an acoustic feature extractor configured to extract acoustic features from the speech signal; and the first content encoder and the speaker encoder are configured to extract the first content features and the first speaker features based on the acoustic features, respectively.

[0015] In some embodiments, the articulatory domain variational autoencoder further comprises an acoustic feature extractor configured to extract acoustic features from the speech signal and an articulatory feature extractor configured to extract articulatory features from the articulatory movement data; the speaker encoder extracts second speaker features based on the acoustic features, and the second content encoder extracts second content features based on the articulatory features.

[0016] To achieve the same inventive purpose, the application also provides an articulatory inverse prediction device of a multi-task learning fused variational autoencoder, comprising:

[0017] A data acquisition module is configured to acquire synchronized speech signal and articulatory movement data.

[0018] An acoustic feature reconstruction module is configured to train an acoustic domain variational autoencoder based on the speech signal, wherein the acoustic domain variational autoencoder comprises a first content encoder, a speaker encoder, a first speech recognizer, a speaker recognizer and a first decoder, the first content encoder and the speaker encoder are respectively configured to extract first content features and first speaker features from the speech signal, input the first content features into the first speech recognizer to recognize the speech content, and input the first speaker features into the speaker recognizer to recognize the speaker; finally, the first content features and the first speaker features are input into the first decoder to reconstruct the acoustic features.

[0019] An articulatory feature reconstruction module is configured to train an articulatory domain variational autoencoder based on the articulatory movement data, wherein the articulatory domain variational autoencoder comprises the speaker encoder, a second content encoder, a second speech recognizer and a second decoder, the speaker encoder is configured to extract second speaker features from the speech signal, the second content encoder is configured to extract second content features from the articulatory movement data, and the second content features are input into the second speech recognizer to recognize the speech content; finally, the second content features and the second speaker features are input into the second decoder to reconstruct the articulatory features.

[0020] An articulatory trajectory generation module is configured to construct a content inverse prediction network and construct an articulatory inverse prediction system based on the first content encoder, the first speaker encoder, the content inverse prediction network and the second decoder, and synthesize articulatory movement trajectories.

[0021] In some embodiments, the content inverse prediction network is composed of a three-layer bidirectional long short-term memory network and a two-layer fully connected neural network, and the loss function includes content feature loss and articulatory movement reconstruction loss to optimize the parameters of the content inverse prediction network and the second decoder, respectively.

[0022] The above technical solutions have the following beneficial effects:

[0023] The pronunciation inverse inference method and device of the variational autoencoder fusing multi-task learning provided by the application directly use the speaker feature extracted from the acoustic domain as the input of the decoder in the pronunciation domain variational autoencoder, thereby eliminating the speaker inverse inference network and reducing the cumulative error caused thereby; meanwhile, the method of multi-task learning is used to constrain the content encoder and the speaker encoder, thereby reducing the noise information in the content feature and the speaker feature and improving the performance of pronunciation inverse inference. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0025] Figure 1 A flowchart of a pronunciation inverse inference method of a variational autoencoder fusing multi-task learning provided by an embodiment of the application;

[0026] Figure 2 A whole framework diagram of a pronunciation inverse inference method of a variational autoencoder fusing multi-task learning provided by an embodiment of the application;

[0027] Figure 3 A training framework diagram of an acoustic variational autoencoder based on multi-task learning of a pronunciation inverse inference method of a variational autoencoder fusing multi-task learning provided by an embodiment of the application;

[0028] Figure 4 A training framework diagram of a pronunciation variational autoencoder based on multi-task learning of a pronunciation inverse inference method of a variational autoencoder fusing multi-task learning provided by an embodiment of the application;

[0029] Figure 5 A structure diagram of a pronunciation inverse inference device of a variational autoencoder fusing multi-task learning provided by an embodiment of the application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some embodiments of the application, but not all the embodiments.

[0031] Examples of the described embodiments are illustrated in the accompanying drawings, throughout which like reference characters represent like elements or elements with similar functionality. The embodiments described below are exemplary and intended to be illustrative of the application, and are not to be construed as limiting the application.

[0032] Embodiment one

[0033] One embodiment of the present application provides a pronunciation inversion method of a fusion multi-task learning variational autoencoder, referring to Figures 1-4 As shown in the figure, comprising:

[0034] Step S1: Collecting synchronized speech signal and pronunciation movement data;

[0035] Step S2: Training an acoustic domain variational autoencoder based on the speech signal, wherein the acoustic domain variational autoencoder includes a first content encoder, a speaker encoder, a first speech recognizer, a speaker recognizer and a first decoder, the first content encoder and the speaker encoder are respectively used to extract first content features and first speaker features from the speech signal, the first content features are input into the first speech recognizer to recognize the speech content, and the first speaker features are input into the speaker recognizer to recognize the speaker; finally, the first content features and the first speaker features are input into the first decoder to reconstruct the acoustic features;

[0036] Step S3: Training a pronunciation domain variational autoencoder based on the pronunciation movement data, wherein the pronunciation domain variational autoencoder includes the speaker encoder, a second content encoder, a second speech recognizer and a second decoder, the speaker encoder is used to extract second speaker features from the speech signal, the second content encoder is used to extract second content features from the pronunciation movement data, and the second content features are input into the second speech recognizer to recognize the speech content; finally, the second content features and the second speaker features are input into the second decoder to reconstruct the pronunciation features;

[0037] Step S4: Constructing a content inversion network, and constructing a pronunciation inversion system based on the first content encoder, the first speaker encoder, the content inversion network and the second decoder, and synthesizing the pronunciation movement trajectory.

[0038] In one specific embodiment of the present application, the training loss function of the acoustic domain variational autoencoder is constructed based on the reconstruction loss, the distribution loss, the speech recognition task loss and the speaker recognition task loss, and is expressed as:

[0039] L1=L rec1 +L KL1 +L task_11 +L task_12

[0040] wherein L rec1 is a reconstruction loss, L KL1 is a distribution loss, L task_11 is a task loss of speech recognition, L task_12 is a task loss of speaker recognition.

[0041] In one specific embodiment of the present application, a training loss function of the articulatory domain variational autoencoder is constructed based on the reconstruction loss, the distribution loss and the task loss of speech recognition, denoted as:

[0042] L2=L rec2 +L KL2 +L task_21

[0043] wherein L rec2 is a reconstruction loss, L KL2 is a distribution loss, L task_21 is a task loss of speech recognition.

[0044] In one specific embodiment of the present application, the content inverse network is composed of a three-layer bidirectional long short-term memory network and a two-layer fully connected neural network; and a loss function of the content inverse network includes a content feature loss and an articulatory movement reconstruction loss, so as to optimize parameters of the content inverse network and the second decoder.

[0045] In one specific embodiment of the present application, the loss function of the content inverse network is determined according to the following formula:

[0046] L3=L c +L rec3

[0047] wherein L c denotes a content feature loss, L rec3 denotes an articulatory movement reconstruction loss.

[0048] In one specific embodiment of the present application, the content inverse network is composed of a three-layer bidirectional long short-term memory network and a two-layer fully connected neural network.

[0049] In one specific embodiment of the present application, the features learned by the variational autoencoder are constrained by the speech recognition task and the speaker recognition task, so as to reduce noise information in the content features and the speaker features, and then the learned new features are used for articulatory inverse respectively, and the decoder of the pre-trained articulatory domain VAE is used to synthesize the articulatory trajectory.

[0050] In one specific embodiment of the present application, the acoustic domain variational autoencoder further comprises an acoustic feature extractor configured to extract acoustic features from the speech signal; the first content encoder and the speaker encoder are configured to extract first content features and first speaker features based on the acoustic features, respectively.

[0051] In one specific embodiment of the present application, the articulatory domain variational autoencoder further comprises an acoustic feature extractor configured to extract acoustic features from the speech signal and an articulatory feature extractor configured to extract articulatory features from the articulatory movement data; the speaker encoder is configured to extract second speaker features based on the acoustic features, and the second content encoder is configured to extract second content features based on the articulatory features.

[0052] As shown in Figure 2 , the whole architecture is composed of a first convolutional network, a first speaker encoder, a first content encoder, a content inverse network, and a second decoder. The speech signal is processed by the first convolutional network, the first speaker encoder, and the first content encoder to obtain the speaker features and the acoustic-based content features, respectively. Then, the articulatory content features are obtained based on the acoustic content features by the content inverse network. Finally, the speaker features and the articulatory content features are input into the second decoder to synthesize the articulatory movement trajectory. Figure 1 The first convolutional network, the first speaker encoder, the first content encoder, and other modules are obtained by training an acoustic feature-based variational autoencoder.

[0053] To eliminate the noise information in the content features and the speaker features, the present application proposes a framework based on multi-task learning. In the training process, the speech recognition task is used to constrain the content features to reduce the noise information in the content features; the speaker recognition task is used to constrain the speaker features to reduce the noise information in the speaker features, as shown in Figure 3 The first convolutional network, the first speaker encoder, the first content encoder, and other modules obtained by training are used in the framework shown in Figure 2 .

[0054] Figure 2 The decoder in the framework shown in Figure 4The pronunciation variational autoencoder network shown is used. Speaker features primarily identify the speaker's identity during pronunciation derivation, without affecting the content information contained in the speech signal. Speaker identity can be directly identified using speaker features based on acoustic signals. Therefore, during the training of the pronunciation variational autoencoder, speaker features are extracted from the acoustic acoustic features by the first speaker encoder of the acoustic domain variational autoencoder; content features are extracted from the pronunciation features by the second content encoder of the pronunciation domain variational autoencoder; and the second decoder reconstructs the input pronunciation features based on the input speaker features and content features. To minimize noise in the content features, we also use a speech recognition task to constrain the content features, reducing noise. The parameters of the second decoder are determined during training. Figure 2 The system shown can be fine-tuned.

[0055] Figure 2 The parameters of the content inverse analysis module are randomly initialized and updated throughout the training process of the entire pronunciation inverse deduction system. The content inverse analysis module uses a structure of a 3-layer bidirectional long short-time recurrent neural network (BLSTM) superimposed with a 2-layer fully connected neural network to realize the mapping from content embedding based on acoustic features to content embedding based on pronunciation motion features.

[0056] 1) The specific loss function for training the acoustic domain variational autoencoder is as follows:

[0057] L1 = L rec1 +L KL1 +L task_11 +L task_12 (1)

[0058]

[0059]

[0060] Among them, L rec1 For the first reconstruction loss, L KL1 For the first distribution loss, L task_11 For the loss of the first auxiliary task, namely speech recognition, L task_12 This is the loss for the second auxiliary task, namely speaker recognition. These are the outputs of the first content encoder and the speaker encoder, respectively; N is the number of speech features. To obey the mean μ i The covariance matrix is ​​Λ i The normal distribution, where Λ i A diagonal matrix; d is The dimension of the dataset. N_ph is the number of phonemes in the dataset. Let be the probability that the speech in frame t is predicted to be the i-th phoneme. is an indicator function (its value is 1 if the true phoneme at time t is i; otherwise, its value is 0) ; N_spk is the number of speakers in the dataset, is the probability that the nth sentence is predicted to be uttered by the ith speaker, is an indicator function (its value is 1 if the nth sentence is uttered by the ith speaker; otherwise, its value is 0). The sum of the first two terms on the right side of equation (1) is the loss function used in the standard training of variational autoencoder; the third and fourth terms on the right side of equation (1) are the loss functions of the auxiliary learning task introduced by the multi-task learning method of the present application.

[0061] 2) The specific loss function for the training of the articulatory variational autoencoder is as follows:

[0062] L2=L rec2 +L KL2 +L task_21 (6)

[0063]

[0064] wherein, L rec2 is the second reconstruction loss, L KL2 is the second distribution loss, and L task_21 is the task loss of speech recognition; are the outputs of the second content encoder and the speaker encoder, respectively; N is the number of articulatory features; is a normal distribution with mean μ i and covariance matrix Λ i , wherein Λ i is a diagonal matrix; d is the dimension of , N_ph is the number of phonemes in the dataset, is the probability that the tth frame of speech is predicted to be the ith phoneme, is an indicator function (its value is 1 if the true phoneme at time t is i; otherwise, its value is 0).

[0065] 3) Training of the content feature inverse propagation network and fine-tuning of the decoder

[0066] L3=L c +L rec3 (10)

[0067]

[0068] wherein, L c represents the content feature loss, and L rec3 represents the articulatory movement reconstruction loss; is the content feature of the true articulatory space corresponding to the ith sample, a content feature of a content embedding reverse network estimation result of a pronunciation space; a real pronunciation organ position vector corresponding to the i th sample, a pronunciation organ position vector corresponding to the i th sample estimated by a decoder.

[0069] The present application directly uses the speaker feature extracted from the acoustic domain as the input of the decoder in the pronunciation domain variational autoencoder, saves the speaker reverse network, and reduces the cumulative error caused thereby; meanwhile, the present application uses the multi-task learning method to constrain the content encoder and the speaker encoder, reduces the noise information in the content feature and the speaker feature, and improves the pronunciation reverse performance.

[0070] Embodiment two

[0071] One embodiment of the present application provides a pronunciation reverse device of a variational autoencoder fused with multi-task learning, referring to Figure 5 as shown, comprising:

[0072] The data acquisition module 10 is used to acquire synchronized speech signals and pronunciation movement data.

[0073] The acoustic domain encoder training module 20 is used to train an acoustic domain variational autoencoder based on the speech signals, wherein the acoustic domain variational autoencoder comprises a content encoder and a speaker encoder, so as to extract a content feature and a speaker feature from the speech signals.

[0074] The pronunciation domain encoder training module 30 is used to train a pronunciation domain variational autoencoder based on the pronunciation movement data, wherein the pronunciation domain variational autoencoder comprises a content encoder and a decoder, so as to extract a pronunciation content feature from the pronunciation movement data and reconstruct a pronunciation movement trajectory.

[0075] The content and speaker constraint module 40 is used to introduce a speech recognition task and a speaker recognition task as auxiliary tasks when training the acoustic domain variational autoencoder, so as to constrain the content feature and the speaker feature respectively, and reduce the noise information therein.

[0076] The pronunciation content constraint module 50 is used to introduce a speech recognition task as an auxiliary task when training the pronunciation domain variational autoencoder, so as to constrain the pronunciation content feature.

[0077] The pronunciation reverse module 60 is used to construct a pronunciation reverse system, comprising the content encoder and the speaker encoder of the acoustic domain variational autoencoder, a content reverse network, and the decoder of the pronunciation domain variational autoencoder.

[0078] The feature mapping module 70 is configured to map acoustic content features to articulatory content features by using the content inverse network.

[0079] The articulatory trajectory generation module 80 is configured to input the articulatory content features and speaker features into the decoder to synthesize articulatory movement trajectories.

[0080] In one specific embodiment of the present application, the training loss function of the acoustic domain variational autoencoder is:

[0081] L1=L rec1 +L KL1 +L task_11 +L task_12

[0082] wherein L rec1 is a reconstruction loss, L KL1 is a distribution loss, L task_11 is a task loss of speech recognition, and L task_12 is a task loss of speaker recognition.

[0083] In one specific embodiment of the present application, the training loss function of the articulatory domain variational autoencoder is:

[0084] L2=L rec2 +L KL2 +L task_21

[0085] wherein L rec2 is a reconstruction loss, L KL2 is a distribution loss, and L task_21 is a task loss of speech recognition.

[0086] In one specific embodiment of the present application, the loss function of the content inverse network is determined according to the following formula:

[0087] L3=L c +L rec3

[0088] wherein L c represents a content feature loss, and L rec3 represents an articulatory movement reconstruction loss.

[0089] In one specific embodiment of the present application, the content inverse network is composed of a three-layer bidirectional long short-term memory network and a two-layer fully connected neural network, and the loss function includes a content feature loss and an articulatory movement reconstruction loss to optimize the parameters of the content inverse network and the second decoder, respectively.

[0090] In a specific embodiment of the present application, the features learned by variational auto-encoding are constrained by speech recognition tasks and speaker recognition tasks, noise information in the content features and the speaker features is reduced, and then the learned new features are used for pronunciation reverse respectively, and the decoder of the pre-trained pronunciation domain VAE is used to synthesize the pronunciation track.

[0091] In a specific embodiment of the present application, the acoustic domain variational auto-encoder further comprises an acoustic feature extractor for extracting acoustic features from the speech signal; the first content encoder and the speaker encoder extract first content features and first speaker features based on the acoustic features respectively.

[0092] In a specific embodiment of the present application, the pronunciation domain variational auto-encoder further comprises an acoustic feature extractor and a pronunciation feature extractor, the acoustic feature extractor is used to extract acoustic features from the speech signal, and the pronunciation feature extractor is used to extract pronunciation features from the pronunciation movement data; the speaker encoder extracts second speaker features based on the acoustic features, and the second content encoder extracts second content features based on the pronunciation features.

[0093] As described above, the present application provides a pronunciation reverse device of a variational auto-encoder for multi-task learning, directly uses the speaker features extracted from the acoustic domain as the input of the decoder in the pronunciation domain variational auto-encoder, omits the speaker reverse network and reduces the cumulative error caused thereby; at the same time, the present application uses the method of multi-task learning to constrain the content encoder and the speaker encoder, reduces the noise information in the content features and the speaker features, and improves the performance of pronunciation reverse.

[0094] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0095] The various embodiments in the specification are described in progressive manner, and each embodiment focuses on the difference from other embodiments, and the same or similar parts between the embodiments can be referred to each other. The embodiments of the present application are described with reference to flowcharts and / or block diagrams of the method, terminal device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the computer or other programmable data processing terminal device produce a device that implements the functions specified in the flow Figure 1 one or more flows and / or blocks Figure 1 one or more flows and / or blocks Figure 1 one or more flows and / or blocks Figure 1 one or more flows and / or blocks Figure 1 one or more flows and / or blocks Figure 1The steps of the functions specified in one or more boxes. Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the invention. Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0096] The methods and apparatus provided by the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

[0097] In the description of this specification, references to terms such as "an embodiment," "some embodiments," "example," "specific example," or "a specific embodiment" or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A pronunciation inverse method integrating a variational autoencoder with multi-task learning, characterized in that, include: Step S1: Acquire synchronized speech signals and vocalization motion data; Step S2: Train an acoustic domain variational autoencoder based on the speech signal, wherein the acoustic domain variational autoencoder includes a first content encoder, a speaker encoder, a first speech recognizer, a speaker recognizer, and a first decoder. The first content encoder and the speaker encoder are respectively used to extract a first content feature and a first speaker feature from the speech signal. The first content feature is input into the first speech recognizer to recognize the speech content, and the first speaker feature is input into the speaker recognizer to recognize the speaker. Finally, the first content feature and the first speaker feature are input into the first decoder to reconstruct the acoustic features. Step S3: Train a phonetic domain variational autoencoder based on the phonetic motion data, wherein the phonetic domain variational autoencoder includes a speaker encoder, a second content encoder, a second speech recognizer, and a second decoder. The speaker encoder is used to extract a second speaker feature from the speech signal, the second content encoder is used to extract a second content feature from the phonetic motion data, and input the second content feature into the second speech recognizer to recognize the speech content; finally, the second content feature and the second speaker feature are input into the second decoder to reconstruct the phonetic features. Step S4: Construct a content inverse network, and construct a pronunciation inverse system based on the first content encoder, the first speaker encoder, the content inverse network, and the second decoder to synthesize the pronunciation motion trajectory.

2. The pronunciation inverse method of the variational autoencoder integrating multi-task learning according to claim 1, characterized in that, The training loss function of the acoustic domain variational autoencoder is constructed based on reconstruction loss, distribution loss, speech recognition task loss, and speaker recognition task loss.

3. The pronunciation inverse method of the variational autoencoder integrating multi-task learning according to claim 1, characterized in that, The training loss function of the pronunciation domain variational autoencoder is constructed based on reconstruction loss, distribution loss, and speech recognition task loss.

4. The pronunciation inverse method of the variational autoencoder integrating multi-task learning according to claim 1, characterized in that, The content inversion network consists of a three-layer bidirectional long short-term memory network and a two-layer fully connected neural network. The loss function includes content feature loss and pronunciation motion reconstruction loss to optimize the parameters of the content inversion network and the second decoder.

5. The pronunciation inverse method of the variational autoencoder integrating multi-task learning according to claim 4, characterized in that, The loss function of the content inverse network is constructed based on content feature loss and pronunciation motion reconstruction loss.

6. The pronunciation inverse method of the variational autoencoder integrating multi-task learning according to claim 1, characterized in that, The content reverse engineering network consists of a three-layer bidirectional long short-term memory network and a two-layer fully connected neural network.

7. The pronunciation inverse method of the variational autoencoder integrating multi-task learning according to claim 1, characterized in that, The acoustic domain variational autoencoder further includes an acoustic feature extractor for extracting acoustic features from the speech signal; the first content encoder and the speaker encoder extract first content features and first speaker features based on the acoustic features, respectively.

8. The pronunciation inverse method of the variational autoencoder integrating multi-task learning according to claim 1, characterized in that, The articulation domain variational autoencoder further includes an acoustic feature extractor and an articulation feature extractor. The acoustic feature extractor is used to extract acoustic features from the speech signal, and the articulation feature extractor is used to extract articulation features from the articulation motion data. The speaker encoder extracts a second speaker feature based on the acoustic features, and the second content encoder extracts a second content feature based on the articulation features.

9. A pronunciation inverse device integrating a variational autoencoder with multi-task learning, characterized in that, include: Data acquisition module: used to acquire synchronized speech signals and vocalization motion data; Acoustic feature reconstruction module: used to train an acoustic domain variational autoencoder based on the speech signal, wherein the acoustic domain variational autoencoder includes a first content encoder, a speaker encoder, a first speech recognizer, a speaker recognizer, and a first decoder. The first content encoder and the speaker encoder are respectively used to extract a first content feature and a first speaker feature from the speech signal, input the first content feature into the first speech recognizer to recognize the speech content, and input the first speaker feature into the speaker recognizer to recognize the speaker; finally, the first content feature and the first speaker feature are input into the first decoder to reconstruct the acoustic features. The pronunciation feature reconstruction module is used to train a pronunciation domain variational autoencoder based on the pronunciation motion data. The pronunciation domain variational autoencoder includes a speaker encoder, a second content encoder, a second speech recognizer, and a second decoder. The speaker encoder extracts second speaker features from the speech signal, the second content encoder extracts second content features from the pronunciation motion data, and inputs the second content features into the second speech recognizer to recognize speech content. Finally, the second content features and the second speaker features are input into the second decoder to reconstruct pronunciation features. The pronunciation trajectory generation module is used to construct a content inverse network and, based on the first content encoder, the first speaker encoder, the content inverse network, and the second decoder, construct a pronunciation inverse system to synthesize pronunciation motion trajectories.

10. The pronunciation inverse device of the variational autoencoder integrating multi-task learning according to claim 9, characterized in that, The content inverse network is composed of a three-layer bidirectional long short-term memory network and a two-layer fully connected neural network; the loss function of the content inverse network includes content feature loss and pronunciation motion reconstruction loss, so as to optimize the parameters of the content inverse network and the second decoder respectively.

Images