Speech generation methods based on phonetics and semantics, and training methods for speech generation models.

Abstract

The application provides a phonetic symbol and semantic-based speech generation method, a phonetic symbol and semantic-based speech generation model training method and device, electronic equipment and a computer readable storage medium. In the embodiment of the application, the semantic feature matrix in the text data and the guide information with phonetic symbol data features are fused to obtain a fusion feature matrix with semantic information and phonetic symbol information, and the fusion feature matrix is input into a speech generation model to obtain speech data corresponding to the text data. Since the phonetic symbol and semantic information are fused in the input information, the speech corresponding to the phonetic symbol is adjusted according to different semantics, so that the speech is no longer averaged, and the mechanical feeling is reduced. The phonetic symbol and semantic-based speech generation method of the embodiment of the application fully utilizes the information contained in the input text, without the need to increase additional input data, and can generate more realistic speech data, improving the speech generation effect.

Description

Technical Field

[0001] This application relates to the field of speech generation, specifically to a speech generation method based on phonetics and semantics, a speech generation model training method based on phonetics and semantics, an apparatus, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Voice generation is used in all aspects of daily life. For example, the text-to-speech function requires generating corresponding speech from text data.

[0003] In existing technologies, text data is typically input into a speech generation model to obtain the speech corresponding to the text data.

[0004] The speech obtained from text data lacks emotional variation, the pronunciation is too uniform, it has a heavy mechanical feel, and it is not natural enough. Summary of the Invention

[0005] This application provides a speech generation method based on phonetics and semantics to solve the problem of unnatural speech pronunciation in prior art.

[0006] Accordingly, embodiments of this application also provide a method, apparatus, electronic device, and computer-readable storage medium for training a speech generation model based on phonetics and semantics, to ensure the implementation and application of the above method.

[0007] In a first aspect, embodiments of this application provide a speech generation method based on phonetic symbols and semantics, the method comprising:

[0008] Input the text data into the semantic model to obtain the semantic feature matrix of the text data;

[0009] The text data is translated into phonetic data, and the phonetic data is input into the bypass guidance module of the text encoding model. Each coupling layer in the bypass guidance module outputs guidance information; the guidance information is used to fuse with the semantic feature matrix.

[0010] The semantic feature matrix is ​​input into the text encoding model, and the guidance information of each coupling layer in the bypass guidance module is sequentially input into each convolutional layer of the text encoding model. The fused feature matrix is ​​output from the text encoding model; the coupling layer and the convolutional layer correspond one-to-one.

[0011] The fused feature matrix is ​​input into the speech generation model to obtain speech data corresponding to the text data.

[0012] Secondly, embodiments of this application provide a method for training a speech generation model based on phonetics and semantics, the method comprising:

[0013] Semantic sample information and phonetic symbol sample information are extracted from text sample data, and the semantic sample information and the phonetic symbol sample information are input into a text encoding model to obtain fused sample feature data;

[0014] Input the speech sample data corresponding to the text sample data into the speech coding model to obtain the speech sample feature matrix;

[0015] By calculating the mean and variance of the fused sample feature matrix and the speech sample feature matrix, the text sample distribution function and the speech sample distribution function are obtained.

[0016] The initial model is trained using the speech sample distribution function, and the loss value is determined based on the difference between the text sample distribution function and the speech sample distribution function.

[0017] Based on the loss value and the preset loss function, the initial model is adjusted to obtain a speech generation model, which is used by the speech generation method based on phonetics and semantics.

[0018] Thirdly, embodiments of this application provide a speech generation apparatus based on phonetic symbols and semantics, the apparatus comprising:

[0019] The semantic module is used to input text data into the semantic model to obtain the semantic feature matrix of the text data;

[0020] The phonetic symbol module is used to translate the text data into phonetic symbol data, and input the phonetic symbol data into the bypass guidance module of the text encoding model. The module outputs guidance information from each coupling layer in the bypass guidance module. The guidance information is used to fuse with the semantic feature matrix.

[0021] The fusion module is used to input the semantic feature matrix into the text encoding model, and to sequentially input the guidance information of each coupling layer in the bypass guidance module into each convolutional layer of the text encoding model, and output the fused feature matrix from the text encoding model; the coupling layer and the convolutional layer correspond one-to-one.

[0022] The generation module is used to input the fused feature matrix into the speech generation model to obtain speech data corresponding to the text data.

[0023] Fourthly, embodiments of this application provide a speech generation model training device based on phonetics and semantics, the device comprising:

[0024] The sample fusion module is used to extract semantic sample information and phonetic sample information from text sample data, and input the semantic sample information and the phonetic sample information into the text encoding model to obtain fused sample feature data;

[0025] The sample speech module is used to input the speech sample data corresponding to the text sample data into the speech coding model to obtain the speech sample feature matrix.

[0026] The distribution function module is used to obtain the text sample distribution function and the speech sample distribution function by calculating the mean and variance of the fused sample feature matrix and the speech sample feature matrix, respectively.

[0027] The loss value module is used to train an initial model using the speech sample distribution function and determine the loss value based on the difference between the text sample distribution function and the speech sample distribution function.

[0028] The training module is used to adjust the initial model according to the loss value and the preset loss function to obtain a speech generation model, which is used by the speech generation method based on phonetics and semantics.

[0029] Fifthly, embodiments of this application provide an electronic device, including a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of one or more methods described in embodiments of this application.

[0030] Sixthly, embodiments of this application provide a readable storage medium that, when instructions in the readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform one or more methods as described in embodiments of this application.

[0031] In this embodiment, the semantic feature matrix in the text data and the guiding information with phonetic data features are fused to obtain a fused feature matrix with semantic and phonetic information. The fused feature matrix is ​​then input into the speech generation model to obtain the speech data corresponding to the text data. Since the input information incorporates phonetic and semantic information, the speech corresponding to the phonetic symbols is adjusted according to different semantics, so that the speech is no longer uniform and the mechanical feeling is reduced. The speech generation method based on phonetic symbols and semantics in this embodiment makes full use of the information contained in the input text and can generate more realistic speech data without adding additional input data, thus improving the effect of speech generation. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a flowchart illustrating the steps of a speech generation method based on phonetics and semantics provided in an embodiment of this application.

[0034] Figure 2 This is a schematic diagram illustrating the generation of a fusion feature matrix provided in an embodiment of this application;

[0035] Figure 3 This is a flowchart illustrating the steps of another speech generation method based on phonetics and semantics provided in an embodiment of this application.

[0036] Figure 4 This is a schematic diagram of a coupling layer partition provided in an embodiment of this application;

[0037] Figure 5 This is a flowchart illustrating the steps of a speech generation model training method based on phonetics and semantics provided in an embodiment of this application.

[0038] Figure 6 This is an architecture diagram of a speech generation model training method based on phonetics and semantics provided in an embodiment of this application;

[0039] Figure 7 This is a structural diagram of a speech generation device based on phonetics and semantics provided in an embodiment of this application;

[0040] Figure 8 This is a structural diagram of a speech generation model training device based on phonetics and semantics provided in an embodiment of this application;

[0041] Figure 9 This is a structural block diagram of an electronic device provided in an embodiment of this application;

[0042] Figure 10 This is a structural block diagram of another electronic device provided in the embodiments of this application. Detailed Implementation

[0043] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0044] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and are not limited in number; for example, a first object can be one or more. Furthermore, the term "and / or" in the specification and claims is used to describe the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. In the embodiments of this application, the term "multiple" refers to two or more, and other quantifiers are similar.

[0045] like Figure 1 As shown in the figure, this application provides a flowchart of the steps of a speech generation method based on phonetics and semantics, as follows: Figure 1 As shown, the method may include:

[0046] Step 101: Input the text data into the semantic model to obtain the semantic feature matrix of the text data.

[0047] In the process of speech generation, target information is typically input into a speech generation model in text data format to obtain speech data corresponding to the text data. When inputting text data into the speech generation model, some related technologies often use the phonetic symbols contained in the text data to generate the corresponding speech data. For Chinese, the phonetic symbols are the pinyin. Phonetic symbols often indicate the most standard pronunciation. However, in daily life, people's pronunciation varies slightly depending on the meaning, with subtle changes in speech rate, pitch, stress, and timbre. Standard pronunciation is rarely used for communication. Therefore, speech data generated using phonetic symbols is too standardized, has a strong mechanical feel, and lacks a natural quality.

[0048] In this embodiment, a semantic feature matrix is ​​obtained by inputting text data into a semantic model. The semantic feature matrix contains the semantic features of the text data extracted by the semantic model. These semantic features enable the same text data to be refined into richer categories based on different semantic information, preventing the speech data corresponding to the text data from being too singular and affecting the speech generation effect.

[0049] For example, a third-party semantic model (BERT, Bidirectional Encoder Representations from Transformers) can be used to generate a semantic feature matrix, and the size of the semantic feature matrix can be set by oneself.

[0050] Step 102: Translate the text data into phonetic data, input the phonetic data into the bypass guiding module of the text encoding model, and output each coupling layer in the bypass guiding module to obtain guiding information.

[0051] In the embodiment of the present application, by translating the text data into phonetic data, generating data with the initials, finals and tone information of the text data, the information in the text data is fully utilized, and the phonetic data is input into the bypass guiding module of the text encoding model. The data of each layer in the module is used as the guiding information of the bypass, and is fused with the semantic feature matrix as the input information at the same time, preparing for generating speech data with semantic and phonetic information.

[0052] In some embodiments, a third-party phonetic annotation tool, such as PyPinyin, can be used to translate the text data into phonetic data.

[0053] For example, for the text data "good person", after translation, phonetic data such as "h3 ao3 r2 en2" will be obtained, where the initials and finals are separated, and the pronunciation of the combined characters is marked with numbers at the back, and different numbers represent corresponding tones.

[0054] Step 103: Input the semantic feature matrix into the text encoding model, and sequentially input the guiding information of each coupling layer in the bypass guiding module into each convolutional layer of the text encoding model, and output a fusion feature matrix from the text encoding model.

[0055] In the embodiment of the present application, the semantic feature matrix is input into the text encoding model, and the guiding information of each coupling layer in the bypass guiding module is sequentially input into each convolutional layer of the text encoding layer, so that when the semantic feature matrix executes the mapping relationship of each convolutional layer of the text encoding model, there is corresponding guiding information of the coupling layer for fusion, making the semantic feature matrix and the guiding information with phonetic data features more stable in fusion, which is beneficial to the stability of the generation result of the speech generation model.

[0056] Figure 2The diagram illustrates the generation of the fused feature matrix. Phonetic transcription data is input into the bypass guidance module and mapped sequentially through the coupling layers. The output of each coupling layer is used as guidance information input to each convolutional layer in the text encoding model. The semantic feature matrix is ​​input into the text encoding model. Before being input into each convolutional layer, the semantic feature matrix is ​​fused with the guidance data in the corresponding coupling layer. The guidance data contains features from the phonetic transcription data; therefore, the fused semantic and phonetic features can be used as input to the convolutional layer for mapping. The output data is then fused with the guidance information of the coupling layer corresponding to the next convolutional layer. This fused information is used as input to the next convolutional layer until the text encoding model completes its operation, yielding the fused feature matrix.

[0057] In this module, the coupling layer in the bypass guidance module corresponds one-to-one with the convolutional layer in the text encoding model. When the text encoding model has NLAYERS convolutional layers, the bypass guidance module has NLAYERS coupling layers.

[0058] For example, when the phonetic transcription data is "h3 ao3 r2 en2", the guidance data in each coupling layer of the bypass guidance module is a 4×D matrix, where D is an internal dimension that can be set by the user, such as guidance data with a size of 4×192. Correspondingly, the semantic feature matrix can be set to a 4×192 matrix, and the resulting fused feature matrix is ​​also a 4×192 matrix.

[0059] Step 104: Input the fused feature matrix into the speech generation model to obtain speech data corresponding to the text data.

[0060] In this embodiment, the fusion feature matrix contains both phonetic and semantic features of the text data, and is input into the speech generation model. Since the speech generation model also uses the method described above to fuse sample data, generate a sample fusion feature matrix, and trains the speech generation model using the sample fusion feature matrix and corresponding speech data, the speech generation model can recognize fusion feature matrices with rich semantic and phonetic information, calculate the speech feature matrix with the highest probability corresponding to the fusion feature matrix, and obtain speech data by decoding the speech feature matrix. Therefore, speech data can be categorized using rich semantic and phonetic information, corresponding to rich speech data, enabling subtle changes in pronunciation based on semantics in daily life, making the generated speech more natural.

[0061] For example, when the text data is "You are such a good person," in some related technologies, the pronunciation of the speech data generated solely from phonetic data is too uniform. Without emphasis, the sentence not only sounds unnatural but also makes the listener focus on "good person," associating it with whether "you" have done some good deeds to earn the praise of "good person." However, when the actual text data is semantically enhanced by emphasizing the word "really," "You are such a good person" conveys a sense of gratitude, making it not only more vivid but also more accurately expressing the speaker's meaning, thus improving the quality of the generated speech.

[0062] In summary, in this embodiment, by fusing the semantic feature matrix in the text data with the guiding information having phonetic data features, a fused feature matrix with semantic and phonetic information is obtained. The fused feature matrix is ​​then input into the speech generation model to obtain the speech data corresponding to the text data. Since the input information incorporates phonetic and semantic information, the speech corresponding to the phonetic symbols is adjusted according to different semantics, making the speech less uniform and reducing the mechanical feel. The speech generation method based on phonetic symbols and semantics in this embodiment makes full use of the information contained in the input text and can generate more realistic speech data without adding additional input data, thus improving the effect of speech generation.

[0063] like Figure 3 As shown in the figure, this application proposes a flowchart of a speech generation method based on phonetics and semantics, as follows: Figure 3 As shown, the method may include:

[0064] Step 201: Input the text data into the semantic model to obtain the semantic feature matrix of the text data.

[0065] For details, please refer to step 101 above; it will not be repeated here.

[0066] Step 202: Translate the text data into phonetic data, input the phonetic data into the bypass guidance module of the text encoding model, and output each coupling layer in the bypass guidance module to obtain guidance information.

[0067] For details, please refer to step 102 above; it will not be repeated here.

[0068] Optionally, step 202 may specifically include:

[0069] Sub-step 2021 involves segmenting the text data into phrases and, based on a preset phonetic symbol library, finding the phonetic symbols mapped to the segmented phrases to obtain phonetic symbol data.

[0070] The phonetic symbol library is a data storage structure that stores corresponding phrases and phonetic symbols. By inputting a phrase into the phonetic symbol library, the corresponding phonetic symbol data can be found.

[0071] In an embodiment of the present application, by performing phrase segmentation on text data, data in units of phrases is obtained, making the phonetic symbol data corresponding to polyphonic characters appearing in the phrases more explicit, and reducing the situation where the generated speech data is incorrect due to the presence of polyphonic characters in the text data.

[0072] For example, when the text data is "你真是好人", the results obtained after segmenting the text data are "你", "真是", "好人". Due to the specific pronunciation of "好人", the mapped phonetic symbol data found in the phonetic symbol library is "h3ao3 r2 en2" instead of "h4 ao4 r2 en2".

[0073] Sub-step 2022: By searching a preset vector codebook, find the corresponding vector for the phonetic symbol data to obtain a phonetic symbol vector matrix.

[0074] Among them, the vector codebook is a data set marked with the vectors corresponding to each phonetic symbol data. According to the vectors corresponding to each phonetic symbol in the phonetic symbol data in the vector codebook, a phonetic symbol vector matrix is obtained. The vectors in the vector codebook are empirical data, aiming to extract more features possessed by the phonetic symbol data and can be custom-set.

[0075] For example, there are a total of 70 initials and finals in Mandarin Chinese, and there are 5 tones. Among them, the light tone can be represented by 0, and the first to fourth tones are represented by 1, 2, 3, and 4 in sequence, that is, there are a total of 70×5 units. Then the dimension of the preset vector codebook is 350×D, where D is the internal dimension and can be set by oneself. For the phonetic symbol data "h3 ao3 r2 en2", find the vectors corresponding to the four units respectively, and the concatenated phonetic symbol vector matrix is a 4×D-dimensional matrix.

[0076] Sub-step 2023: Input the phonetic symbol vector matrix into the bypass guidance module.

[0077] In an embodiment of the present application, input the phonetic symbol vector matrix into the bypass guidance module, so that each coupling layer of the bypass guidance module can fuse the corresponding guidance information with the input of each convolutional layer of the text encoding model.

[0078] For example, the phonetic symbol vector matrix of the phonetic symbol data "h3 ao3 r2 en2" is a 4×D-dimensional matrix, each layer of guidance information is also a 4×D-dimensional matrix, the size of the semantic feature matrix can also be set to 4×D, and the fused feature matrix obtained after fusion is a 4×D-dimensional matrix, where D is the self-set internal dimension.

[0079] Optionally, each coupling layer of the bypass guidance module is divided into a first region and a second region. Step 202 may specifically include:

[0080] Sub-step 2024: For any coupling layer in the bypass guidance model, and the adjacent coupling layer of any coupling layer, when the phonetic data of the first region of any coupling layer is not changed and the phonetic data of the second region is changed, the phonetic data of the first region of the adjacent coupling layer is changed and the phonetic data of the second region is not changed; or when the phonetic data of the first region of any coupling layer is changed and the phonetic data of the second region is not changed, the phonetic data of the first region of the adjacent coupling layer is not changed and the phonetic data of the second region is changed.

[0081] In this embodiment, the size of the phonetic symbol vector matrix input in the bypass guidance module can be used as the size of the coupling layer. Figure 4 This refers to the partitioning method of the coupling layer, where the coupling layer has D rows. By choosing a constant d smaller than D, the coupling layer can be divided into two regions, as shown in the figure. The first region is Z1-Z2. d The second region is Z. d+1 -Z D .

[0082] Specifically, when the phonetic symbol vector matrix passes through the coupling layer of the bypass guidance module, only one region of the two regions in the coupling layer undergoes mapping transformation, while the other region retains its original data. Furthermore, in the next coupling layer, such as... Figure 2 The dimension flipping is performed as shown, meaning that the regions transformed in coupling layer 1 are not transformed in coupling layer 2, while the regions that are not transformed in coupling layer 1 are transformed in coupling layer 2. This flipping is also performed in adjacent coupling layers until all coupling layers have been executed.

[0083] like Figure 2 As shown, the function of the bypass guidance module is to map the input phonetic symbol vector matrix into a normal distribution matrix. Finally, the self-supervised loss value output by the bypass guidance module is the difference between the matrix output by the bypass guidance module and the normal distribution matrix.

[0084] Sub-step 2025: After the phonetic data in each coupling layer has been transformed, the guidance information for each layer is obtained.

[0085] In this embodiment, the data in each coupling layer of the bypass guidance model are output separately to obtain the guidance information of each layer of phonetic data.

[0086] In one embodiment, the coupling layer in the bypass guidance module has the same number of layers as the convolutional layer in the text encoding model, and both are even-numbered layers.

[0087] In this embodiment, by setting the number of coupling layers in the bypass guidance module to an even number, both parts of the phonetic symbol vector matrix undergo the same number of transformations, reducing the error caused by different number of transformations. At the same time, setting the number of coupling layers to be the same as the number of convolutional layers allows the guidance information and the input information of the convolutional layer to be fused in a one-to-one correspondence.

[0088] Step 203: When data is about to be input into any convolutional layer of the text encoding model, the guiding information of each pair of corresponding coupling layers and the first data of the convolutional layer are fused to obtain fused data.

[0089] The first data is the data passed from the upper convolutional layer to the lower convolutional layer, which is the output data of the upper convolutional layer.

[0090] In the embodiments of this application, such as Figure 2 As shown, when convolutional layer 2 is about to input data, the guiding information of the corresponding coupling layer 2 and the first data of convolutional layer 2 are fused to obtain fused data. The first data for convolutional layer 2 is the data output by convolutional layer 1, and the first data for convolutional layer 1 is the semantic feature matrix.

[0091] Optionally, step 203 may specifically include:

[0092] Sub-step 2031 combines the guidance information of each corresponding coupling layer with the first data of the current convolutional layer in the following manner:

[0093]

[0094] Where O is the fused data, I is the first data of the convolutional layer, C is the guiding information of the coupling layer, mean(I) is the mean of I, std(I) is the standard deviation of I, W() and B() are the two fully connected layers of the text encoding model, W(C) represents the product of the guiding information and a fully connected layer, and B(C) represents the product of the guiding information and another fully connected layer.

[0095] In the embodiments of this application, the guidance information of each pair of corresponding coupling layers and the first data of the current convolutional layer are combined in the manner described above, so that the semantic feature matrix and the guidance information with phonetic data features are more stably integrated, which is conducive to the stability of the speech generation model generation results.

[0096] Step 204: Use the fused data as input data to the convolutional layer, and output the second data from the convolutional layer.

[0097] In the embodiments of this application, the second data is the output data after the input fusion data of the convolutional layer. For the convolutional layer NLAYERS, the second data is the feature fusion matrix.

[0098] like Figure 2 As shown, the second data output by convolutional layer 1 and the guiding data output by coupling layer 2 are fused together to form the input data of convolutional layer 2, and the second data of convolutional layer 1 is the first data of convolutional layer 2.

[0099] Step 205: After all convolutional layers output sequentially, the feature fusion matrix is ​​obtained.

[0100] In this embodiment, by fusing data between the convolutional layer of the text encoding model and the coupling layer of the bypass guidance module, the semantic information in the semantic feature matrix is ​​fused with the phonetic information in the phonetic vector matrix. This allows the information contained in the input text data to be fully utilized. In the speech generation model, the speech data with the highest probability corresponding to the semantic information and phonetic information is calculated. Since the speech data is mapped from data with semantic information and phonetic information, the speech data is a pronunciation with corresponding adjustments that includes semantic information and phonetic information. Therefore, the obtained pronunciation data is no longer averaged and is more natural.

[0101] Step 206: Input the fused feature matrix into the speech generation model to obtain speech data corresponding to the text data.

[0102] This step can be referred to step 104 above, and will not be repeated here.

[0103] In summary, in this embodiment, by fusing the semantic feature matrix in the text data with the guiding information containing phonetic data features, a fused feature matrix with semantic and phonetic information is obtained. This fused feature matrix is ​​then input into the speech generation model to obtain the speech data corresponding to the text data. Since the input information incorporates both phonetic and semantic information, the speech corresponding to the phonetic symbols is adjusted according to different semantics, preventing the speech from becoming uniform and reducing its mechanical feel. Furthermore, by fusing the data between the one-to-one coupling layers and convolutional layers in a specific manner, the speech generation model's output is stabilized. The phonetic and semantic-based speech generation method in this embodiment fully utilizes the information contained in the input text, generating more realistic speech data stably without adding additional input data, thus improving the speech generation effect.

[0104] Figure 5 This is a flowchart illustrating the steps of a speech generation model training method based on phonetics and semantics, as provided in an embodiment of this application. Figure 5 As shown, the method may include:

[0105] Step 301, extract the semantic sample information and phonetic symbol sample information from the text sample data, and input the semantic sample information and the phonetic symbol sample information into a text encoding model to obtain fused sample feature data.

[0106] Among them, the semantic sample information is obtained by inputting the text sample data into a semantic model, and the phonetic symbol sample information is obtained by translating the text sample data.

[0107] For example, if the text sample data is "你好" (Hello), then the phonetic symbol sample information is "n3 i3 h3 ao3".

[0108] In the embodiments of the present application, by adding semantic sample information and phonetic symbol sample information to the input text information, subtle differences can be recognized in the input text data, and a mapping relationship is constructed between the input data with these subtle differences and the speech data, so that the speech generated by the speech generation model also has subtle differences, and the pronunciation is no longer averaged, making the generated speech more natural.

[0109] Optionally, step 301 may specifically include:

[0110] Sub-step 3011, input the text sample data into a semantic model to obtain the sample semantic feature matrix of the text sample data.

[0111] Sub-step 3012, translate the text sample data into phonetic symbol sample data, and input the phonetic symbol sample data into the bypass guiding module of the text encoding model, and output each coupling layer in the bypass guiding module to obtain sample guiding information.

[0112] Sub-step 3013, input the sample semantic feature matrix into the text encoding model, and sequentially input the sample guiding information of each coupling layer in the bypass guiding module into each convolutional layer of the text encoding model, and output a fused sample feature matrix from the text encoding model.

[0113] For sub-steps 3011 - 3013, as Figure 6 shown, by inputting the text sample data into a semantic model, a sample semantic feature matrix is obtained, and the text sample data is used to obtain phonetic symbol sample data by looking up a vector codebook, and the phonetic symbol sample data is input into the bypass guiding module of the text encoding model, where the bypass guiding module can implement the function of mapping the phonetic symbol sample data into a normal distribution. Finally, the sample semantic feature matrix is input into the text encoding model, and the sample guiding information of each coupling layer in the bypass guiding module is sequentially input into each convolutional layer of the text encoding model, and fusion is performed according to the fusion data generation formula in sub-step 2031, and finally a fused sample feature matrix is output from the text encoding model.

[0114] In some embodiments, the bypass guidance module eventually outputs a self-supervised loss value, which is used to record the difference between the data output by the last coupling layer of the bypass guidance module and the normally distributed data.

[0115] Step 302: Input the speech sample data corresponding to the text sample data into the speech coding model to obtain the speech sample feature matrix.

[0116] The length of each sentence in the text sample data and its corresponding speech sample data does not exceed 20 seconds, because sentences longer than 20 seconds are rare in daily communication. Therefore, by controlling the sample data, the impact of long sentences with low meaning on the model is reduced.

[0117] In this embodiment of the application, the linear spectrum of the speech sample data is extracted by the speech coding model to obtain the speech sample feature matrix, and the speech sample data can be reconstructed by the linear spectrum information in the speech sample feature matrix.

[0118] In some embodiments, the linear spectrum of speech can be extracted using third-party signal processing tools, such as the torchaudio tool in the PyTorch neural network framework.

[0119] For example, if the recording length of "Good Person" is 1 second, and the setting for extracting the linear spectrum is "frame shift 10 milliseconds", then the size of the obtained linear spectrum is 100×D, where D is the internal dimension set by the user.

[0120] Step 303: By calculating the mean and variance of the fused sample feature matrix and the speech sample feature matrix, the text sample distribution function and the speech sample distribution function are obtained.

[0121] In this embodiment of the application, the mean μ of the speech sample feature matrix is ​​calculated. Q and variance σ Q The speech sample distribution function is obtained, and the mean μ of the fused sample feature matrix is ​​calculated. θ and variance σ θ This yields the text sample distribution function.

[0122] In some embodiments, the speech sample distribution function is randomly sampled before training the initial model. A random number rand is randomly selected, and the random sampling result Z of the speech sample distribution function is calculated, where Z = μ. Q +rand×σ Q As mentioned in the previous example, when the size of the linear spectrum is 100×D, the dimension of random sampling is also 100×D. Since speech data has strong randomness, random sampling is used to enhance the robustness of the trained speech generation model, where D is a self-set internal dimension.

[0123] In some embodiments, such as Figure 6 As shown, the distribution function of randomly sampled speech samples is input into the decoder to obtain the speech waveform. The speech waveform is compared with the speech sample data to obtain the generation loss value. The generation loss value is used to record the loss value of the speech sample data after generating the speech sample distribution function.

[0124] Step 304: Train an initial model using the speech sample distribution function, and determine the loss value based on the difference between the text sample distribution function and the speech sample distribution function;

[0125] In some embodiments, an initial model is trained using a speech sample distribution function, wherein the initial model has a generating function f for mapping the speech sample distribution function to a normal distribution. θ The initial model training result is f θ (Z), by f θ (Z) and μ θ and σ θ Performing timing alignment means aligning the pronunciation duration f of each syllable. θ (Z), μ θ and σ θ The corresponding number of copies of the phonetic symbol data can be made to form the corresponding number of copies of f. θ (Z) Same dimension, continuing from the previous example, μ θ and σ θ There are 4 phonetic symbols, and f θ The dimension of (Z) is 100, so the same dimension is maintained by copying, for example [1,1,1,1,1,1,2,2,2,3,3,3,3…4,4,4,4,4,4], and the number of copies is recorded. Figure 6 As shown, the fused sample feature matrix is ​​input into the duration prediction model to obtain the predicted duration, which is the number of replications given by the duration prediction model. The difference between the two replication counts is recorded as the duration loss.

[0126] In some embodiments, the relative distance (KL, Kullback-Leibler Divergence) between the text sample distribution function and the speech sample distribution function is used as one of the loss values, and the calculation formula is as follows:

[0127]

[0128] Where D is the KL distance, σ Q Let μ be the variance of the speech sample feature matrix. θ σ is the mean of the fused sample feature matrix. θ To fuse the variance of the sample feature matrix, f θ (Z) represents the result generated from the initial model.

[0129] Step 305: Adjust the initial model according to the loss value and the preset loss function to obtain a speech generation model, which is used by the speech generation method based on phonetics and semantics.

[0130] In some embodiments, the self-supervised loss value in sub-step 3012, the generation loss value in step 303, the KL distance and the market loss value in step 304 are added together as the result of the loss function. The model is then adjusted using the result of the loss function to finally obtain the speech generation model.

[0131] In summary, in this embodiment, by fusing the semantic feature matrix in the text data with the guiding information having phonetic data features, a fused feature matrix with semantic and phonetic information is obtained. The fused feature matrix is ​​then input into the speech generation model to obtain the speech data corresponding to the text data. Since the input information incorporates phonetic and semantic information, the speech corresponding to the phonetic symbols is adjusted according to different semantics, making the speech less uniform and reducing the mechanical feel. The speech generation method based on phonetic symbols and semantics in this embodiment fully utilizes the information contained in the input text and can generate more realistic speech data without adding additional input data, thus improving the effect of speech generation.

[0132] Figure 7 This application provides a speech generation device based on phonetics and semantics, such as... Figure 7 As shown, the device may include:

[0133] Semantic module 401 is used to input text data into a semantic model to obtain a semantic feature matrix of the text data;

[0134] The phonetic symbol module 402 is used to translate the text data into phonetic symbol data, input the phonetic symbol data into the bypass guidance module of the text encoding model, and output each coupling layer in the bypass guidance module to obtain guidance information; the guidance information is used to fuse with the semantic feature matrix.

[0135] The fusion module 403 is used to input the semantic feature matrix into the text encoding model, and to sequentially input the guidance information of each coupling layer in the bypass guidance module into each convolutional layer of the text encoding model, and output the fused feature matrix from the text encoding model; the coupling layer and the convolutional layer correspond one-to-one.

[0136] The generation module 404 is used to input the fused feature matrix into the speech generation model to obtain speech data corresponding to the text data.

[0137] Optionally, the phonetic symbol module 402 may specifically include:

[0138] The phonetic symbol submodule is used to segment the text data into phrases and, based on a preset phonetic symbol library, to find the phonetic symbols mapped to the segmented phrases to obtain phonetic symbol data; the phonetic symbol library stores the phonetic symbols mapped to the phrases.

[0139] The initial vector submodule is used to find the corresponding vector for the phonetic symbol data by searching a preset vector codebook, so as to obtain a phonetic symbol vector matrix; the vector codebook is marked with the vector corresponding to each phonetic symbol data.

[0140] The input submodule is used to input the phonetic symbol vector matrix into the bypass guidance module.

[0141] Optionally, the phonetic symbol module 402 may specifically include:

[0142] The dimension flipping submodule is used to, for any coupling layer in the bypass guidance model and the adjacent coupling layer of any coupling layer, keep the phonetic data of the first region of the coupling layer unchanged and the phonetic data of the second region changed, while keeping the phonetic data of the first region of the adjacent coupling layer unchanged; or keep the phonetic data of the first region of the coupling layer changed and the phonetic data of the second region changed, while keeping the phonetic data of the second region of the coupling layer unchanged.

[0143] The guidance information submodule is used to obtain the guidance information for each layer after the phonetic data in each coupling layer has been transformed.

[0144] Optionally, the fusion module 403 may specifically include:

[0145] The fusion submodule is used to fuse the guidance information of each pair of corresponding coupling layers and the first data of the convolutional layer when data is about to be input into any convolutional layer of the text encoding model, so as to obtain fused data; for the first convolutional layer, the first data is the semantic feature matrix.

[0146] The sequential execution submodule is used to take the fused data as input data to the convolutional layer and output second data from the convolutional layer; the second data is the first data of the next convolutional layer, and for the last convolutional layer, the second data is the feature fusion matrix;

[0147] The output submodule is used to obtain the feature fusion matrix after all convolutional layers have output their values ​​sequentially.

[0148] Optionally, the fusion submodule may specifically include:

[0149] The fusion unit is used to combine the guidance information of each corresponding coupling layer with the first data of the current convolutional layer in the following manner:

[0150]

[0151] Where O is the fused data, I is the first data of the convolutional layer, C is the guiding information of the coupling layer, mean(I) is the mean of I, std(I) is the standard deviation of I, W() and B() are the two fully connected layers of the text encoding model, W(C) represents the product of the guiding information and a fully connected layer, and B(C) represents the product of the guiding information and another fully connected layer.

[0152] In summary, in this embodiment, by fusing the semantic feature matrix in the text data with the guiding information having phonetic data features, a fused feature matrix with semantic and phonetic information is obtained. The fused feature matrix is ​​then input into the speech generation model to obtain the speech data corresponding to the text data. Since the input information incorporates phonetic and semantic information, the speech corresponding to the phonetic symbols is adjusted according to different semantics, making the speech less uniform and reducing the mechanical feel. The speech generation method based on phonetic symbols and semantics in this embodiment fully utilizes the information contained in the input text and can generate more realistic speech data without adding additional input data, thus improving the effect of speech generation.

[0153] Figure 8 This application provides a speech generation model training device based on phonetics and semantics, such as... Figure 8 As shown, the device may include:

[0154] The sample fusion module 501 is used to extract semantic sample information and phonetic sample information from text sample data, and input the semantic sample information and the phonetic sample information into the text encoding model to obtain fused sample feature data;

[0155] The sample speech module 502 is used to input the speech sample data corresponding to the text sample data into the speech coding model to obtain the speech sample feature matrix.

[0156] The distribution function module 503 is used to obtain the text sample distribution function and the speech sample distribution function by calculating the mean and variance of the fused sample feature matrix and the speech sample feature matrix, respectively.

[0157] The loss value module 504 is used to train an initial model using the speech sample distribution function and determine the loss value based on the difference between the text sample distribution function and the speech sample distribution function.

[0158] The training module 505 is used to adjust the initial model according to the loss value and the preset loss function to obtain a speech generation model, which is used by the speech generation method based on phonetics and semantics.

[0159] Optionally, the sample fusion module 501 may specifically include:

[0160] The sample semantic submodule is used to input the text sample data into the semantic model to obtain the sample semantic feature matrix of the text sample data.

[0161] The sample guidance information submodule is used to translate the text sample data into phonetic symbol sample data, input the phonetic symbol sample data into the bypass guidance module of the text encoding model, and output each coupling layer in the bypass guidance module to obtain sample guidance information;

[0162] The sample fusion submodule is used to input the sample semantic feature matrix into the text encoding model, and to sequentially input the sample guidance information of each coupling layer in the bypass guidance module into each convolutional layer of the text encoding model, and output the fused sample feature matrix from the text encoding model.

[0163] In summary, in this embodiment, by fusing the semantic feature matrix in the text data with the guiding information having phonetic data features, a fused feature matrix with semantic and phonetic information is obtained. The fused feature matrix is ​​then input into the speech generation model to obtain the speech data corresponding to the text data. Since the input information incorporates phonetic and semantic information, the speech corresponding to the phonetic symbols is adjusted according to different semantics, making the speech less uniform and reducing the mechanical feel. The speech generation method based on phonetic symbols and semantics in this embodiment fully utilizes the information contained in the input text and can generate more realistic speech data without adding additional input data, thus improving the effect of speech generation.

[0164] See Figure 9 The electronic device 400 may include one or more of the following components: processing component 402, memory 404, power supply component 406, multimedia component 408, audio component 410, input / output (I / O) interface 412, sensor component 414, and communication component 416.

[0165] Processing component 402 typically controls the overall operation of electronic device 400, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 402 may include one or more processors 420 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 402 may include one or more modules to facilitate interaction between processing component 402 and other components. For example, processing component 402 may include a multimedia module to facilitate interaction between multimedia component 408 and processing component 402.

[0166] Memory 404 is used to store various types of data to support the operation of electronic device 400. Examples of this data include instructions for any application or method operating on electronic device 400, contact data, phonebook data, messages, pictures, multimedia, etc. Memory 404 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0167] Power supply component 406 provides power to various components of electronic device 400. Power supply component 406 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 400.

[0168] Multimedia component 408 includes an interface that provides an output interface between electronic device 400 and a user. In some embodiments, the interface may include a liquid crystal display (LCD) and a touch panel (TP). If the interface includes a touch panel, the interface may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may not only sense the boundaries of touch or swipe actions but also detect the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 408 includes a front-facing camera and / or a rear-facing camera. When electronic device 400 is in an operating mode, such as a shooting mode or a multimedia mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0169] Audio component 410 is used to output and / or input audio signals. For example, audio component 410 includes a microphone (MIC) used to receive external audio signals when electronic device 400 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 404 or transmitted via communication component 416. In some embodiments, audio component 410 also includes a speaker for outputting audio signals.

[0170] Input / output (I / O) interface 412 provides an interface between processing component 402 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0171] Sensor assembly 414 includes one or more sensors for providing state assessments of various aspects of electronic device 400. For example, sensor assembly 414 may detect the on / off state of electronic device 400, the relative positioning of components such as the display and keypad of electronic device 400, changes in position of electronic device 400 or a component of electronic device 400, the presence or absence of user contact with electronic device 400, orientation or acceleration / deceleration of electronic device 400, and temperature changes of electronic device 400. Sensor assembly 414 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 414 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 414 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0172] Communication component 416 facilitates wired or wireless communication between electronic device 400 and other devices. Electronic device 400 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof. In one exemplary embodiment, communication component 416 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 416 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0173] In an exemplary embodiment, the electronic device 400 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to implement a method for demonstrating a vehicle-road cooperative scenario provided in this application embodiment.

[0174] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 404 including instructions, which can be executed by a processor 420 of an electronic device 400 to perform the above-described method. For example, the non-transitory storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0175] Figure 10 This is a block diagram of an electronic device 500 according to another embodiment of this application. For example, the electronic device 500 may be provided as a server. See also Figure 10 The electronic device 500 includes a processing component 522, which further includes one or more processors, and memory resources represented by a memory 532 for storing instructions, such as applications, that can be executed by the processing component 522. The applications stored in the memory 532 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 522 is configured to execute instructions to perform a demonstration method for a vehicle-to-everything (V2X) scenario provided in this application embodiment.

[0176] Electronic device 500 may also include a power supply component 526 configured to perform power management of electronic device 500, a wired or wireless network interface 550 configured to connect electronic device 500 to a network, and an input / output (I / O) interface 558. Electronic device 500 may operate on an operating system stored in memory 532, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.

[0177] In embodiments of this application, memory 632 can be used to store software programs and various data. Memory 632 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, applications or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, memory 632 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 632 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0178] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly handles operations related to the operating system, user interface, and applications, while the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into the processor.

[0179] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described super-resolution reconstruction method for images and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0180] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0181] This application also provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the super-resolution reconstruction method embodiment of the image described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0182] It should be noted that all information and data obtained in the embodiments of this application were obtained with the authorization of the information / data holder. All actions to obtain signals, information, or data in this application were carried out in compliance with the relevant data protection laws and policies of the country where the application is located, and with the authorization of the owner of the corresponding device.

[0183] The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, this application is not directed to any particular programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of this application.

[0184] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0185] Similarly, it should be understood that, for the purpose of simplification and aiding understanding of one or more aspects of the application, various features of the application have sometimes been grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of the application. However, this approach of disclosure should not be construed as reflecting an intention that the claimed application requires more features than expressly recited in each claim. Rather, as reflected in the following claims, the application aspect comprises fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the application.

[0186] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0187] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the sequencing device according to this application. This application can also be implemented as a device or apparatus program for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can take the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0188] It should be noted that the above embodiments are illustrative of this application and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0189] All user information (including but not limited to user device information, user personal information, etc.) and related data involved in this application are information authorized by the user or by the parties involved.

[0190] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0191] The above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

[0192] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A phonetic and semantic based speech generation method, characterized by, The method comprises: inputting text data into a semantic model to obtain a semantic feature matrix of the text data; translating the text data into phonetic symbol data, inputting the phonetic symbol data into a bypass guide module of a text encoding model, and outputting each coupled layer in the bypass guide module to obtain guide information; the guide information is used for fusion with the semantic feature matrix; inputting the semantic feature matrix into the text encoding model, and inputting guide information of each coupled layer in the bypass guide module into each convolutional layer of the text encoding model in sequence to output a fusion feature matrix from the text encoding model; the coupled layers correspond to the convolutional layers one by one; inputting the fusion feature matrix into a speech generation model to obtain speech data corresponding to the text data.

2. The method of claim 1, wherein, The translation of the text data into phonetic symbol data and the input of the phonetic symbol data into the bypass guide module of the text encoding model comprises: performing phrase segmentation on the text data, and finding phonetic symbols mapped by the segmented phrases according to a preset phonetic symbol library to obtain phonetic symbol data; the phonetic symbol library stores the phonetic symbols mapped by the phrases; finding corresponding vectors for the phonetic symbol data by searching a preset vector codebook to obtain a phonetic symbol vector matrix; each phonetic symbol data is labeled with a corresponding vector in the vector codebook; inputting the phonetic symbol vector matrix into the bypass guide module.

3. The method of claim 1, wherein, Each coupled layer of the bypass guide module is divided into a first region and a second region, and the input of the phonetic symbol data into the bypass guide module of the text encoding model and the output of each coupled layer in the bypass guide module to obtain guide information comprise: for any coupled layer in the bypass guide model and adjacent coupled layers of the any coupled layer, when the first region of the any coupled layer is kept from transforming the phonetic symbol data and the second region of the any coupled layer is kept transforming the phonetic symbol data, the first region of the adjacent coupled layer is kept transforming the phonetic symbol data and the second region of the adjacent coupled layer is kept from transforming the phonetic symbol data; or when the first region of the any coupled layer is kept transforming the phonetic symbol data and the second region of the any coupled layer is kept from transforming the phonetic symbol data, the first region of the adjacent coupled layer is kept from transforming the phonetic symbol data and the second region of the adjacent coupled layer is kept transforming the phonetic symbol data; after the transformation of the phonetic symbol data in each coupled layer is completed, guide information of each layer is obtained.

4. The method of claim 1, wherein, The number of coupled layers in the bypass guide module is the same as the number of convolutional layers in the text encoding model and both are even layers.

5. The method of claim 1, wherein, The input of the semantic feature matrix into the text encoding model and the input of guide information of each coupled layer in the bypass guide model into each convolutional layer of the text encoding model in sequence to obtain a fusion feature matrix comprise: when data is about to be input into any convolutional layer of the text encoding model, guide information of each group of one-to-one corresponding coupled layers and first data of the convolutional layer are fused to obtain fusion data; for the first convolutional layer, the first data is the semantic feature matrix; input the fusion data as data of the convolution layer, and output second data from the convolution layer; the second data is first data of a next convolution layer, and for a last convolution layer, the second data is the feature fusion matrix; After all the convolution layers output in sequence, the feature fusion matrix is obtained.

6. The method of claim 5, wherein, The fusion of the guide information of each group of one-to-one corresponding coupling layers and the first data of the current convolution layer to obtain fusion data comprises: The guide information of each group of one-to-one corresponding coupling layers and the first data of the current convolution layer are combined in the following manner: Wherein, O is the fusion data, I is the first data of the convolution layer, C is the guide information of the coupling layer, mean(I) is the mean of I, std(I) is the standard deviation of I, W() and B() are two fully connected layers of the text encoding model, W(C) represents the product of the guide information and one fully connected layer, and B(C) represents the product of the guide information and another fully connected layer. 7.A method for training a phonetic and semantic-based speech generation model, characterized by, The method comprises: Extracting semantic sample information and phonetic sample information in text sample data, and inputting the semantic sample information and the phonetic sample information into a text encoding model to obtain fusion sample feature data; Inputting voice sample data corresponding to the text sample data into a voice encoding model to obtain a voice sample feature matrix; Obtaining a text sample distribution function and a voice sample distribution function by calculating the mean and variance of the fusion sample feature matrix and the voice sample feature matrix respectively; Training an initial model using the voice sample distribution function, and determining a loss value according to the difference between the text sample distribution function and the voice sample distribution function; Adjusting the initial model according to the loss value and a preset loss function to obtain a voice generation model, which is used for the phoneme and semantic based voice generation method in any one of claims 1 to 6.

8. The method of claim 7, wherein, The extraction of semantic sample information and phonetic sample information in text sample data, and the input of the semantic sample information and the phonetic sample information into a text encoding model to obtain fusion sample feature data comprises: Inputting the text sample data into a semantic model to obtain a sample semantic feature matrix of the text sample data; Translating the text sample data into phonetic sample data, and inputting the phonetic sample data into a bypass guide module of the text encoding model to output each coupling layer in the bypass guide module to obtain sample guide information; Inputting the sample semantic feature matrix into a text encoding model, and inputting the sample guide information of each coupling layer in the bypass guide module into each convolution layer of the text encoding model in sequence to output a fusion sample feature matrix from the text encoding model.

9. An electronic device, comprising: A device comprising a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions being executed by the processor to implement the steps of the method of any one of claims 1 to 8.

10. A readable storage medium, characterized by, When the instructions in the readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 8.

Images