Speech-to-unit-based end-to-end southern Fujian language translation and synthesis method and system

Abstract

The invention discloses an end-to-end southern Fujian language translation and synthesis method and system based on voice-to-unit, and relates to the computer technology. The translation method comprises the following steps of: acquiring southern Fujian audio data from a user side, inputting the southern Fujian audio data into an encoder of a Transform network for encoding after resampling and segmentation, audio clip preprocessing and position encoding, sending encoded low-dimensional representation and decoder output into a decoder for autoregression decoding to obtain a translation sequence, and removing special mark symbols to obtain a Chinese text translation result. The synthesis method comprises the following steps: acquiring a Chinese text from a user side, filtering part of punctuation marks, acquiring word embedding and position embedding, encoding through a Transformer encoder to obtain an intermediate hidden variable, decoding through a decoder to obtain an initial Mel spectrum output, performing post-processing correction to obtain the Mel spectrum as an acoustic feature, and generating a southern Fujian audio through a fine-tuned Hifi-Gan vocoder. The method improves the accuracy of speech translation of the southern Fujian language, synthesizes the speech of the southern Fujian language vividly and smoothly, and is suitable for multiple occasions.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an end-to-end Minnan dialect translation and synthesis method and system based on speech to unit. Background Art

[0002] As a minority language, Minnan dialect faces huge challenges in obtaining annotated speech data. There is a lack of open source large-scale Minnan dialect datasets on the Internet. In addition, there are few existing models for recognizing Minnan dialect speech into Chinese text, and all of them use traditional cascade models, which require training models of multiple modules and then connecting them in series. The training process is complicated, memory usage is high, and the upper limit of recognition accuracy is low. Most of the existing Minnan dialect speech synthesis technologies still use phoneme-based synthesis, which is cumbersome to use and cannot directly synthesize Minnan dialect audio based on Chinese text.

[0003] Therefore, how to provide an end-to-end Minnan dialect speech recognition and synthesis system based on deep learning to solve problems such as insufficient corpus data, complex cascade models, and indirect speech synthesis has become a technical challenge that needs to be solved urgently. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide an end-to-end Minnan translation and synthesis method and system based on speech to unit. Minnan is a non-written low-resource language with a small number of users. The present invention aims to achieve the mutual conversion between Minnan audio and Chinese text, that is, to efficiently and accurately translate Minnan audio into Chinese text, and to naturally synthesize Chinese text speech into Minnan audio.

[0005] In a first aspect, the present invention provides an end-to-end Minnan dialect translation method based on speech to unit, comprising the following steps:

[0006] Step S10, obtaining the Minnan dialect audio data to be translated from the user terminal, resampling the original audio, and dividing the audio data into a plurality of audio segments;

[0007] Step S11, preprocessing each audio clip to obtain a frequency domain array capable of representing audio features;

[0008] Step S12, position encoding is performed on a plurality of frequency domain arrays, and the arrays are input as acoustic features into an encoder of a Transformer network for encoding;

[0009] Step S13, the encoded low-dimensional representation and the output of the decoder in the previous step are sent to the decoder, and decoded in an autoregressive manner, and this step is repeated to obtain a translation sequence corresponding to the Minnan audio;

[0010] Step S14: remove special marking symbols to obtain the Chinese text translation result corresponding to the original Minnan dialect audio.

[0011] Furthermore, the step S10 is specifically as follows:

[0012] After obtaining the Minnan audio to be translated from the user end, the audio data is resampled to 16kHz, and the audio data is cut into several audio segments according to the preset duration and active audio detection technology;

[0013] Furthermore, the step S11 is specifically as follows:

[0014] For each audio clip, pre-emphasis, framing, windowing, short-time Fourier transform, and Mel filtering are sequentially performed to obtain a number of frequency domain arrays;

[0015] Furthermore, the step S12 is specifically as follows:

[0016] Perform position encoding on the frequency domain array and input the encoded result into the encoder part of the Transformer network. The encoder extracts the features of the frequency domain array again to obtain a low-dimensional representation, which will be used as the input data of the decoder.

[0017] Furthermore, the step S13 is specifically as follows:

[0018] The decoder takes a low-dimensional representation and a start symbol as the initial input. The decoder first maps it to a latent variable, and then maps the probability distribution of the predefined vocabulary based on the latent variable, and then feeds this result back to the decoder as the next input data, and continues this process until the end symbol is output, which means that the Chinese text sequence generation process of autoregressive decoding is completed;

[0019] Furthermore, the step S14 is specifically as follows:

[0020] Remove special markers, such as the ones that represent the start of a sequence prediction. <startoftranscript>, indicating the end of the sequence <endoftext>, indicating spaces <nospeech>and for unknown characters <unk>After completing the above elimination operation, the Chinese text translation result corresponding to the original Minnan audio can be obtained;

[0021] The end-to-end Minnan translation system based on speech to unit includes the following modules:

[0022] The audio data preprocessing module is used to obtain the Minnan audio data to be translated from the user end, resample the original audio, and divide the audio data into several audio segments. The resampled and segmented audio data is feature encoded to obtain a frequency domain array that can represent the audio features;

[0023] The frequency domain feature encoding module is used to positionally encode several frequency domain arrays and use them as acoustic features to be input into the encoder of the Transformer network for encoding;

[0024] The autoregressive decoding module is used to obtain the translation sequence corresponding to the Minnan audio. The encoded low-dimensional representation and the output of the decoder in the previous step are sent to the decoder for decoding in an autoregressive manner. This step is repeated to obtain the translation sequence corresponding to the Minnan audio.

[0025] Result post-processing module: used to remove special marking symbols and obtain the Chinese text translation result corresponding to the original Minnan audio;

[0026] Furthermore, the audio data preprocessing module is specifically:

[0027] After obtaining the Minnan audio to be translated from the user, the audio data is resampled to 16kHz, and the audio data is cut into several audio segments according to the preset duration and active audio detection technology. Pre-processing processes such as pre-emphasis, framing, windowing, short-time Fourier transform and Mel filtering are implemented for each audio segment in turn to obtain several frequency domain arrays;

[0028] Furthermore, the frequency domain feature encoding module is specifically:

[0029] Perform position encoding on the frequency domain array and input the encoded result into the encoder part of the Transformer network. The encoder extracts the features of the frequency domain array again to obtain a low-dimensional representation, which will be used as the input data of the decoder.

[0030] Furthermore, the autoregressive decoding module is specifically:

[0031] The decoder takes a low-dimensional representation and a start symbol as the initial input. The decoder first maps it to a latent variable, and then maps the probability distribution of the predefined vocabulary based on the latent variable, and then feeds this result back to the decoder as the next input data, and continues this process until the end symbol is output, which means that the Chinese text sequence generation process of autoregressive decoding is completed;

[0032] Furthermore, the result post-processing module is specifically:

[0033] Remove special markers, such as the ones that represent the start of a sequence prediction. <startoftranscript>, indicating the end of the sequence <endoftext>, indicating spaces <nospeech>and for unknown characters <unk>After completing the above elimination operation, the Chinese text translation result corresponding to the original Minnan audio can be obtained;

[0034] On the other hand, the present invention provides an end-to-end Minnan dialect synthesis method based on speech to unit, comprising the following steps:

[0035] Step S20, obtaining Chinese text to be synthesized into Minnan dialect audio from the user terminal, and filtering some punctuation marks in the Chinese text;

[0036] Step S21: Obtain the word embedding of the preprocessed Chinese text through a predefined vocabulary, encode the Chinese text according to the position information, obtain the position encoding, and optimize it in combination with the word frequency information to obtain the position embedding, combine the word embedding and the position embedding and transmit them to the Transformer encoder for encoding to obtain the intermediate latent variable;

[0037] Step S22: The Transformer decoder decodes the intermediate latent variables with the help of the attention mechanism and obtains the initial Mel spectrum output in an autoregressive manner;

[0038] Step S23, performing post-processing operations on the initial Mel spectrum to correct possible erroneous output results of the decoder, thereby obtaining a corrected Mel spectrum as an acoustic feature;

[0039] Step S24, generating final Minnan dialect audio according to acoustic features through the fine-tuned Hifi-Gan vocoder;

[0040] Furthermore, the step S20 is specifically as follows:

[0041] After obtaining the Chinese text to be synthesized into the Minnan dialect audio from the user end, punctuation marks that are not substantially helpful to the tone of the synthesized audio, such as book title marks and quotation marks, are screened out, while punctuation marks that are auxiliary to the tone of the synthesized audio, such as commas and periods used to indicate the length of pauses, and question marks and exclamation marks that can reflect the pitch of the tone are retained;

[0042] Furthermore, the step S21 is specifically as follows:

[0043] The word embedding of the preprocessed Chinese text is obtained through a predefined vocabulary, and the Chinese text is encoded according to the position information to obtain the position embedding. The two are combined and sent to the encoder for encoding to finally obtain the intermediate latent variable; the words that do not appear in the vocabulary are processed using methods such as byte pair encoding.

[0044] Furthermore, the step S22 is specifically as follows:

[0045] The output of the encoder and the start code indicating the start of the autoregression are input to the decoder. The decoder outputs three frames of Mel spectrum, and the last frame of the three frames is input to the decoder again. The autoregression decoding operation is repeated in this way until the complete initial Mel spectrum output is obtained.

[0046] Furthermore, the step S23 is specifically as follows:

[0047] Post-processing of the initial Mel spectrum. Given the unidirectional nature of autoregressive decoding, if errors occur during the decoding process, they are difficult to correct immediately. Therefore, a post-processing module constructed by a convolutional layer is introduced to correct the initial Mel spectrum, and finally a corrected Mel spectrum is obtained and used as an acoustic feature;

[0048] Furthermore, the step S24 is specifically as follows:

[0049] Based on the original Hifi-Gan vocoder, it is fine-tuned according to the self-made Minnan dialect corpus, and the fine-tuned Hifi-Gan vocoder is introduced. The final Minnan dialect audio is generated according to the acoustic features output by the post-processing module.

[0050] The end-to-end Minnan dialect synthesis system based on speech-to-unit includes the following modules:

[0051] A text preprocessing module is used to obtain Chinese text to be synthesized into Minnan dialect audio from the user end, and filter some punctuation marks in the Chinese text;

[0052] The text encoder module is used to obtain the word embedding and position embedding of the preprocessed Chinese text and transmit it to the Transformer encoder for encoding to obtain intermediate latent variables;

[0053] Mel spectrum decoder module, used to obtain the initial Mel spectrum output. The Transformer decoder decodes the intermediate latent variables with the help of the attention mechanism and obtains the initial Mel spectrum output in an autoregressive manner;

[0054] The Mel spectrum post-processing module is used to perform post-processing operations on the initial Mel spectrum to correct the erroneous output results that may occur in the decoder, and then obtain the corrected Mel spectrum as the acoustic feature;

[0055] The vocoder module is used to generate the final Minnan audio according to the acoustic characteristics through the fine-tuned Hifi-Gan vocoder.

[0056] Furthermore, the text preprocessing module is specifically:

[0057] After obtaining the Chinese text to be synthesized into Minnan dialect audio from the user end, punctuation marks that have no substantial help to the tone of the synthesized audio, such as book title marks and quotation marks, are screened out, while punctuation marks that have an auxiliary effect on the tone of the synthesized audio are retained, such as commas and periods used to indicate the length of pauses, and question marks and exclamation marks that can reflect the pitch of the tone.

[0058] Furthermore, the text encoder module is specifically:

[0059] The word embedding of the preprocessed Chinese text is obtained through a predefined vocabulary, and the Chinese text is encoded according to the position information to obtain the position embedding. The two are combined and sent to the encoder for encoding to finally obtain the intermediate latent variable.

[0060] Furthermore, the Mel spectrum decoder module is specifically:

[0061] The output of the encoder and the start code indicating the start of the autoregression are input to the decoder. The decoder outputs three frames of Mel spectrum, and the last frame of the three frames is input to the decoder again. The autoregression decoding operation is repeated until the complete initial Mel spectrum output is obtained.

[0062] Furthermore, the Mel spectrum post-processing module is specifically:

[0063] Post-processing of the initial Mel spectrum. Given the unidirectional nature of autoregressive decoding, if errors occur during the decoding process, they are difficult to correct immediately. Therefore, a post-processing module constructed by a convolutional layer is introduced to correct the initial Mel spectrum, and finally a corrected Mel spectrum is obtained and used as an acoustic feature.

[0064] Furthermore, the vocoder module is specifically:

[0065] Based on the original Hifi-Gan vocoder, it is fine-tuned according to the self-made Minnan dialect corpus, and the fine-tuned Hifi-Gan vocoder is introduced. The final Minnan dialect audio is generated according to the acoustic features output by the post-processing module.

[0066] The advantages of the present invention are:

[0067] The Minnan audio is resampled, cut and preprocessed into a frequency domain array, and its low-dimensional representation is extracted by the Transformer encoder and sent to the decoder for decoding to obtain the translated text; the Chinese text is converted into an embedded vector by punctuation filtering, and the intermediate latent variables are obtained by the encoder, and then the initial Mel spectrum is obtained by the Transformer decoder autoregression, and the acoustic features are obtained by post-processing, and finally the Minnan audio is generated by the fine-tuned Hifi-Gan vocoder. End-to-end model training requires a large-scale data set to improve accuracy. The present invention mines Minnan language materials on the Internet, constructs a large-scale data set of Minnan audio and Chinese text, and uses this to weakly supervise the training of Minnan translation and synthesis models to achieve higher accuracy, and realize efficient, accurate translation and natural synthesis of Minnan, a non-written language with a small number of users and a small number of training data sets. For example, in the scenarios of intelligent voice assistants, audio book production and cross-language communication in Minnan, it can quickly and accurately provide users with high-quality translation and synthesis services, greatly improving the dissemination efficiency and communication effect of Minnan, and realizing efficient, accurate translation and natural synthesis of Minnan. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a flow chart of an end-to-end Minnan dialect translation method based on speech to unit of the present invention.

[0069] Figure 2 It is a flow chart of an end-to-end Minnan dialect synthesis method based on speech to unit of the present invention.

[0070] Figure 3 It is a structural diagram of an end-to-end Minnan dialect translation system based on speech to unit of the present invention. DETAILED DESCRIPTION

[0071] The technical solution in the embodiment of this application has the following general idea: an end-to-end deep learning model is trained based on the Transformer architecture. The model is applied to the translation and generation of Minnan dialect, which effectively solves the problem of lack of standardized text data support for languages ​​without written form and poor training and reasoning performance, and improves the efficiency and quality of translation and synthesis of non-written languages.

[0072] Please refer to Figure 1 As shown, a preferred embodiment of the end-to-end Minnan dialect translation method based on speech to unit of the present invention comprises the following steps:

[0073] Step S10, obtaining the Minnan dialect audio data to be translated from the user terminal, resampling the original audio, and dividing the audio data into a plurality of audio segments;

[0074] Specifically, after the user uploads the Minnan audio file to be translated through the network interface, the audio processing tool FFmpeg is used to uniformly resample the original audio data to a fixed sampling rate. The sampling rate used in this embodiment is 16kHz, which can effectively avoid the high computational burden caused by the high sampling rate while capturing the main features of the human voice. After resampling, the audio data is cut into several audio clips based on the preset duration and active audio detection. In this embodiment, 30s is used as the cutting window, and the existing active audio detection tool is introduced to ensure that there is no active audio at the cutting point. If there is, the cutting window is appropriately shortened.

[0075] To improve the robustness of the system, for abnormal audio input, such as when the uploaded audio file is damaged or the format is not supported, the system will give a corresponding error prompt and ask the user to re-upload or provide suggestions for file format conversion. At the same time, considering the impact of different environmental noise levels on audio cutting, the threshold of active audio detection can be dynamically adjusted according to the environmental noise to achieve more accurate audio segmentation. In addition, when cutting audio clips, in addition to the duration factor, the energy distribution of the audio and the change points of the acoustic characteristics can also be combined to avoid splitting the complete speech semantic unit and ensure semantic coherence.

[0076] Step S11, preprocessing each audio clip to obtain a frequency domain array capable of representing audio features;

[0077] Specifically, use the torchaudio library to read the audio clip and perform preprocessing. When the sound signal propagates in the air medium, high-frequency energy is more easily lost. The audio clip needs to be pre-emphasized to compensate for the high-frequency loss. The audio clip is pre-emphasized to compensate for the loss of high-frequency energy when the sound signal propagates in the air medium. The pre-emphasis formula is:

[0078]

[0079] in, It is the pre-emphasis coefficient, which can be adjusted adaptively according to different audio types (such as male and female voices, different speaking speeds, etc.). The initial value is set to 0.97.

[0080] Due to the fast-changing characteristics of speech signals, and the Fourier transform is suitable for analyzing stable signals, it is necessary to divide the indefinite-length audio into small segments of fixed length by framing. In this embodiment, the cutting is performed at a frame rate of 25 milliseconds. In order to smoothly attenuate the two ends after framing and obtain a higher-quality spectrum, a Hamming window is used with a frame shift length of 10 milliseconds. After applying the short-time Fourier transform, the Mel filter bank can be used to extract the Fbank feature, achieving the result of simulating the human ear to process the sound.

[0081] Step S12, position encoding is performed on a plurality of frequency domain arrays, and the arrays are input as acoustic features into an encoder of a Transformer network for encoding;

[0082] Specifically, this embodiment crawls resources such as Minnan dialect TV dramas, speeches, variety shows, etc. through the network, and obtains the subtitle text corresponding to the audio through text recognition technology (such as the open source Tesseract OCR combined with the voice subtitle extraction tool). Cut the audio and video data in units of 30 seconds to construct a data set for training. Position encoding is implemented using a learnable parameter matrix, and the Transformer network is trained to obtain a usable model. In this embodiment, the encoder is composed of twelve encoder blocks, each of which is composed of a multi-head self-attention layer, a feedforward layer, a normalization layer, and a residual connection. For the lth encoder block, the calculation formula is:

[0083] X l =LayerNorm((X l-1 +MultiHeadSelfAttention(X l-1 ))+FeedForward(X l-1 +MultiHeadSelfAttention(X l-1 ))

[0084] When training the Transformer network, the Adam optimizer was used and the initial learning rate was set to 0.001.

[0085] Step S13, the encoded low-dimensional representation and the output of the decoder in the previous step are sent to the decoder, and decoded in an autoregressive manner, and this step is repeated to obtain a translation sequence corresponding to the Minnan audio;

[0086] Specifically, the decoder is composed of decoder blocks, each of which uses a multi-head self-attention mechanism to process the input feature sequence, and uses a causal mask to ensure orderliness to obtain a self-attention result. The specific implementation of the causal mask is to mask the elements after the current position to ensure that the decoding process is autoregressive. Then the cross-attention mechanism associates and fuses the self-attention result with the audio semantic information encoded by the encoder, so that the decoder can combine more comprehensive information. During the association process, the weights are adjusted according to different degrees of semantic association, and the attention weights are visualized to better understand the decoding process. Then, the cross-attention output is nonlinearly transformed through a feedforward neural network to further extract features, and each layer has residual connections and layer normalization. After multiple layers of decoding, a tag sequence corresponding to the text is generated at the output.

[0087] Here is an example of a Hokkien audio clip that means "The weather is very good today". First, the decoder receives the start marker <startoftranscript>As the first input and combined with the low-dimensional representation z from the encoder 0 , the first character "今" is decoded. Next, "今" is sent back to the decoder as the new input for the next round of decoding, combined with the updated hidden state z 1 , and the next most likely character "日" is decoded. This process is repeated until the final decoding of <endoftext>Indicates the end of autoregressive decoding.

[0088] In order to improve the accuracy and diversity of decoding, a beam search strategy can be introduced with the beam width set to 5 to avoid the local optimal problem that may be caused by greedy search.

[0089] Step S14, removing special marking symbols to obtain the Chinese text translation result corresponding to the original Minnan audio;

[0090] Specifically, if there is a decoding sequence <startoftranscript>The weather is very nice today. <endoftext>, after removing the special marking symbols, we get the final translation result "The weather is very good today.".

[0091] Please refer to Figure 2 As shown, a preferred embodiment of the end-to-end Minnan dialect synthesis method based on speech to unit of the present invention comprises the following steps:

[0092] Step S20, obtaining Chinese text to be synthesized into Minnan dialect audio from the user terminal, and filtering some punctuation marks in the Chinese text;

[0093] Specifically, a Chinese text to be synthesized into Minnan dialect audio is obtained from a user terminal, and punctuation marks that are not helpful for the tone of the synthesized audio, such as book title marks and quotation marks, are removed, and punctuation marks that are helpful for the tone of the synthesized audio, such as commas and periods that indicate the length of pauses, and question marks and exclamation marks that indicate the pitch of a tone are left;

[0094] In addition to filtering punctuation marks, numbers, abbreviations, special words, etc. in the text are normalized to make it more in line with the requirements of speech synthesis.

[0095] Step S21, obtaining the word embedding and position embedding of the preprocessed Chinese text, and transmitting them to the Transformer encoder for encoding, thereby obtaining intermediate latent variables;

[0096] Specifically, position encoding and word encoding are implemented using a learnable parameter matrix, and the Transformer network is trained. In this embodiment, the encoder is composed of eight encoder blocks, each of which is composed of a multi-head self-attention layer, a feedforward layer, a normalization layer, and a residual connection. For the lth encoder block, the calculation formula is:

[0097] X l = LayerNorm((X l-1 + MultiHeadSelfAttention(X l-1 )) + FeedForward(X l-1 + MultiHeadSelfAttention(X l-1 ))

[0098] When training the Transformer network, the RMSprop optimizer was used, the learning rate was set to 0.0005, and Dropout regularization was used to prevent overfitting, and the Dropout rate was set to 0.2.

[0099] Step S22: The Transformer decoder decodes the intermediate latent variables with the help of the attention mechanism and obtains the initial Mel spectrum output in an autoregressive manner;

[0100] Specifically, the starting marker frame and the encoder's low-dimensional representation z 0 Decode three frames of Mel spectrum. Send the last frame of the three frames to the decoder again, along with the updated low-dimensional representation z 1 Continue the operation and autoregressive decoding to obtain the complete initial Mel spectrum output;

[0101] Step S23, performing post-processing operations on the initial Mel spectrum to correct possible erroneous output results of the decoder, thereby obtaining a corrected Mel spectrum as an acoustic feature;

[0102] Specifically, for the post-processing of the initial Mel spectrum, since the Transformer decoder is unidirectional and non-cascaded, if errors occur during the process, they will not be corrected in time. A post-processing module consisting of two convolutional layers is introduced to correct the initial Mel spectrum and obtain the corrected Mel spectrum as the acoustic feature; the structure of the convolutional layer includes the combination of different convolution kernel sizes (such as 3×3 and 5×5), step sizes (such as 1 and 2) and padding methods (such as SAME and VALID), and ReLU is used as the activation function. Batch normalization is added to the convolutional layer to improve the stability and training speed of the network.

[0103] Step S24, generating final Minnan dialect audio according to acoustic features through the fine-tuned Hifi-Gan vocoder;

[0104] Based on the original Hifi-Gan vocoder, the fine-tuned Hifi-Gan vocoder is introduced according to the self-made Minnan language corpus, and the final Minnan language audio is generated according to the acoustic features output by the post-processing module;

[0105] When fine-tuning, the Adam optimizer is used with an initial learning rate of 0.0001, a batch size of 32, and 1000 iterations.

[0106] like Figure 3 A preferred embodiment of the end-to-end Minnan dialect translation system based on speech to unit of the present invention comprises the following modules:

[0107] The audio data preprocessing module is used to obtain the Minnan audio data to be translated from the user end, resample the original audio, and cut the audio data into audio segments. Feature encoding is performed on the resampled and cut audio data to extract speech features;

[0108] The audio data preprocessing module is used to obtain the Minnan audio data to be translated from the user end, resample the original audio, and divide the audio data into several audio segments. The resampled and segmented audio data is feature encoded to obtain a frequency domain array that can represent the audio features;

[0109] The frequency domain feature encoding module is used to positionally encode several frequency domain arrays and use them as acoustic features to be input into the encoder of the Transformer network for encoding;

[0110] The autoregressive decoding module is used to obtain the translation sequence corresponding to the Minnan audio. The encoded low-dimensional representation and the output of the decoder in the previous step are sent to the decoder for decoding in an autoregressive manner. This step is repeated to obtain the translation sequence corresponding to the Minnan audio.

[0111] Result post-processing module: used to remove special marking symbols and obtain the Chinese text translation result corresponding to the original Minnan audio.

[0112] The audio data preprocessing module is specifically:

[0113] After obtaining the Minnan audio to be translated from the user, the audio data is resampled to 16kHz, and the audio data is cut into several audio segments according to the preset duration and active audio detection technology. Pre-processing processes such as pre-emphasis, framing, windowing, short-time Fourier transform and Mel filtering are implemented in sequence for each audio segment to obtain several frequency domain arrays.

[0114] The frequency domain feature encoding module is specifically:

[0115] Perform position encoding on the frequency domain array and input the encoded result into the encoder part of the Transformer network. The encoder extracts the features of the frequency domain array again to obtain a low-dimensional representation, which will be used as the input data of the decoder.

[0116] The autoregressive decoding module is specifically:

[0117] The decoder takes a low-dimensional representation and a start symbol as the initial input. The decoder first maps it to a latent variable, and then maps the probability distribution of the predefined vocabulary based on the latent variable, and then feeds this result back to the decoder as the next input data, and continues this process until the end symbol is output, which means that the Chinese text sequence generation process of autoregressive decoding is completed.

[0118] The result post-processing module is specifically:

[0119] Remove special markers, such as the ones that represent the start of a sequence prediction. <startoftranscript>, indicating the end of the sequence <endoftext>, indicating spaces <nospeech>and for unknown characters <unk>After completing the above elimination operation, the Chinese text translation result corresponding to the original Minnan audio can be obtained;

[0120] A preferred embodiment of an end-to-end Minnan dialect synthesis system based on speech-to-units of the present invention comprises the following modules in sequence:

[0121] A text preprocessing module is used to obtain Chinese text to be synthesized into Minnan dialect audio from the user end, and filter some punctuation marks in the Chinese text;

[0122] The text encoder module is used to obtain the word embedding and position embedding of the preprocessed Chinese text and transmit it to the Transformer encoder for encoding to obtain intermediate latent variables;

[0123] Mel spectrum decoder module, used to obtain the initial Mel spectrum output. The Transformer decoder decodes the intermediate latent variables with the help of the attention mechanism and obtains the initial Mel spectrum output in an autoregressive manner;

[0124] The Mel spectrum post-processing module is used to perform post-processing operations on the initial Mel spectrum to correct the erroneous output results that may occur in the decoder, and then obtain the corrected Mel spectrum as the acoustic feature;

[0125] The vocoder module is used to generate the final Minnan audio according to the acoustic characteristics through the fine-tuned Hifi-Gan vocoder;

[0126] The text preprocessing module is specifically:

[0127] After obtaining the Chinese text to be synthesized into the Minnan dialect audio from the user end, punctuation marks that are not substantially helpful to the tone of the synthesized audio, such as book title marks and quotation marks, are screened out, while punctuation marks that are auxiliary to the tone of the synthesized audio, such as commas and periods used to indicate the length of pauses, and question marks and exclamation marks that can reflect the pitch of the tone are retained;

[0128] The text encoder module is specifically:

[0129] The word embedding of the preprocessed Chinese text is obtained through a predefined vocabulary, and the Chinese text is encoded according to the position information to obtain the position embedding. The two are combined and sent to the encoder for encoding to finally obtain the intermediate latent variable;

[0130] The Mel spectrum decoder module is specifically:

[0131] The output of the encoder and the start code indicating the start of the autoregression are input to the decoder. The decoder outputs three frames of Mel spectrum, and the last frame of the three frames is input to the decoder again. The autoregression decoding operation is repeated in this way until the complete initial Mel spectrum output is obtained.

[0132] The Mel spectrum post-processing module is specifically:

[0133] Post-processing of the initial Mel spectrum. Given the unidirectional nature of autoregressive decoding, if errors occur during the decoding process, they are difficult to correct immediately. Therefore, a post-processing module constructed by a convolutional layer is introduced to correct the initial Mel spectrum, and finally a corrected Mel spectrum is obtained and used as an acoustic feature;

[0134] The vocoder module is specifically:

[0135] Based on the original Hifi-Gan vocoder, it is fine-tuned according to the self-made Minnan dialect corpus, and the fine-tuned Hifi-Gan vocoder is introduced. The final Minnan dialect audio is generated according to the acoustic features output by the post-processing module.

[0136] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.< / unk> < / nospeech> < / endoftext> < / startoftranscript> < / endoftext> < / startoftranscript> < / endoftext> < / startoftranscript> < / unk> < / nospeech> < / endoftext> < / startoftranscript> < / unk> < / nospeech> < / endoftext> < / startoftranscript>

Claims

1. An end-to-end Minnan dialect translation method based on speech to unit, characterized in that: The following steps are involved: Step S10, obtaining the Minnan dialect audio data to be translated, resampling the original audio, and cutting it into a plurality of audio segments according to a preset duration by using an active audio detection technology; Step S11, preprocessing each audio clip to obtain a frequency domain array capable of representing audio features; The preprocessing sequentially performs pre-emphasis, frame division, windowing, short-time Fourier transform and Mel filtering; Step S12: Encode the frequency domain array position and use it as an acoustic feature to input into the Transformer network encoder for encoding, extract features to obtain a low-dimensional representation; Step S13, using the encoded low-dimensional representation and the start symbol as the initial input of the decoder, mapping the probability distribution through latent variables, decoding in an autoregressive manner, and repeating this step to generate a translation sequence corresponding to the Minnan audio; Step S14: remove special marking symbols to obtain the Chinese text translation result corresponding to the original Minnan dialect audio.

2. The end-to-end Minnan dialect translation method based on speech to unit as claimed in claim 1, characterized in that: In step S13, the generation of the translation sequence corresponding to the Minnan audio is specifically: a low-dimensional representation and a start symbol are used as the initial input of the decoder, the decoder first maps it into a latent variable, and then maps out the probability distribution of a predefined vocabulary based on the latent variable, and then feeds this result back to the decoder as the next input data, and continues this process until the end symbol is output, which means that the Chinese text sequence generation process of autoregressive decoding is completed.

3. The end-to-end Minnan dialect translation method based on speech to unit as claimed in claim 1, characterized in that: In step S14, the special marking symbols are removed, including the ones representing the start of sequence prediction. <startoftranscript>, indicating the end of the sequence <endoftext>, indicating spaces <nospeech>and for unknown characters <unk> And other symbols.< / unk> < / nospeech> < / endoftext> < / startoftranscript> 4. An end-to-end Minnan dialect translation system based on speech to unit, characterized in that: Includes the following modules: The audio data preprocessing module is used to obtain the Minnan audio data to be translated from the user end, resample the original audio, and divide the audio data into several audio segments, perform feature encoding on the resampled and segmented audio data, and obtain a frequency domain array that can represent the audio features; The frequency domain feature encoding module is used to positionally encode several frequency domain arrays and use them as acoustic features to be input into the encoder of the Transformer network for encoding; The autoregressive decoding module is used to send the encoded low-dimensional representation and the output of the previous decoder to the decoder for decoding in an autoregressive manner, thereby obtaining the translation sequence corresponding to the Minnan audio; The result post-processing module is used to remove special marking symbols and obtain the Chinese text translation result corresponding to the original Minnan audio.

5. A speech-to-unit based end-to-end Minnan dialect translation system as claimed in claim 4, characterized in that: The audio data preprocessing module adopts a parallel computing architecture to process preprocessing operations of multiple audio clips simultaneously to improve processing efficiency and meet real-time translation requirements.

6. An end-to-end Minnan dialect synthesis method based on speech to unit, characterized in that: The following steps are involved: Step S20: obtaining Chinese text to be synthesized into Minnan dialect audio from the user terminal, and filtering some punctuation marks in the Chinese text; Step S21: Obtain the word embedding of the preprocessed Chinese text through a predefined vocabulary, encode the Chinese text according to the position information, obtain the position encoding, and optimize it in combination with the word frequency information to obtain the position embedding, combine the word embedding and the position embedding and transmit them to the Transformer encoder for encoding to obtain the intermediate latent variable; Step S22: The Transformer decoder decodes the intermediate latent variables with the help of the attention mechanism and obtains the initial Mel spectrum output in an autoregressive manner; Step S23: post-processing the initial Mel spectrum. Given that autoregressive decoding is unidirectional, if an error occurs during the decoding process, it is difficult to correct it immediately. In this case, a Mel spectrum post-processing module constructed by a convolutional layer is introduced to correct the initial Mel spectrum, correct the possible erroneous output results of the decoder, obtain the corrected Mel spectrum, and use it as an acoustic feature. Step S24: fine-tune the original Hifi-Gan vocoder according to the self-made Minnan dialect corpus, introduce the fine-tuned Hifi-Gan vocoder, and generate the final Minnan dialect audio according to the acoustic features output by the Mel spectrum post-processing module.

7. The end-to-end Minnan dialect synthesis method based on speech to unit as claimed in claim 6, characterized in that: In step S20, the filtering process of some punctuation marks in the Chinese text is specifically as follows: punctuation marks that are of no substantial help to the synthesized audio tone, including book title marks and quotation marks; punctuation marks that have an auxiliary effect on the synthesized audio tone are retained, including commas and periods used to indicate the length of pauses, and question marks and exclamation marks that can reflect the pitch of the tone.

8. The end-to-end Minnan dialect synthesis method based on speech to unit as claimed in claim 6, characterized in that: In step S22, the Transformer decoder decodes the intermediate latent variables with the help of the attention mechanism and obtains the initial Mel spectrum output in an autoregressive manner. Specifically: The output of the encoder and the start code indicating the start of the autoregression are input to the decoder. The decoder outputs three frames of Mel spectrum, and the last frame of the three frames is input to the decoder again. The autoregression decoding operation is repeated until the complete initial Mel spectrum output is obtained.

9. An end-to-end Minnan dialect synthesis system based on speech to unit, characterized in that: The following modules are included in sequence: The text preprocessing module is used to filter punctuation marks in Chinese text and normalize the text, including removing redundant spaces, converting special characters, and cleaning up repeated punctuation marks to ensure the quality and standardization of the input text; The text encoder module is used to generate word embeddings and position embeddings and encode them to obtain intermediate latent variables; Mel spectrum decoder module, used for autoregressive decoding to obtain the initial Mel spectrum; Mel spectrum post-processing module, used to modify the initial Mel spectrum to obtain acoustic features; Vocoder module for generating Minnan audio using a fine-tuned Hifi-Gan vocoder.

Images