Accent conversion method and device, equipment and storage medium

Abstract

The invention belongs to the technical field of artificial intelligence, and discloses an accent conversion method, device and equipment and a storage medium, and the accent conversion method comprises the steps: obtaining a to-be-converted voice, and carrying out the analysis to obtain a first pronunciation set in the to-be-converted voice; obtaining a standard speech corpus, inputting the standard speech corpus into the pronunciation correction model for training, and forming a mapping relation between discrete standard pronunciation units in the pronunciation correction model and standard pronunciation; inputting the voice to be converted into a pronunciation correction model, and determining a discrete standard pronunciation unit corresponding to the first pronunciation in the first pronunciation set; based on the discrete standard pronunciation unit, correcting the first pronunciation into a standard pronunciation corresponding to the discrete standard pronunciation unit; and combining the corrected standard pronunciations of the first pronunciations to form a complete standard voice signal, and outputting the complete standard voice signal. According to the invention, problems existing in an accent conversion mode in the prior art are solved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to an accent conversion method, device, equipment and storage medium. Background Art

[0003] In order to convert the same content from one voice expression to another voice expression, the prior art proposes adding reference voice in the conversion process, which means that when converting the content from one voice expression to another voice expression, the original voice is required as a reference for the conversion operation. This method may be limited by factors such as the quality and duration of the original voice. For example, if part of the original voice is blurred or missing, it may affect the accuracy of the accent conversion. Moreover, the existence of reference voice may increase the complexity of the system and its dependence on resources.

[0004] To this end, the prior art proposes to adopt a one-to-one system, that is, this system only converts one voice expression of the same content into another voice expression during training, but cannot convert multiple language expressions of the same content into another voice expression. For example, to convert the Chinese accent into the British accent is a separate training system, and to convert into the American accent requires another separate training system. This method is inefficient, and when the number of accents that need to be converted increases, a lot of training work and storage resources are required to maintain these one-to-one systems.

[0005] In summary, there is an urgent need for an accent conversion method in the prior art to overcome the problems existing in the accent conversion method in the prior art. Summary of the invention

[0006] The present invention provides an accent conversion method, device, equipment and storage medium to solve the problems existing in the accent conversion methods in the prior art.

[0007] In a first aspect, the present invention provides an accent conversion method, comprising:

[0008] Acquire the speech to be converted, and analyze to obtain a first pronunciation set in the speech to be converted;

[0009] Acquire standard speech corpus, input the standard speech corpus into a pronunciation correction model for training, and form a mapping relationship between discrete standard pronunciation units and standard pronunciation in the pronunciation correction model;

[0010] Inputting the speech to be converted into a pronunciation correction model to determine a discrete standard pronunciation unit corresponding to the first pronunciation in the first pronunciation set;

[0011] Based on the discrete standard pronunciation unit, correcting the first pronunciation to the standard pronunciation corresponding to the discrete standard pronunciation unit;

[0012] The standard pronunciations corrected by the first pronunciations are combined to form a complete standard voice signal and output it.

[0013] In a second aspect, the present invention provides an accent conversion device, comprising:

[0014] An analysis module, used for acquiring the speech to be converted, and analyzing to obtain a first pronunciation set in the speech to be converted;

[0015] A training module, used for acquiring standard speech corpus, inputting the standard speech corpus into a pronunciation correction model for training, and forming a mapping relationship between discrete standard pronunciation units and standard pronunciation in the pronunciation correction model;

[0016] A determination module, configured to input the speech to be converted into a pronunciation correction model, and determine a discrete standard pronunciation unit corresponding to the first pronunciation in the first pronunciation set;

[0017] A correction module, configured to correct the first pronunciation to a standard pronunciation corresponding to the discrete standard pronunciation unit based on the discrete standard pronunciation unit;

[0018] The combining module is used to combine the standard pronunciations corrected by the first pronunciations to form a complete standard voice signal and output it.

[0019] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned accent conversion method when executing the computer program.

[0020] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, which implements the steps of the above-mentioned accent conversion method when executed by a processor.

[0021] In the scheme implemented by the above-mentioned accent conversion method, the speech to be converted is converted into a first pronunciation set. For each first pronunciation in the first pronunciation set, a discrete standard pronunciation unit corresponding to each standard pronunciation is set in the pronunciation correction model, so that each first pronunciation is corrected by using the discrete standard pronunciation unit to correct it to a standard pronunciation, so that each first pronunciation in the speech to be converted can be corrected to a standard pronunciation in the end. Obviously, this scheme does not adopt the scheme of adding reference speech in the conversion process in the prior art, so it avoids the influence of factors such as the quality and duration of the original speech on the accuracy of accent conversion. In addition, since the present application converts the first pronunciation of the speech to be converted into a standard pronunciation, even if there are multiple types of the first pronunciation to be converted, it will still be converted into a standard pronunciation in the end to obtain the standard pronunciation of the speech to be converted, which avoids the problem that a large amount of training work and storage resources are required to maintain a one-to-one system in the prior art when multiple accent types need to be converted. In summary, this scheme can solve the problems existing in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0023] Figure 1 is a flow chart of an accent conversion method according to an embodiment of the present invention;

[0024] Figure 2 is a flow chart of step S20 in one embodiment of the present invention;

[0025] Figure 3 is another flow chart of the accent conversion method according to one embodiment of the present invention;

[0026] Figure 4 is a flow chart of step S31 in one embodiment of the present invention;

[0027] Figure 5 is a flow chart of step S60 in one embodiment of the present invention;

[0028] Figure 6 is another flowchart of step S60 in one embodiment of the present invention;

[0029] Figure 7 is a structural schematic diagram of an accent conversion device in one embodiment of the present invention;

[0030] Figure 8 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0031] Fig. 9 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] See also Figure 1 As shown, Figure 1 A flowchart of an accent conversion method provided by an embodiment of the present invention includes the following steps:

[0034] Step S10, obtaining the speech to be converted, and analyzing to obtain a first pronunciation set in the speech to be converted.

[0035] It should be noted that the speech to be converted in this step is the speech that needs to be converted. For example, when it is necessary to convert Chinese with an American accent into Chinese with a Mandarin accent, the speech to be converted is Chinese with an American accent; when it is necessary to convert Chinese with a British accent into Chinese with a Mandarin accent, the speech to be converted is Chinese with a British accent; when it is necessary to convert Chinese with a Cantonese accent into Chinese with a Mandarin accent, the speech to be converted is Chinese with a Cantonese accent.

[0036] It should be noted that the first pronunciation set of the speech to be converted includes all first pronunciations corresponding to the speech to be converted. The first pronunciation of the speech to be converted is the pronunciation of each word or phrase in the speech to be converted. For example, if the first pronunciation is the pronunciation of a word in the speech to be converted, the first pronunciation can be the initial consonant, final vowel and tone of the corresponding word.

[0037] It should be noted that the method of analyzing and obtaining the first pronunciation set in the speech to be converted can be any feasible method, for example, a pre-trained speech recognition system based on a hidden Markov model can be used. The speech signal is regarded as an observation sequence generated by a series of hidden states (corresponding to pronunciation units such as phonemes). The most likely phoneme sequence can also be found by calculating the probability of each hidden state generating an observation sequence. The spectral envelope of the speech is obtained by performing linear prediction analysis on the speech signal. Different pronunciation units (such as vowels and consonants) have different spectral characteristics. The pronunciation unit can be judged based on information such as the peak value, valley value and resonance peak of the spectral envelope.

[0038] Step S20, obtaining standard speech corpus, inputting the standard speech corpus into the pronunciation correction model for training, and forming a mapping relationship between discrete standard pronunciation units and standard pronunciation in the pronunciation correction model.

[0039] It should be noted that the standard speech corpus in this step is the speech corpus composed of the standard pronunciation obtained by converting the speech to be converted. For example, the accent conversion method of this embodiment ultimately needs to convert the accent of the speech to be converted into the Mandarin pronunciation of Chinese. Then the standard speech corpus in this step is the corpus of the Mandarin pronunciation of Chinese.

[0040] It should be noted that, in order to obtain enough standard speech corpora, a first number of standard speech corpora of sentences may be obtained, wherein the first number is greater than a preset threshold.

[0041] It should be noted that the discrete standard pronunciation units in this step are the corresponding words or characters in the standard speech corpus, and accordingly, the standard pronunciations that establish a mapping relationship with the discrete standard pronunciation units are the pronunciations of the corresponding words or characters. For example, for the standard speech corpus "I love you", the discrete standard pronunciation units that can be obtained are "I", "love" and "you", and the corresponding standard pronunciations that establish a mapping relationship with the above discrete standard pronunciation units are "wo", "ai" and "ni".

[0042] In one embodiment of the present invention, Figure 2 is a flow chart of step S20 in an embodiment of the present invention, as shown in Figure 2 As shown, step S20 includes:

[0043] Step S21, obtaining standard speech corpus;

[0044] Step S22, inputting the standard speech corpus into a pre-training model for training, and obtaining a set of discrete speech units corresponding to the standard speech corpus through training;

[0045] Step S23, clustering each discrete speech unit in the discrete speech unit set to obtain a discrete speech unit center set;

[0046] Step S24, taking each discrete speech unit center in the discrete speech unit center set as a discrete standard pronunciation unit to generate a discrete standard pronunciation unit set;

[0047] Step S25, extracting the standard pronunciation corresponding to each discrete standard pronunciation unit in the discrete standard pronunciation unit set, and establishing a mapping relationship between the discrete standard pronunciation unit and the standard pronunciation;

[0048] Step S26: generating a pronunciation correction model including each discrete standard pronunciation unit in the discrete standard pronunciation unit set according to the mapping relationship between each discrete standard pronunciation unit and the standard pronunciation in the discrete standard pronunciation unit set.

[0049] It should be noted that the pre-trained model in step S22 can be any model that can train a set of discrete speech units corresponding to the standard speech corpus when the standard speech corpus is input into the pre-trained model. For example, the pre-trained model can be hubert or wav2vec. Huber or wav2vec can capture various speech features from the standard speech corpus, such as the initial consonant features and final consonant features in the standard speech corpus, and then use these initial consonant features and final consonant features as discrete speech units to finally form a set of discrete speech units.

[0050] It can be understood that in step S23, the discrete speech units in the discrete speech unit set are clustered, and similar discrete speech units in the discrete speech unit set can be clustered together, and each group of similar discrete speech units has a clustering center (discrete speech unit center) after clustering, and this discrete speech unit center can represent this group of similar discrete speech units.

[0051] It should be noted that the clustering algorithm in step S23 may be any achievable algorithm, for example, the clustering algorithm may be a K-means algorithm.

[0052] Furthermore, in step S24, the discrete standard pronunciation unit set finally formed is the set of all corresponding discrete standard pronunciation units in the standard speech corpus. For example, if the discrete standard pronunciation units are initial consonants and finals in the standard speech corpus, then the discrete standard pronunciation set is the set consisting of all initial consonants and finals in the standard speech corpus.

[0053] Step S40: input the speech to be converted into a pronunciation correction model to determine a discrete standard pronunciation unit corresponding to the first pronunciation in the first pronunciation set.

[0054] It should be noted that the standard pronunciation of the discrete standard pronunciation unit corresponding to the first pronunciation is the pronunciation for the same content. For example, when the Cantonese pronunciation of "I love you" needs to be converted into Mandarin pronunciation, the first pronunciation can be the Cantonese pronunciation of "I", and the standard pronunciation of the discrete standard pronunciation unit corresponding to the first pronunciation is the Mandarin pronunciation of "I".

[0055] It should be noted that the discrete standard pronunciation unit corresponding to the first pronunciation in the first pronunciation set may be determined in any achievable manner. For example, the similarity between the first pronunciation in the first pronunciation set and the standard pronunciation corresponding to all discrete standard pronunciation units in the pronunciation correction model may be calculated, and the discrete standard pronunciation unit corresponding to the standard pronunciation with the highest similarity to the first pronunciation may be obtained as the discrete standard pronunciation unit corresponding to the first pronunciation.

[0056] Furthermore, in this step, after the speech to be converted is input into the pronunciation correction model, all discrete standard pronunciation units corresponding to all first pronunciations in the first pronunciation set corresponding to the speech to be converted can be finally obtained.

[0057] Step S50: Based on the discrete standard pronunciation unit, the first pronunciation is corrected to the standard pronunciation corresponding to the discrete standard pronunciation unit.

[0058] It is understandable that in this step, the first pronunciation may be corrected to the standard pronunciation corresponding to the discrete standard pronunciation unit in any achievable manner, for example, the first pronunciation may be directly replaced by the standard pronunciation corresponding to the first pronunciation.

[0059] In one embodiment of the present invention, Figure 3 The flowchart of another accent conversion method provided by the embodiment of the present invention further includes, after step S20 and before step S50:

[0060] Step S31, obtaining sample information, and generating a set of non-standard voices of different speakers corresponding to the sample information;

[0061] Step S32, inputting each non-standard voice in the non-standard voice set into a pronunciation correction model for training, and outputting a standard pronunciation corresponding to the non-standard voice;

[0062] Step S33, inputting the standard pronunciation into the loss function of the pronunciation correction model to calculate the loss value of the standard pronunciation;

[0063] Step S34, when the loss value is within the preset error range, generate a final pronunciation correction model, when the loss value exceeds the preset error range, change the parameters of the pronunciation correction model, and retrain the pronunciation correction model until the loss value is within the preset error range.

[0064] It should be noted that the sample information in step S31 can be any sample information with actual content, such as text information or image information (such as image information of a cat). In this way, in step S31, the non-standard speech set of different speakers corresponding to the sample information can be a non-standard speech set of different speakers for the text information. For example, if the text information is "the vehicle insurance is 500 yuan", speaker A can read the text speech in Cantonese to obtain a non-standard speech, and speaker B can read the text speech in Sichuan dialect to obtain another non-standard speech, and then these two non-standard speech forms a non-standard speech set.

[0065] It should be noted that the pronunciation correction model in step S33 is any model that can correct non-standard speech to standard pronunciation. In this way, when the non-standard speech in the non-standard speech set is input as a sample into the pronunciation correction model for training, since the pronunciation correction model has discrete standard pronunciation units, the non-standard speech can be corrected to the standard pronunciation corresponding to the discrete standard pronunciation units by using the discrete standard pronunciation units.

[0066] It should be noted that, for step S33 and step S34, the smaller the output value of the loss function of the pronunciation correction model is, the better the correction effect of the pronunciation correction model on the processed speech is. Therefore, the preset error range in step S34 can be set according to the specific needs of the application and can be a smaller value.

[0067] It can be understood that, based on steps S31 to S34, a pronunciation correction model having a better correction effect on the first pronunciation can be obtained.

[0068] In one embodiment of the present invention, Figure 4 is a flow chart of step S31, such as Figure 4 As shown, step S31 includes:

[0069] Step S311, acquiring sample information, and generating non-standard speech subsets of different speakers corresponding to the sample information;

[0070] Step S312: Overlap and / or combine the non-standard voices in the non-standard voice subset to generate a non-standard voice set.

[0071] It is understandable that, when the training sample of the pronunciation correction model is more complex, if the training result of the pronunciation correction model for the training sample is good, the pronunciation correction model can still perform well for the correction of simple input data. Obviously, step S312 overlaps and / or combines the non-standard voices in the non-standard voice subset to obtain a more complex non-standard voice set.

[0072] Step S60: Combining the standard pronunciations corrected from the first pronunciations to form a complete standard speech signal and outputting it.

[0073] It is understandable that after the first pronunciations in the first pronunciation set are corrected to standard pronunciations and combined, the standard voice signal finally outputted is the standard voice corresponding to the speech to be processed. In this way, the need to convert the speech to be processed into the required accent can be met.

[0074] In one embodiment of the present invention, the pronunciation correction model is an encoder-decoder structure under a transformer architecture.

[0075] It is understandable that the Transformer architecture is mainly composed of two parts: encoder and decoder. This architecture abandons the traditional recurrent neural network structure and processes sequence data entirely based on the attention mechanism.

[0076] The multi-head attention mechanism allows the model to simultaneously focus on different parts of the input sequence in different representation subspaces. For example, when processing a text sequence corresponding to speech, one head may focus on the semantics of a word, and another head may focus on the grammatical role of the word. For each head, the attention scores for the query, key, and value are calculated, and then the results of these heads are concatenated and linearly transformed to obtain the final attention output.

[0077] This mechanism enables the model to effectively capture long-distance dependencies in the input sequence. In the pronunciation correction model, for a sentence containing multiple words (first pronunciation), the multi-head attention mechanism can help the model understand the relationship between different words, such as how the pronunciation of a word is affected by the pronunciation of the previous and next words, which is very critical for correcting pronunciation.

[0078] Since the Transformer architecture does not have the ability to process sequence order information like the recurrent neural network architecture, it is necessary to introduce position encoding to indicate the position of elements in the sequence. The position encoding is usually a vector with the same dimension as the input, which is added to the input word vector (or speech unit vector). The position encoding can be fixed (such as generated using sine and cosine functions) or learned. In the pronunciation correction model, the position encoding helps the model know the order of pronunciation units in the sentence. For example, when processing a phoneme sequence in a sentence, the model can determine the order of phonemes based on the position encoding, and then correctly correct the pronunciation order.

[0079] For a pronunciation correction model, the input to the encoder is a sequence containing information related to the speech to be processed. This sequence can be a representation converted from the speech to be processed corresponding to a sentence selected from a text corpus, which may be a sequence of phonemes, a sequence of acoustic features, etc. For example, if the speech to be processed for the input sentence "I like music" is considered, the encoder will take the sequence of pronunciation units corresponding to this audio as the input.

[0080] The encoder uses a multi - layer multi - head attention mechanism and a feed - forward neural network to extract features and encode the input sequence. In each layer, the multi - head attention mechanism first performs self - attention calculation on the input sequence to capture the relationships between various elements within the sequence. Taking the previous sentence as an example, the model will focus on the relationships between pronunciation units such as "我 (wǒ)", "喜 (xǐ)", "欢 (huān)", "音 (yīn)", "乐 (lè)", for instance, the pronunciation of "欢 (huān)" may be affected by "喜 (xǐ)" and "音 (yīn)".

[0081] Then, the feed - forward neural network performs a non - linear transformation on the output of the multi - head attention mechanism to further extract high - level features. This process is repeated in each layer. As the number of layers increases, the model can extract more abstract and high - level pronunciation features, which will be passed to the decoder for pronunciation correction.

[0082] The input to the decoder is a target sequence corresponding to the encoder input. In a pronunciation correction model, this target sequence is usually a certain representation corresponding to the speech to be processed, such as a sequence of phonemes or a sequence of word indices of the speech to be processed. The output of the decoder is the corrected pronunciation sequence, that is, the desired standard pronunciation. For example, for the previous non - native "我喜欢音乐 (Wǒ xǐhuān yīnyuè)", the input to the decoder may be the correct phoneme sequence of the native version sentence "我喜欢音乐 (Wǒ xǐhuān yīnyuè)", and the output is the sequence corresponding to the corrected native pronunciation. That is, the decoder includes discrete standard pronunciation units.

[0083] The decoder also uses the multi - head attention mechanism, but it not only focuses on its own input sequence (target sequence), but also on the output of the encoder. Through the cross - attention mechanism, the decoder can combine the pronunciation features of the speech to be processed extracted by the encoder and the standard pronunciation features of its own input to generate the corrected pronunciation. For example, when correcting a certain phoneme in the pronunciation of the speech to be processed, the decoder will refer to information such as the acoustic features of the pronunciation of this phoneme in the speech to be processed corresponding in the encoder, its relationship with other phonemes, etc., and at the same time combine the standard pronunciation to generate the correct standard pronunciation output.

[0084] During the generation process, the decoder converts the output of the last layer into a probability distribution through a linear layer and a softmax function. This probability distribution represents the probability of each possible pronunciation unit (such as a phoneme) appearing, and then selects the appropriate pronunciation unit according to the probability to gradually generate the complete corrected pronunciation sequence.

[0085] In one embodiment of the present invention, Figure 5 is a flow chart of step S60, such as Figure 5 As shown, step S60 includes:

[0086] Step S611, establishing a correspondence model between discrete standard pronunciation units and the standard speech corpus;

[0087] Step S612, combining the standard speech after the correction of each of the first pronunciations;

[0088] Step S613: input the combined standard speech into the corresponding relationship model to generate and output a complete standard speech signal.

[0089] It can be understood that the discrete standard pronunciation unit ultimately splits the complete speech into multiple discrete pronunciation units. Therefore, after the speech to be processed is passed through the discrete standard pronunciation unit to obtain the standard pronunciations corresponding to the speech to be processed, the standard pronunciations are discrete. In order to enable the user to obtain a smoother pronunciation, a correspondence model between the discrete standard pronunciation unit and the standard speech corpus can be established as in step S611. Since the speech of the standard speech corpus is continuous and fluent, in steps S612 and S613, after the standard speech after the first pronunciation is corrected is input into the correspondence model, the final generated completed standard speech signal is fluent and continuous.

[0090] In one embodiment of the present invention, Figure 6 is another flow chart of step S60, such as Figure 6 As shown, step S60 includes:

[0091] Step S621, inputting the speech to be converted into a speaker encoder to extract timbre information of the speech to be converted;

[0092] Step S622, combining the standard pronunciations after the correction of the first pronunciations;

[0093] Step S623, integrating the timbre information into the combined standard pronunciation to form a complete standard voice signal and output it.

[0094] In step S621, the speaker encoder is a module or model structure specially designed to extract speaker-related features from speech. Its core task is to analyze the input speech signal and then extract feature representations that can characterize information such as the speaker's unique timbre characteristics. The speaker encoder can be constructed based on deep learning methods, such as using deep neural networks, convolutional neural networks (CNNs), or their combined architectures. It is trained with a large amount of speech data to learn the pattern differences in acoustic features, spectral features, etc. of the speech of different speakers, so as to distinguish and extract the timbre information of the corresponding speaker. For example, for two people with different timbres saying the same sentence, the speaker encoder can capture subtle and distinguishing features such as fundamental frequency, resonance peak distribution, and tone length change pattern, thereby determining their respective timbre characteristics.

[0095] In step S623, a key goal of accent conversion is to preserve the speaker's identity characteristics while changing the voice accent, and timbre information is one of the important manifestations of the speaker's identity. If only pronunciation correction and combination are performed, the generated speech will meet the standards in terms of accent and content accuracy, but the timbre characteristics of the original speaker will be lost, and it will sound like a different person is speaking. By integrating the previously extracted timbre information into the combined standard pronunciation, it can be ensured that the final output speech has the correct standard accent and allows the audience to recognize that it is the original speaker who is expressing the content.

[0096] The means of integrating the timbre information can be any achievable means, such as using technical means related to speech synthesis to achieve the integration of timbre information. For example, in parametric speech synthesis, the parameters corresponding to the timbre information (such as parameters reflecting the fundamental frequency, formant and other characteristics) can be integrated with the speech synthesis parameters corresponding to the combined standard pronunciation. By adjusting these synthesis parameters, the generated speech signal has the timbre characteristics of the original speaker while having the standard pronunciation. For example, the acoustic features of the combined standard pronunciation can also be adjusted to match the extracted timbre information. For example, according to the sound frequency characteristics reflected in the timbre information, the frequency-related parameters of the standard pronunciation are appropriately scaled, offset, etc., so that the standard pronunciation is "stained" with the timbre characteristics of the original speaker. In this process, the degree and method of feature adjustment need to be carefully controlled to ensure that the timbre is integrated without destroying the accuracy of the standard pronunciation.

[0097] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0098] In one embodiment, an accent conversion device is provided, which corresponds one-to-one to the accent conversion method in the above embodiment. Figure 7 As shown, the accent conversion device includes an analysis module 710, a training module 720, a determination module 730, a correction module 740 and a combination module 750. The detailed description of each functional module is as follows:

[0099] An analysis module 710 is used to obtain the speech to be converted, and analyze and obtain a first pronunciation set in the speech to be converted;

[0100] The training module 720 is used to obtain standard speech corpus, input the standard speech corpus into the pronunciation correction model for training, and form a mapping relationship between discrete standard pronunciation units and standard pronunciation in the pronunciation correction model;

[0101] A determination module 730, configured to input the speech to be converted into a pronunciation correction model, and determine a discrete standard pronunciation unit corresponding to the first pronunciation in the first pronunciation set;

[0102] A correction module 740, configured to correct the first pronunciation to a standard pronunciation corresponding to the discrete standard pronunciation unit based on the discrete standard pronunciation unit;

[0103] The combining module 750 is used to combine the standard pronunciations corrected by the first pronunciations to form a complete standard speech signal and output it.

[0104] In one embodiment, the training module 720 is specifically used to:

[0105] Obtain standard voice corpus;

[0106] Inputting the standard speech corpus into a pre-training model for training, and obtaining a set of discrete speech units corresponding to the standard speech corpus through training;

[0107] Clustering each discrete speech unit in the discrete speech unit combination to obtain a discrete speech unit center set;

[0108] Taking each discrete speech unit center in the discrete speech unit center set as a discrete standard pronunciation unit to generate a discrete standard pronunciation unit set;

[0109] Extracting the standard pronunciation corresponding to each discrete standard pronunciation unit in the discrete standard pronunciation unit set, and establishing a mapping relationship between the discrete standard pronunciation unit and the standard pronunciation;

[0110] According to the mapping relationship between each discrete standard pronunciation unit and the standard pronunciation in the discrete standard pronunciation unit set, a pronunciation correction model including each discrete standard pronunciation unit in the discrete standard pronunciation unit set is generated.

[0111] In one embodiment, the training module 720 is further configured to:

[0112] Acquire sample information and generate non-standard speech sets of different speakers corresponding to the sample information;

[0113] Inputting each non-standard voice in the non-standard voice set into the pronunciation correction model for training, and outputting the standard pronunciation corresponding to the non-standard voice;

[0114] Inputting the standard pronunciation into the loss function of the pronunciation correction model to calculate the loss value of the standard pronunciation;

[0115] When the loss value is within the preset error range, a final pronunciation correction model is generated; when the loss value exceeds the preset error range, the parameters of the pronunciation correction model are changed and the pronunciation correction model is retrained until the loss value is within the preset error range.

[0116] In one embodiment, the combination module 750 is specifically used for:

[0117] Establishing a corresponding relationship model between discrete standard pronunciation units and the standard speech corpus;

[0118] combining the standard speech corrected by the first pronunciations;

[0119] The combined standard speech is input into the corresponding relationship model to generate a complete standard speech signal and output it.

[0120] In one embodiment, the training module 720 is further configured to:

[0121] Acquire sample information, and generate non-standard speech subsets of different speakers corresponding to the sample information;

[0122] The non-standard speech in the non-standard speech subset is overlapped and / or combined to generate a non-standard speech set.

[0123] In one embodiment, the pronunciation correction model is an encoder-decoder structure under a transformer architecture.

[0124] The present invention provides an accent conversion device, which converts the speech to be converted into a first pronunciation set. For each first pronunciation in the first pronunciation set, a discrete standard pronunciation unit corresponding to each standard pronunciation is set in the pronunciation correction model, so that each first pronunciation is corrected by using the discrete standard pronunciation unit to correct it to a standard pronunciation, so that each first pronunciation in the speech to be converted can be corrected to a standard pronunciation. Obviously, this scheme does not adopt the scheme of adding reference speech in the conversion process in the prior art, so it avoids the influence of factors such as the quality and duration of the original speech on the accuracy of accent conversion. In addition, since the present application converts the first pronunciation of the speech to be converted into a standard pronunciation, even if there are multiple types of the first pronunciation to be converted, it will still be converted into a standard pronunciation in the end to obtain the standard pronunciation of the speech to be converted, which avoids the problem that a large amount of training work and storage resources are required to maintain a one-to-one system in the prior art when multiple accent types need to be converted. In summary, this scheme can solve the problems existing in the prior art.

[0125] The specific definition of the accent conversion device can be found in the definition of the accent conversion method above, which will not be repeated here. Each module in the above-mentioned accent conversion device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0126] Based on the above accent conversion method, such as Figure 8 As shown, an embodiment of the present invention further provides a structural schematic diagram of an accent conversion device, the conversion device includes a processor 81 and a memory 82 coupled to the processor 81. The memory 82 stores a computer program, and when the computer program is executed by the processor 81, the processor 81 executes the steps of the accent conversion method in the above embodiment.

[0127] For other details about how the processor 81 in the above-mentioned accent conversion device implements the above-mentioned technical solution, please refer to the description of the accent conversion method provided in the above-mentioned embodiment of the invention, which will not be repeated here.

[0128] Among them, the processor 81 can also be called a CPU (Central Processing Unit), and the processor 81 may be an integrated circuit chip with signal processing capabilities; the processor 81 can also be a general-purpose processor, DSP (Digital Signal Process), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, among which the general-purpose processor can be a microprocessor or the processor 81 can also be any conventional processor, etc.

[0129] like Fig. 9 As shown, the embodiment of the present invention also provides a schematic diagram of the structure of a computer-readable storage medium, on which a readable computer program 91 is stored; wherein the computer program 91 can be stored in the above storage medium in the form of a software product, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, magnetic disk or optical disk, ROM (Read-Only Memory), RAM (Random Access Memory) and other media that can store program codes, or terminal devices such as computers, servers, mobile phones, and tablets.

[0130] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0131] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0132] In addition, each functional module in each embodiment of the present invention may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may be stored in a computer-readable storage medium.

[0133] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0134] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium), or a semiconductor medium (e.g., an SSD (solid state disk, solid state hard disk)).

[0135] The technical solution provided by the present invention is introduced in detail above. The present invention applies specific examples to illustrate the principle and implementation mode of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation mode and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

[0136] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, optical storage, etc.) containing computer-usable program code.

[0137] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0138] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0140] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for accent conversion, characterized in that: include: Acquire the speech to be converted, and analyze to obtain a first pronunciation set in the speech to be converted; Acquire standard speech corpus, input the standard speech corpus into a pronunciation correction model for training, and form a mapping relationship between discrete standard pronunciation units and standard pronunciation in the pronunciation correction model; Inputting the speech to be converted into a pronunciation correction model to determine a discrete standard pronunciation unit corresponding to the first pronunciation in the first pronunciation set; Based on the discrete standard pronunciation unit, correcting the first pronunciation to the standard pronunciation corresponding to the discrete standard pronunciation unit; The standard pronunciations corrected by the first pronunciations are combined to form a complete standard voice signal and output it.

2. The accent conversion method according to claim 1, characterized in that: The step of acquiring standard speech corpus, inputting the standard speech corpus into a pronunciation correction model for training, and forming a mapping relationship between the discrete standard pronunciation units and the standard pronunciation in the pronunciation correction model includes: Obtain standard voice corpus; Inputting the standard speech corpus into a pre-training model for training, and obtaining a set of discrete speech units corresponding to the standard speech corpus through training; Clustering each discrete speech unit in the discrete speech unit combination to obtain a discrete speech unit center set; Taking each discrete speech unit center in the discrete speech unit center set as a discrete standard pronunciation unit to generate a discrete standard pronunciation unit set; Extracting the standard pronunciation corresponding to each discrete standard pronunciation unit in the discrete standard pronunciation unit set, and establishing a mapping relationship between the discrete standard pronunciation unit and the standard pronunciation; According to the mapping relationship between each discrete standard pronunciation unit and the standard pronunciation in the discrete standard pronunciation unit set, a pronunciation correction model including each discrete standard pronunciation unit in the discrete standard pronunciation unit set is generated.

3. The accent conversion method according to claim 1, characterized in that: After acquiring the standard speech corpus, inputting the standard speech corpus into the pronunciation correction model for training, and forming a mapping relationship between the discrete standard pronunciation unit and the standard pronunciation in the pronunciation correction model, and before correcting the first pronunciation to the standard pronunciation corresponding to the discrete standard pronunciation unit based on the discrete standard pronunciation unit, the method further includes: Acquire sample information and generate non-standard speech sets of different speakers corresponding to the sample information; Inputting each non-standard voice in the non-standard voice set into the pronunciation correction model for training, and outputting the standard pronunciation corresponding to the non-standard voice; Inputting the standard pronunciation into the loss function of the pronunciation correction model to calculate the loss value of the standard pronunciation; When the loss value is within the preset error range, a final pronunciation correction model is generated; when the loss value exceeds the preset error range, the parameters of the pronunciation correction model are changed and the pronunciation correction model is retrained until the loss value is within the preset error range.

4. The accent conversion method according to claim 1, characterized in that: The combining the corrected standard pronunciations of the first pronunciations to form a complete standard speech signal and outputting the signal comprises: Establishing a corresponding relationship model between discrete standard pronunciation units and the standard speech corpus; combining the standard speech corrected by the first pronunciations; The combined standard speech is input into the corresponding relationship model to generate a complete standard speech signal and output it.

5. The accent conversion method according to claim 1, characterized in that: The combining the corrected standard pronunciations of the first pronunciations to form a complete standard speech signal and outputting the signal comprises: Inputting the speech to be converted into a speaker encoder to extract timbre information of the speech to be converted; combining the standard pronunciations corrected by the first pronunciations; The timbre information is integrated into the combined standard pronunciation to form a complete standard voice signal and output it.

6. The accent conversion method according to claim 3, characterized in that: The step of acquiring sample information and generating a set of non-standard voices of different speakers corresponding to the sample information includes: Acquire sample information, and generate non-standard speech subsets of different speakers corresponding to the sample information; The non-standard speech in the non-standard speech subset is overlapped and / or combined to generate a non-standard speech set.

7. The accent conversion method according to claim 1, characterized in that: The pronunciation correction model is an encoder-decoder structure under the transformer architecture.

8. An accent conversion device, characterized in that: include: An analysis module, used for acquiring the speech to be converted, and analyzing to obtain a first pronunciation set in the speech to be converted; A training module, used for acquiring standard speech corpus, inputting the standard speech corpus into a pronunciation correction model for training, and forming a mapping relationship between discrete standard pronunciation units and standard pronunciation in the pronunciation correction model; A determination module, configured to input the speech to be converted into a pronunciation correction model, and determine a discrete standard pronunciation unit corresponding to the first pronunciation in the first pronunciation set; A correction module, configured to correct the first pronunciation to a standard pronunciation corresponding to the discrete standard pronunciation unit based on the discrete standard pronunciation unit; The combining module is used to combine the standard pronunciations corrected by the first pronunciations to form a complete standard voice signal and output it.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the accent conversion method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the accent conversion method according to any one of claims 1 to 7 are implemented.

Images