An IN and MI based voice conversion method, system, terminal and storage medium

Abstract

The application discloses a speech conversion method and system based on IN and MI, a terminal and a storage medium. The method comprises the following steps: obtaining the mel spectrum of source speech data; inputting the mel spectrum of the source speech data into a pre-trained speech conversion model based on IN and MI; using the IN algorithm to obtain the text information embedding corresponding to the source speech data, using an encoder to obtain the speaker information embedding related to the speaker, and using the MI algorithm to measure the correlation degree of the text information embedding and the speaker information embedding; and generating speech data corresponding to the target speaker's voice based on the text information embedding and the speaker information embedding through a decoder. The application can further strengthen the feature decoupling function of speech features and speaker features, achieve better speech conversion effect, better perform personalized speech synthesis tasks such as short video dubbing, and expand the application scenarios of the speech conversion model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent speech, in particular to a voice conversion method and system based on IN and MI, a terminal and a storage medium. BACKGROUND

[0002] Voice conversion technology refers to converting the identity information of a source speaker into the identity information of a target speaker while keeping the voice content unchanged, and has very important applications in medical services, secure communications and life entertainment. In real life, voice conversion can play a strong application value in many fields, such as voice navigation, converting navigation voice into the favorite star voice of the driver according to the personal preference of the driver, and realizing personalized voice generation in line with user preferences. At present, the popular methods in the field of voice conversion include IN (instance normalization) algorithm or GAN, and the voice conversion effect of the existing methods needs to be further improved.

[0003] With the development of self-made short videos, at present, many self-media people have strong video creation inspiration and can create many interesting animated short videos. However, due to reasons such as lack of funds and insufficient manpower, self-media people are difficult to hire professional voice actors to voice for self-made animated short videos, and can only use silent animation or machine-synthesized sound to voice for animated short videos, which cannot produce diverse personalized customized voice, so that the final effect of the video is greatly discounted, greatly affecting the development of the short video field. SUMMARY

[0004] The present application provides a voice conversion method and system based on IN and MI, a terminal and a storage medium, aiming to solve the technical problems that the voice conversion effect of the existing voice conversion method needs to be improved and the voice of the animated short video cannot produce diverse personalized customized voice.

[0005] To solve the above technical problems, the technical scheme adopted by the present application is as follows:

[0006] A voice conversion method based on IN and MI, comprising:

[0007] obtaining the mel spectrum of the source voice data;

[0008] inputting the mel spectrum of the source voice data into a pre-trained voice conversion model based on IN and MI, the voice conversion model using the IN algorithm to obtain the text information embedding corresponding to the source voice data, using the encoder to obtain the speaker information embedding related to the speaker, and using the MI algorithm to measure the correlation degree of the text information embedding and the speaker information embedding;

[0009] Based on the text information embedding and the speaker information embedding, voice data corresponding to a target speaker's tone is generated through a decoder.

[0010] The technical scheme adopted by the embodiment of the present application further includes that the mel spectrum of the source voice data includes:

[0011] The short-time Fourier transform algorithm is adopted to convert the source voice data to obtain the mel spectrum of the source voice data.

[0012] The technical scheme adopted by the embodiment of the present application further includes that the conversion of the source voice data through the short-time Fourier transform algorithm to obtain the mel spectrum of the source voice data includes:

[0013] The source voice data X, the window function type, the window length WinLen, the overlap point number OverLap, the sampling frequency and the Fourier point number are determined.

[0014] The window sliding number and the signal length nx are calculated, and the number n of window sliding is calculated according to the signal length nx, the window length WinLen and the overlap point number OverLap between windows.

[0015] The value of each column is determined to obtain a matrix with n columns and WinLen rows.

[0016] The window function converted into a column vector is expanded into a matrix w with n columns, the matrix w is point multiplied, and the point multiplication result is subjected to fast Fourier transform to obtain a time-frequency matrix.

[0017] Mel frequency conversion is performed according to the time-frequency matrix, and a mel spectrum is output; wherein the mel frequency conversion formula is: B(f)=1125ln(1+f / 700) wherein f is the frequency and B is the mel frequency.

[0018] The technical scheme adopted by the embodiment of the present application further includes that the voice conversion model based on IN and MI includes a first encoder, an IN module, a second encoder, a decoder and a classifier; the first encoder is used to extract a content vector containing and only containing content information from the mel spectrum of the source voice data; the IN module is used to filter out tone information from the mel spectrum to obtain a content vector irrelevant to a speaker; the second encoder is used to extract a speaker information embedding related to a speaker; the decoder is used to learn and generate the mel spectrum of the target speaker according to the content vector and the speaker information embedding; a gradient inversion layer is further included between the classifier and the first encoder, and the gradient inversion layer is used to make the gradient negative when backpropagating to the layer, so that the optimization objectives of the first encoder and the classifier are completely opposite.

[0019] The technical scheme adopted by the embodiment of the present application further includes that the training process of the voice conversion model based on IN and MI includes:

[0020] In the model training stage, taking the mel spectrum x of the source voice data as the input, the text information embedding (c) corresponding to the source voice data is obtained through the IN module, and the speaker information embedding (s) related to the speaker is obtained through the second encoder; the text information embedding (c) and the speaker information embedding (s) are input into the decoder to obtain the reconstructed voice x1, and the voice reconstruction task is realized, and the voice reconstruction task loss is L recon ‖x―x1‖1; at the same time, MI(c,s) is calculated and made smaller, and MI(c,s) represents the mutual information of the text information embedding (c) and the speaker information embedding (s).

[0021] The technical scheme adopted by the embodiment of the present application further includes that the calculation formula of the MI(c,s) is:

[0022]

[0023] Wherein, p(c) and p(s) represent the marginal probability respectively, and p(c,s) represents the joint probability distribution.

[0024] The technical scheme adopted by the embodiment of the present application further includes that the voice data corresponding to the target speaker's tone is generated by the decoder based on the text information embedding and the speaker information embedding.

[0025] The mel spectrum corresponding to the target speaker's tone is generated by the decoder based on the text information embedding and the speaker information embedding.

[0026] The mel spectrum corresponding to the target speaker's tone is converted by the vocoder, and the converted audio file is output.

[0027] Another technical scheme adopted by the embodiment of the present application is a voice conversion system based on IN and MI, which includes:

[0028] The mel spectrum extraction module is used to obtain the mel spectrum of the source voice data.

[0029] The voice conversion module is used to input the mel spectrum of the source voice data into the pre-trained voice conversion model based on IN and MI, the voice conversion model uses the IN algorithm to obtain the text information embedding corresponding to the source voice data, uses the encoder to obtain the speaker information embedding related to the speaker, uses the MI algorithm to measure the correlation degree of the text information embedding and the speaker information embedding, and generates the voice data corresponding to the target speaker's tone based on the text information embedding and the speaker information embedding through the decoder.

[0030] Another technical solution adopted by the embodiment of the present application is a terminal, comprising a processor and a memory coupled to the processor, wherein,

[0031] The memory stores program instructions for implementing the IN and MI based speech conversion method described above;

[0032] The processor is configured to execute the program instructions stored in the memory to perform the end-to-end IN and MI based speech conversion operation.

[0033] Another technical solution adopted by the embodiment of the present application is a storage medium storing program instructions executable by a processor, the program instructions being used to execute the IN and MI based speech conversion method described above.

[0034] The IN and MI based speech conversion method, system, terminal and storage medium of the embodiment of the present application utilize the combination of IN algorithm and MI algorithm to improve the speech conversion model, so that the IN algorithm and MI algorithm are combined to improve the speech conversion model, the text information embedding (c) corresponding to the source speech data is obtained through the IN algorithm, the speaker information embedding (s) related to the speaker is obtained through the encoder, and the correlation degree between the text information embedding and the speaker information embedding is measured by using the MI algorithm, so that the model training target is more reasonable by constraining the mutual information MI (c, s), and the feature decoupling function of the speech feature and the speaker feature can be further strengthened, better speech conversion effect is achieved, better performance is achieved in personalized speech synthesis tasks such as short video dubbing, and the application scenarios of the speech conversion model are expanded. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is a flowchart of the IN and MI based speech conversion method of the first embodiment of the present application;

[0036] Figure 2 is a flowchart of the IN and MI based speech conversion method of the second embodiment of the present application;

[0037] Figure 3 is a structure diagram of the IN and MI based speech conversion model of the embodiment of the present application;

[0038] Figure 4 is a system diagram of the IN and MI based speech conversion system of the embodiment of the present application;

[0039] Figure 5 is a terminal system diagram of the embodiment of the present application;

[0040] Figure 6It is a storage medium system schematic diagram of the embodiment of the present application. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0042] The terms "first", "second", "third" in the present application are only for descriptive purpose, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second", "third" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise explicitly and specifically limited. All directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present application are only used to explain the relative position relationship, movement condition, etc. between components in a certain posture (as shown in the drawings), and if the certain posture changes, the directional indications also change accordingly. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product or device.

[0043] In this document, reference to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive of one another. As will be apparent to those of ordinary skill in the art, embodiments described herein can be combined with other embodiments.

[0044] Please refer to Figure 1 It is a flowchart of the IN and MI based speech conversion method of the first embodiment of the present application. The IN and MI based speech conversion method of the first embodiment of the present application includes the following steps:

[0045] S100: Obtain the mel spectrum of the source speech data;

[0046] S101: input the mel spectrum of the source speech data into the pre-trained IN and MI based speech conversion model, the speech conversion model uses the IN algorithm to obtain the text information embedding corresponding to the source speech data, uses an encoder to obtain the speaker information embedding related to the speaker, and uses the MI algorithm to measure the correlation degree of the text information embedding and the speaker information embedding;

[0047] S102: generate speech data corresponding to the target speaker's tone of voice based on the text information embedding and the speaker information embedding through a decoder.

[0048] Referring to Figure 2 is a flowchart of the IN and MI based speech conversion method of the second embodiment of the present application. The IN and MI based speech conversion method of the second embodiment of the present application comprises the following steps:

[0049] S200: obtain source speech data to be converted;

[0050] S201: convert the source speech data using the STFT (short-time Fourier transform) algorithm to obtain the mel spectrum of the source speech data;

[0051] In this step, the audio data is composed of several single-frequency sound waves, that is, when sampling the signal in a period of time, only the amplitude can be captured. The STFT algorithm is used to window the source speech data, then segment and perform FFT (fast Fourier transform), the amplitude spectrum of the windowed audio data is calculated using the FFT algorithm, and the amplitude spectrum is converted through a mel filter bank of a certain dimension, and the result obtained is the mel spectrum x of the source speech data. The FFT is a high-efficiency algorithm of DFT (Discrete Fourier Transform), the amplitude spectrum represents the distribution of the amplitude of the signal with frequency, and after the audio data is transformed into the frequency domain using the FFT algorithm, the amplitude spectrum can be obtained by performing a modulo operation on the speech data.

[0052] Specifically, the process of obtaining the mel spectrum using the STFT specifically comprises:

[0053] First step: determine the source speech data X, the window function type, the window length WinLen, the overlap point number OverLap, the sampling frequency, the Fourier point number and other related parameters;

[0054] Second step: calculate the window sliding number and the signal length nx, and calculate the number of window sliding n according to the signal length nx, the window length WinLen and the overlap point number OverLap between the windows.

[0055] The third step is to determine the value of each column to obtain a matrix with n columns and WinLen rows;

[0056] The fourth step is to expand the window function converted into a column vector into an n-column matrix w, multiply the matrix w, and perform a fast Fourier transform on the multiplication result to obtain a time-frequency matrix; wherein, during the Fourier transformation process, when the signal length is less than the Fourier point number, the algorithm will automatically perform zero padding, and then perform a fast Fourier transform;

[0057] The fifth step is to perform a Mel frequency conversion according to the time-frequency matrix and output a Mel spectrum; wherein, the Mel frequency conversion formula is: B(f) = 1125ln(1+f / 700) wherein f is the frequency, and B is the Mel frequency.

[0058] S202: input the Mel spectrum of the source speech data into the pre-trained speech conversion model, the speech conversion model uses the IN algorithm to obtain the text information embedding corresponding to the source speech data, uses the encoder to obtain the speaker information embedding related to the speaker, and uses the MI algorithm to measure the correlation degree of the text information embedding and the speaker information embedding;

[0059] In this step, the speech conversion model is an IN (instance normalization) and MI (Mutual Information) based speech conversion model, the IN algorithm is a data normalization technology, when the algorithm is applied to the style transfer task, the static information related to the speaker can be removed. The MI algorithm is used to measure the correlation degree of two distributions, the smaller the MI(X, Y), the stronger the independence of the distribution X and the distribution Y, when X and Y are independent of each other, MI(X, Y) reaches the minimum value 0.

[0060] Specifically, as Figure 3Fig. 1 shows a structure of an IN and MI based speech conversion model according to an embodiment of the present application. The IN and MI based speech conversion model according to an embodiment of the present application comprises a first encoder, an IN module, a second encoder, a decoder and a classifier. The first encoder is configured to extract a content embedding from a mel-spectrogram of source speech data, the content embedding containing and only containing content information. The IN module is configured to filter out timbre information in the source speech data to obtain a latent code that is as speaker-independent as possible, the latent code being the content embedding. The second encoder is configured to extract a speaker embedding related to a speaker to supplement the timbre information missing in the latent code. The decoder is configured to learn to generate a mel-spectrogram X' of a target speaker based on the content embedding and the speaker embedding. A GRL layer is further included between the classifier and the first encoder. The GRL layer can cause the gradient to be negative when backpropagated to the layer, so that the optimization objectives of the first encoder and the classifier are completely opposite. That is, the classifier wants to accurately classify the speaker identity from the content embedding, while the first encoder wants to generate a content embedding that is independent of the speaker to deceive the classifier.

[0061] In an embodiment of the present application, the training process of the IN and MI based speech conversion model comprises: the model input is a mel-spectrogram x of source speech data, the text information embedding c corresponding to the source speech data is obtained through the IN module, the speaker information embedding s related to the speaker is obtained through the second encoder, and then the text information embedding c and the speaker information embedding s are input into the decoder to obtain reconstructed speech x1, so as to realize a speech reconstruction task in the training stage. The loss of the speech reconstruction task is L recon =‖x―x1‖1. At the same time, MI(c,s) is calculated and made smaller. MI(c,s) represents the mutual information between the text information embedding c and the speaker information embedding s, and is used to measure the degree of correlation between the text information embedding c and the speaker information embedding s. The smaller MI(c,s) is, the more independent the text information embedding c and the speaker information embedding s are, that is, the lower the correlation between the text information embedding c and the speaker information embedding s is. The calculation formula of MI(c,s) is:

[0062]

[0063] Wherein, p(c) and p(s) represent edge probabilities respectively, and p(c,s) is a joint probability distribution.

[0064] S203: Based on the text information embedding and the speaker information embedding, a mel spectrum corresponding to the target speaker's voice is generated through a decoder.

[0065] S204: The mel spectrum corresponding to the target speaker's voice is converted using a vocoder, and a converted audio file is output.

[0066] Based on the above, the IN and MI based speech conversion method of the embodiment of the application improves the speech conversion model by combining the IN algorithm and the MI algorithm, obtains the text information embedding (c) corresponding to the source speech data through the IN algorithm, obtains the speaker information embedding (s) related to the speaker through the encoder, and measures the correlation degree of the text information embedding and the speaker information embedding using the MI algorithm, so that the model training target is more reasonable through the constraint mutual information MI(c,s), and the feature decoupling function of the speech features and the speaker features can be further strengthened, better speech conversion effect is achieved, and the application scenarios of the speech conversion model are expanded.

[0067] In an optional implementation, the result of the IN and MI based speech conversion method can also be uploaded to a blockchain.

[0068] Specifically, corresponding summary information is obtained based on the result of the IN and MI based speech conversion method, and specifically, the summary information is obtained by hashing the result of the IN and MI based speech conversion method, for example, by using the sha256s algorithm. Uploading the summary information to the blockchain can ensure its security and fair transparency to users. Users can download the summary information from the blockchain to check whether the result of the IN and MI based speech conversion method has been tampered with. The blockchain referred to in this example is a new application mode of computer technology such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm. Blockchain, in essence, is a decentralized database, which is a series of data blocks associated using cryptographic methods, each of which contains information about a batch of network transactions, used to verify the validity (anti-fake) of the information and generate the next block. The blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer.

[0069] Please refer to Figure 4 is a system schematic diagram of the end-to-end IN and MI based speech conversion system of the embodiment of the application. The end-to-end IN and MI based speech conversion system 40 of the embodiment of the application comprises:

[0070] The source data acquisition module 41 is configured to acquire source voice data to be converted.

[0071] The Mel spectrum extraction module 42 is configured to convert the source voice data by using an STFT (short-time Fourier transform) algorithm to acquire a Mel spectrum of the source voice data. The audio data is composed of several single-frequency sound waves, that is, only the amplitude can be captured when the signal is sampled in a period of time. The STFT algorithm is used to window the source voice data, and then the FFT (fast Fourier transform) algorithm is used to segment and calculate the amplitude spectrum of the windowed audio data. The amplitude spectrum is converted by a Mel filter bank of a certain dimension, and the result is the Mel spectrum x of the source voice data. The Mel spectrum x is a spectrum graph converted from frequency to Mel scale. The FFT is a high-efficiency algorithm of DFT (Discrete Fourier Transform), and the amplitude spectrum represents the distribution of the amplitude of the signal with frequency. After the audio data is transformed to the frequency domain by using the FFT algorithm, the amplitude spectrum can be acquired by performing a modulo operation on the voice data.

[0072] Specifically, the process of acquiring the Mel spectrum by using the STFT includes the following steps.

[0073] Step 1: Determine the related parameters of the source voice data X, the type of window function, the window length WinLen, the number of overlapping points OverLap, the sampling frequency, the number of Fourier points, and the like.

[0074] Step 2: Calculate the number of window sliding times and the signal length nx, and calculate the number of times n of window sliding according to the signal length nx, the window length WinLen, and the number of overlapping points OverLap between the windows.

[0075] Step 3: Determine the value of each column to obtain a matrix with n columns and WinLen rows.

[0076] Step 4: Expand the window function converted into a column vector into a matrix w with n columns, perform point multiplication on the matrix w, and perform fast Fourier transform on the point multiplication result to obtain a time-frequency matrix. During the Fourier transformation, when the signal length is less than the number of Fourier points, the algorithm automatically performs zero padding, and then performs fast Fourier transform.

[0077] Step 5: Perform Mel frequency conversion according to the time-frequency matrix, and output the Mel spectrum. The Mel frequency conversion formula is B(f) = 1125ln(1+f / 700), where f is the frequency and B is the Mel frequency.

[0078] The voice conversion module 43 is configured to input the mel spectrum of the source voice data into a pre-trained voice conversion model. The voice conversion model is configured to obtain a text information embedding corresponding to the source voice data by using an IN algorithm, obtain a speaker information embedding related to a speaker by using an encoder, and measure a correlation degree between the text information embedding and the speaker information embedding by using an MI algorithm. The voice conversion model is an IN (instance normalization) and MI (Mutual Information) based voice conversion model. The IN algorithm is a data normalization technology. When the IN algorithm is applied to a style transfer task, static information related to the speaker can be removed. The MI algorithm is configured to measure the correlation degree between two distributions. The smaller the MI (X, Y) is, the stronger the independence between the distribution X and the distribution Y is. When the distribution X and the distribution Y are independent of each other, the MI (X, Y) reaches a minimum value 0.

[0079] Specifically, as shown in the following table, the voice conversion model based on the IN and the MI includes a first encoder, an IN module, a second encoder, a decoder, and a classifier. Figure 3 The first encoder is configured to extract a content embedding containing and only containing content information from the mel spectrum of the source voice data. The IN module is configured to filter out the timbre information in the source voice data to obtain a latent code irrelevant to the speaker as much as possible, that is, the content embedding. The second encoder is configured to extract a speaker information embedding related to the speaker to supplement the timbre information missing in the latent code. The decoder is configured to learn and generate the mel spectrum X' of the target speaker according to the content embedding and the speaker information embedding. A GRL layer is further included between the classifier and the first encoder. The GRL layer can make the gradient negative when the gradient is back propagated to the layer, so that the optimization objectives of the first encoder and the classifier are completely opposite. That is, the classifier hopes to accurately classify the speaker identity from the content embedding, while the first encoder hopes to generate the content embedding irrelevant to the speaker to deceive the classifier.

[0080] In the embodiment of the application, the IN and MI based speech conversion model training process includes: the model input is the mel spectrum x of the source speech data, the text information embedding (c) corresponding to the source speech data is obtained through the IN module, the speaker information embedding (s) related to the speaker is obtained through the second encoder, and then the text information embedding (c) and the speaker information embedding (s) are input into the decoder to obtain the reconstructed speech x1, so as to realize a speech reconstruction task in the training stage, and the speech reconstruction task loss is L recon ‖x―x1‖1. At the same time, MI (c, s) is calculated and made smaller, MI (c, s) represents the mutual information of the text information embedding (c) and the speaker information embedding (s), and is used to measure the correlation degree between the text information embedding (c) and the speaker information embedding (s), the smaller MI (c, s) is, the more independent the text information embedding (c) and the speaker information embedding (s) are, that is, the lower the correlation of the text information embedding (c) and the speaker information embedding (s) is. The calculation formula of MI (c, s) is:

[0081]

[0082] Wherein, p (c) and p (s) represent edge probabilities respectively, and p (c, s) is a joint probability distribution.

[0083] The mel spectrum conversion module 44 is configured to convert the mel spectrum corresponding to the target speaker's voice tone by using a vocoder, and output a converted audio file.

[0084] Based on the above, the IN and MI based speech conversion system of the second embodiment of the application improves the speech conversion model by combining the IN algorithm and the MI algorithm, obtains the text information embedding (c) corresponding to the source speech data through the IN algorithm, obtains the speaker information embedding (s) related to the speaker through the encoder, and measures the correlation degree of the text information embedding and the speaker information embedding by using the MI algorithm, so as to make the model training target more reasonable by constraining the mutual information MI (c, s), and at the same time, the feature decoupling function of the speech feature and the speaker feature can be further strengthened, a better speech conversion effect is realized, the personalized speech synthesis task such as the short video dubbing of animation can be better performed, and the application scene of the speech conversion model is expanded.

[0085] Please refer to Figure 5 , which is a schematic diagram of a terminal system according to an embodiment of the application. The terminal 50 includes a processor 51 and a memory 52 coupled to the processor 51.

[0086] The memory 52 stores program instructions for implementing the above-mentioned IN and MI based speech conversion method.

[0087] The processor 51 is configured to execute program instructions stored in the memory 52 to perform the end-to-end IN and MI based speech conversion operation.

[0088] The processor 51 can also be referred to as a CPU (Central Processing Unit). The processor 51 can be an integrated circuit chip having a processing capability of signals. The processor 51 can also be a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application-Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0089] Referring to Figure 5 , Figure 5 The system schematic diagram of the storage medium of the embodiment of the present application is shown in FIG. 1. The storage medium of the embodiment of the present application stores a program file 61 capable of realizing all the methods described above, wherein the program file 61 can be stored in the storage medium in the form of a software product, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method of each embodiment of the present application. The storage medium described above includes a U disk, a mobile hard disk, a ROM (Read-Only Memory), a RAM (Random Access Memory), a magnetic disk or an optical disk, and various media capable of storing program codes, or a terminal device such as a computer, a server, a mobile phone, a tablet, etc.

[0090] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the system embodiments described above are merely schematic, for example, the division of units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, units or components, and can be electrical, mechanical or other forms.

[0091] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit. The above is only an implementation manner of the present application, and does not limit the patent scope of the present application. Any equivalent system or equivalent process transformation made by using the content of the present application specification and drawings, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A method for voice conversion based on IN and MI, characterized by, The method comprises: obtaining the mel spectrum of source voice data; inputting the mel spectrum of the source voice data into a pre-trained IN and MI based voice conversion model, wherein the voice conversion model uses an IN algorithm to obtain a text information embedding corresponding to the source voice data, uses an encoder to obtain a speaker information embedding related to a speaker, and uses an MI algorithm to measure the correlation between the text information embedding and the speaker information embedding; the IN and MI based voice conversion model comprises a first encoder, an IN module, a second encoder, a decoder and a classifier; the first encoder is used to extract a content vector containing and only containing content information from the mel spectrum of the source voice data; the IN module is used to filter out timbre information from the mel spectrum to obtain a content vector unrelated to a speaker; the second encoder is used to extract a speaker information embedding related to a speaker; the decoder is used to learn and generate the mel spectrum of a target speaker according to the content vector and the speaker information embedding; a gradient reversal layer is further included between the classifier and the first encoder, and the gradient reversal layer is used to make the gradient negative when backpropagating to the layer, so that the optimization objectives of the first encoder and the classifier are completely opposite; The training process of the IN and MI based speech conversion model comprises: in the model training stage, taking the mel-frequency spectrum of the source speech data as input, obtaining the text information embedding corresponding to the source speech data through the IN module , obtaining the speaker information embedding related to the speaker through the second encoder ; inputting the text information embedding and the speaker information embedding into the decoder to obtain the reconstructed speech , realizing the speech reconstruction task, and the loss of the speech reconstruction task is ; meanwhile, MI(c,s) is calculated and made smaller, wherein MI(c,s) represents the mutual information of the text information embedding and the speaker information embedding . ​ based on the text information embedding and the speaker information embedding, generating voice data corresponding to the timbre of the target speaker through the decoder; based on the result of the IN and MI based voice conversion method, obtaining corresponding summary information, wherein the summary information is obtained by hashing the result of the IN and MI based voice conversion method, and the summary information is uploaded to a blockchain.

2. The IN and MI based voice conversion method of claim 1, wherein, The method comprises: using a short-time Fourier transform algorithm to convert the source voice data to obtain the mel spectrum of the source voice data.

3. The IN and MI based voice conversion method of claim 2, wherein, The method comprises: determining the source voice data X, the window function type, the window length WinLen, the overlap point number OverLap, the sampling frequency and the Fourier point number; calculating the window sliding number and the signal length nx, and calculating the number of window sliding n according to the signal length nx, the window length WinLen and the overlap point number OverLap between windows; determining the value of each column to obtain a matrix with n columns and WinLen rows; expanding the window function converted into a column vector into a matrix w with n columns, point multiplying the matrix w, and performing a fast Fourier transform on the point multiplication result to obtain a time-frequency matrix; performing mel frequency conversion according to the time-frequency matrix and outputting the mel spectrum; wherein the mel frequency conversion formula is: B(f)=1125ln(1+f / 700) wherein f is the frequency and B(f) is the mel frequency.

4. The IN and MI based voice conversion method of claim 1, wherein, The calculation formula of the MI(c,s) is: MI(c,s) = 1 - e~(c / s)2 wherein, and denote marginal probabilities, (c,s) denote joint probability distributions.

5. The IN and MI based voice conversion method according to any one of claims 1 to 4, characterized by, The method comprises: Based on the text information embedding and the speaker information embedding, a mel spectrum corresponding to a target speaker voice is generated by a decoder; The mel spectrum corresponding to the target speaker voice is converted by a vocoder, and a converted audio file is output.

6. An IN and MI based voice conversion system for implementing the IN and MI based voice conversion method according to any one of claims 1-5, characterized by Comprise: A mel spectrum extraction module is configured to obtain a mel spectrum of source speech data; A speech conversion module is configured to input the mel spectrum of the source speech data into a pre-trained IN and MI based speech conversion model, the speech conversion model is configured to obtain a text information embedding corresponding to the source speech data by using an IN algorithm, obtain a speaker information embedding related to a speaker by using an encoder, measure a correlation degree between the text information embedding and the speaker information embedding by using an MI algorithm, and generate speech data corresponding to a target speaker voice based on the text information embedding and the speaker information embedding by using a decoder.

7. A terminal, characterized by comprising: The terminal comprises a processor and a memory coupled to the processor, wherein The memory stores program instructions for implementing the IN and MI based speech conversion method of any one of claims 1 to 5; The processor is configured to execute the program instructions stored in the memory to perform the IN and MI based speech conversion method.

8. A storage medium, characterized by The memory stores processor executable program instructions for executing the IN and MI based speech conversion method of any one of claims 1 to 5.

Images