Sound conversion method and device based on artificial intelligence, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a sound conversion method and device based on artificial intelligence, equipment and a medium. According to the method, the pitch feature of the source speaker is difficult to obtain, the frame number of the pitch feature is determined, the timbre feature extraction and the emotion feature of the target speaker are obtained, and the frame number of the timbre feature and the frame number of the emotion feature are expanded to be equal to the frame number of the pitch feature; and aligning the pitch feature, the expanded timbre feature and the expanded emotion feature, fusing the aligned features to obtain a first fusion feature, extracting the source voice content, fusing the source voice content with the first fusion feature to obtain a second fusion feature, and obtaining the converted voice according to the second fusion feature. According to the method and the device, in the voice conversion process, the emotion information and the pitch information corresponding to the target speaker are fused into the content information of the source speaker, so that the obtained converted target voice can better reflect the voice of the target speaker, and the voice conversion effect is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an artificial intelligence-based sound conversion method, device, equipment and medium. Background Art

[0002] Speech conversion is a speech technology that retains the content information of the source speaker's speech and converts it into the target speaker's voice. This technology has a wide range of application scenarios, such as "voice-changing bows". In addition, the development of speech conversion technology is of great significance to the fields of personalized speech synthesis, voiceprint recognition, and voiceprint security. In the prior art, the existing speech conversion method is to transfer one person's voice to another person's speech content without emotion, making the converted speech effect poor. Therefore, in the process of speech conversion, how to improve the conversion effect has become an urgent problem to be solved. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a method, device, equipment and medium for voice conversion based on artificial intelligence to solve the problem of poor voice effect after conversion during the voice conversion process.

[0004] In a first aspect, an embodiment of the present application provides a sound conversion method based on artificial intelligence, the sound conversion method comprising: Acquire a source speech of a source speaker, extract a fundamental frequency in the source speech, perform pitch feature extraction on the fundamental frequency to obtain a pitch feature, and determine the number of frames of the pitch feature; Acquire a target speech of a target speaker, extract Mel spectrum information in the target speech to obtain a target Mel spectrum, perform timbre feature extraction on the target Mel spectrum to obtain a timbre feature, perform emotion feature extraction on the target Mel spectrum to obtain an emotion feature; Expanding the number of frames of the timbre feature and the emotional feature to be equal to the number of frames of the pitch feature, to obtain an expanded timbre feature and an expanded emotional feature; Aligning the pitch feature, the expanded timbre feature, and the expanded emotion feature, and fusing them to obtain a first fused feature; Speech content is extracted from the source speech to obtain source speech content, the source speech content is fused with the first fusion feature to obtain a second fusion feature, spectrum is reconstructed on the second fusion feature to obtain a reconstructed Mel spectrum, and converted speech is obtained according to the reconstructed Mel spectrum.

[0005] In a second aspect, an embodiment of the present application provides a sound conversion device based on artificial intelligence, the sound conversion device comprising: A first acquisition module is used to acquire a source speech of a source speaker, extract a fundamental frequency in the source speech, perform pitch feature extraction on the fundamental frequency to obtain a pitch feature, and determine a frame number of the pitch feature; The second acquisition module is used to acquire the target speech of the target speaker, extract the Mel spectrum information in the target speech to obtain the target Mel spectrum, perform timbre feature extraction on the target Mel spectrum to obtain the timbre feature, and perform emotion feature extraction on the target Mel spectrum to obtain the emotion feature; An expansion module, used for expanding the number of frames of the timbre feature and the emotional feature to be equal to the number of frames of the pitch feature, so as to obtain an expanded timbre feature and an expanded emotional feature; A fusion module, used for aligning and fusing the pitch feature, the expanded timbre feature and the expanded emotion feature to obtain a first fusion feature; The reconstruction module is used to extract the speech content of the source speech to obtain the source speech content, fuse the source speech content with the first fusion feature to obtain the second fusion feature, perform spectrum reconstruction on the second fusion feature to obtain a reconstructed Mel spectrum, and obtain the converted speech according to the reconstructed Mel spectrum.

[0006] In a third aspect, an embodiment of the present application provides a terminal device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the artificial intelligence-based sound conversion method as described in the first aspect is implemented.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the artificial intelligence-based sound conversion method as described in the first aspect is implemented.

[0008] Compared with the prior art, the present invention has the following beneficial effects: The source speech of the source speaker is obtained, the fundamental frequency in the source speech is extracted, the pitch feature of the fundamental frequency is extracted to obtain the pitch feature, the frame number of the pitch feature is determined, the target speech of the target speaker is obtained, the Mel spectrum information in the target speech is extracted to obtain the target Mel spectrum, the timbre feature of the target Mel spectrum is extracted to obtain the timbre feature, the emotion feature of the target Mel spectrum is extracted to obtain the emotion feature, the frame number of the timbre feature and the emotion feature are expanded to be equal to the frame number of the pitch feature, the expanded timbre feature and the expanded emotion feature are obtained, the pitch feature, the expanded timbre feature and the expanded emotion feature are aligned and then fused to obtain a first fused feature, the speech content of the source speech is extracted to obtain the source speech content, the source speech content is fused with the first fused feature to obtain a second fused feature, the spectrum of the second fused feature is reconstructed to obtain a reconstructed Mel spectrum, and the converted speech is obtained according to the reconstructed Mel spectrum. In the present application, during the speech conversion process, the emotion information and pitch information of the corresponding target speaker are integrated into the content information of the source speaker, so that the converted target speech can better reflect the voice of the target speaker, thereby improving the speech conversion effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0010] Figure 1 This is a schematic diagram of an application environment of a sound conversion method based on artificial intelligence provided by an embodiment of the present application; Figure 2 It is a flowchart of a sound conversion method based on artificial intelligence provided by an embodiment of the present application; Figure 3 It is a structural schematic diagram of a sound conversion device based on artificial intelligence provided in an embodiment of the present application; Figure 4 It is a structural diagram of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0011] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0012] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0013] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0014] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0015] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0016] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0017] The embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0018] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0019] It should be understood that the size of the serial numbers of the steps in the following embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0020] The present invention provides an artificial intelligence-based sound conversion method, which can be applied in the following aspects: Figure 1 In the application environment, the client communicates with the server. The client includes but is not limited to smart TVs, PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud terminal devices, personal digital assistants (PDAs), and other devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers.

[0021] See also Figure 2 , is a flow chart of a method for converting voice based on artificial intelligence provided by an embodiment of the present application, and the method for converting voice based on artificial intelligence is applied to the server. Figure 2 As shown, the artificial intelligence-based sound conversion method may include the following steps: S201: Acquire source speech of a source speaker, extract fundamental frequency in the source speech, perform pitch feature extraction on the fundamental frequency to obtain pitch features, and determine the number of frames of the pitch features.

[0022] In step S201, the source speech to be converted in speech conversion is obtained, the pitch features in the source speech are extracted, and the number of frames of the pitch features is determined, so as to expand the timbre features and emotional features of the target speaker according to the number of frames of the pitch features.

[0023] In this embodiment, the source speech of the source speaker can be obtained in the voice interaction system, the fundamental frequency in the source speech can be extracted, and a continuous fundamental frequency curve can be obtained. Specifically, the source speech is preliminarily extracted, and the spectrum of the source speech is extracted. The harmonic bright lines on the spectrum are used to correct the double frequencies and half frequencies generated in the extraction process. For the part where the fundamental frequency cannot be extracted, the fundamental frequency is extracted using the fundamental frequency extraction method, and then the position without the fundamental frequency is interpolated using the spline function, so as to obtain a continuous fundamental frequency curve. Here, the source speech is preliminarily extracted, and any non-frequency domain algorithm can be used, such as Praat's autocorrelation method, AMDF algorithm, YIN algorithm, and fundamental frequency recognition algorithm based on statistical models. Among them, Praat is the name of a software, and AMDF is the average magnitude difference function. According to the corresponding fundamental frequency curve, the fundamental frequency is feature extracted to obtain the corresponding pitch feature.

[0024] When determining the number of frames of the pitch feature, the continuous fundamental frequency curve may be divided into equal lengths to obtain the corresponding number of frames. For example, a division step length may be set, and the number of frames of the pitch feature may be determined based on the duration of the continuous fundamental frequency curve and the set step length.

[0025] Optionally, pitch features are extracted from the fundamental frequency to obtain pitch features, including: Extract the fundamental frequency in the source speech using a preset vocoder; The fundamental frequency is logarithmically normalized to obtain a normalized feature, which is then determined as a pitch feature.

[0026] In this embodiment, the vocoder may be a world vocoder. After the source speech is input into the world vocoder, the world vocoder may use the DIO algorithm to analyze the source speech, thereby obtaining the fundamental frequency information corresponding to the source speech, processing the fundamental frequency to obtain the corresponding pitch feature, and during the processing, taking the logarithm of the fundamental frequency and performing normalization processing to obtain the normalized feature, and determining the normalized feature as the pitch feature.

[0027] It should be noted that the vocoder in the embodiment of the present application can also be a neural network vocoder, which may include but is not limited to an autoregressive-based neural network vocoder such as a waveRNN vocoder, a Gan network-based neural network vocoder such as a melGan vocoder, and a hifiGan vocoder.

[0028] S202: Acquire a target speech of a target speaker, extract Mel spectrum information in the target speech to obtain a target Mel spectrum, perform timbre feature extraction on the target Mel spectrum to obtain a timbre feature, perform emotion feature extraction on the target Mel spectrum to obtain an emotion feature.

[0029] In step S202, the target speech of the target speaker is the speech information that the source speaker needs to convert, the Mel spectrum information in the target speech is extracted, the spectrum information is scaled to a coordinate system similar to human ear perception characteristics to facilitate better analysis and processing, and the timbre feature extraction and emotion feature extraction are performed on the target Mel spectrum to obtain corresponding timbre features and emotion features.

[0030] In this embodiment, a web crawler can be written, and after setting a data source, data can be crawled in a targeted manner to obtain target voice data, wherein the data source can be various types of network platforms, social media, or certain specific audio databases, etc., and the target voice data can be the target speaker's music material, speech report, chat dialogue, etc. The target voice data can also be obtained by other means, not limited to this.

[0031] When extracting the Mel spectrum information in the target speech, the target speech is first converted into the corresponding Fourier transform spectrum, and then converted into a Mel spectrum that is more in line with human hearing using a preset function. In this way, the one-dimensional time series signal that is difficult to process can be converted into two-dimensional frequency domain data that is easy to process and richer in information. The Mel spectrum coefficients in the Mel cepstrum are a set of key coefficients used to establish the Mel cepstrum. From the fragments of the target speech, a set of spectra that can represent the target speech can be obtained, and the Mel cepstrum coefficients are the spectrum derived from this spectrum (that is, the spectrum of the spectrum). Unlike general spectra, the biggest feature of the Mel cepstrum is that the frequency bands on the Mel spectrum are evenly distributed on the Mel scale, that is, compared with the generally seen, linear spectrum representation method, such frequency bands are closer to the nonlinear human auditory system (Audio System). For example: In audio compression technology, Mel spectrum is often used for processing.

[0032] The timbre features of the target Mel spectrum are extracted to obtain the timbre features. When extracting the timbre features, a timbre encoder is used to extract the corresponding timbre features. The timbre encoder is a source domain private encoder used to extract the private features of the Mel spectrum. The timbre features are used to characterize the identity of the speaker. By inputting the Mel spectrum into the timbre encoder, the corresponding timbre features can be extracted from the Mel spectrum.

[0033] The target Mel spectrum is subjected to emotional feature extraction to obtain emotional features. When extracting emotional features, an emotional encoder is used to extract the corresponding emotional features. For example, the emotional encoder can be composed of two layers of convolutional neural network layers and two layers of bidirectional long short-term memory networks. The convolution kernel sizes of the two layers of convolutional neural network layers are 7×7 and 20×7 respectively. The batch normalization layer, the ReLU nonlinear activation layer and the maximum pooling layer are connected in sequence after the convolution layer. The kernel sizes of the maximum pooling layer are 2×2 and 1×5 respectively. Through the convolution operation, a 74×128-dimensional intermediate emotional representation sequence M=[m1,m2,...,mn,...,mN] is obtained, where mn is the feature vector of the nth position of the feature. The emotional-related features in the FBank acoustic features are extracted through two layers of convolutional neural network layers, and the features are used as the input features of the long short-term memory network layer to output the corresponding emotional features. In the emotional encoder, a two-layer bidirectional long short-term memory network is used to model the temporal relationship of the input intermediate sequence feature M. The latent vector representations of the two-layer bidirectional long short-term memory network come from the forward and reverse long short-term memory networks respectively. The number of hidden layer nodes in each layer of long short-term memory network is 128. At the same time, nonlinear activation can be used to obtain the final latent vector. The output sequence of N time steps together constitutes the emotional features.

[0034] It should be noted that before using the timbre encoder and the emotion encoder, the timbre encoder and the emotion encoder need to be trained. The training process may include: obtaining an initial timbre encoder and an initial emotion encoder, obtaining sample training data, wherein the sample training data includes the corresponding speaker's voice and the speaker's pitch features and speech content, using the initial timbre encoder to extract the timbre features corresponding to the speaker's voice, using the initial emotion encoder to extract the emotion features corresponding to the speaker's voice, and calculating the mutual information loss according to the speaker's corresponding pitch features, speech content, timbre features and emotion features. The calculation formula is as follows: in, is the mutual information loss corresponding to the timbre encoder, is the mutual information loss corresponding to the emotion encoder, For the timbre characteristics, Voice content, is the pitch feature, For emotional characteristics, is the mutual information between timbre features and speech content, is the mutual information between timbre features and pitch features, is the mutual information between timbre features and emotional features, The mutual information between emotional features and speech content, is the mutual information between emotional features and pitch features, is the mutual information between emotional features and timbre features, is the total loss. Through training, when the total loss is the maximum value, the training is stopped to obtain the trained timbre encoder and emotion encoder. The trained timbre encoder is used to extract the timbre features of the Mel spectrum information in the target speech to obtain the corresponding timbre features. The trained emotion encoder is used to extract the emotion features of the Mel spectrum information in the target speech to obtain the corresponding emotion features.

[0035] S203: Expand the number of frames of the timbre feature and the emotional feature to be equal to the number of frames of the pitch feature, to obtain an expanded timbre feature and an expanded emotional feature.

[0036] In step S203, in order to better integrate the timbre features and the emotional features into the speech content of the source speech, the timbre features and the emotional features are expanded to obtain corresponding features with the same number of frames as the pitch features in the source speech.

[0037] In this embodiment, in order to increase the number of frames of timbre features and emotional features, the number of frames of timbre features and emotional features are expanded, and the number of frames of timbre features and emotional features are expanded to be equal to the number of frames of pitch features to obtain expanded timbre features and expanded emotional features. During expansion, a preset adapter can be used in this embodiment to expand the timbre features and emotional features.

[0038] Optionally, the number of frames of the timbre feature and the emotional feature are expanded to be equal to the number of frames of the pitch feature, to obtain the expanded timbre feature and the expanded emotional feature, including: The timbre feature is copied and expanded to obtain an expanded timbre feature having a number of frames equal to that of the pitch feature; The emotional features are copied and expanded to obtain expanded emotional features with the same number of frames as the pitch features.

[0039] In this embodiment, the timbre feature is expanded by copying to obtain an expanded timbre feature with the same number of frames as the pitch feature, and the emotion feature is expanded by copying to obtain an expanded emotion feature with the same number of frames as the pitch feature. By copying to expand the features, the original information of the corresponding timbre feature and emotion feature is retained.

[0040] S204: Align the pitch feature, the expanded timbre feature, and the expanded emotion feature, and then fuse them to obtain a first fused feature.

[0041] In step S204, the pitch feature, the expanded timbre feature and the expanded emotion feature are integrated so as to integrate the emotion information and timbre information of the target speaker into the content of the source speech, thereby improving the effect of speech conversion.

[0042] In this embodiment, the pitch feature, the expanded timbre feature and the expanded emotional feature are first aligned, the number of frames of the expanded timbre feature and the expanded emotional feature is equal to the number of frames of the pitch feature, and each frame of the pitch feature, the expanded timbre feature and the expanded emotional feature is aligned and then fused. During the fusion, the pitch feature, the expanded timbre feature and the expanded emotional feature of the corresponding frames can be added and fused, or the high feature, the expanded timbre feature and the expanded emotional feature can be aligned and then spliced ​​and fused to obtain the fused feature.

[0043] S205: extracting speech content from the source speech to obtain source speech content, fusing the source speech content with the first fusion feature to obtain a second fusion feature, reconstructing the spectrum of the second fusion feature to obtain a reconstructed Mel spectrum, and obtaining a converted speech according to the reconstructed Mel spectrum.

[0044] In step S205, the pitch feature, the expanded timbre feature and the expanded emotion feature are fused with the speech content in the source speech to obtain a complete speech information feature, and the complete speech information feature is speech reconstructed to obtain a reconstructed Mel spectrum, so as to synthesize the corresponding converted speech according to the reconstructed Mel spectrum.

[0045] In this embodiment, the source speech is extracted to obtain the source speech content, which represents the text information of the source speech. The source speech content is superimposed and fused with the first fusion feature to obtain the second fusion feature. The second fusion feature is spectrally reconstructed to obtain a reconstructed Mel spectrum. When the spectrum is reconstructed, a decoder is used to perform spectrum reconstruction. The decoder is trained to obtain a trained decoder. The trained decoder is used to decode and reconstruct the second fusion feature. The training process of the decoder includes: obtaining sample training data, the sample training data includes the real Mel spectrum in the corresponding sample training data and the original second fusion feature of the corresponding sample training data, the original second fusion feature represents the timbre feature, the expanded emotional feature, the expanded pitch feature, and the feature after superposition and fusion with the speech content in the corresponding sample training data; obtaining the initial decoder and the discriminator in the preset adversarial network, training the initial decoder as the generator in the preset adversarial network, and discriminating the decoding result obtained by the initial decoder decoding the original fusion feature through the discriminator to obtain the discrimination result, and when the discrimination result meets the preset condition, the training of the initial decoder is terminated to obtain a trained decoder. Among them, the decoding result obtained by the initial decoder decoding the original fusion feature through the discriminator is discriminated, including: using the initial decoder to decode the original fusion feature to obtain the decoding result of the corresponding sample training data; based on the real Mel spectrum, the decoding result of the corresponding sample training data is classified by the discriminator to obtain the classification probability value, and the classification probability value is determined as the discrimination result of the decoding result. When the classification probability value is greater than a preset threshold, it is considered that the initial decoder as a generator can reconstruct and decode into a Mel spectrum close to the true Mel spectrum.

[0046] It should be noted that when training the initial decoder, the same speech can be used as sample training data, and the emotional features, timbre features, speech content, and pitch features can be extracted from the same speech. When performing inference, the emotional features and timbre features use the extracted features from the target speech of the target speaker, and the pitch features and speech content use the features from the source speech of the source speaker.

[0047] The reconstructed Mel spectrum is synthesized into the corresponding converted speech using a vocoder. The vocoder in this embodiment may also be a neural network vocoder, which may include but is not limited to an autoregressive neural network vocoder (such as a waveRNN vocoder) and a Gan network-based neural network vocoder (such as a melGan vocoder and a hifiGan vocoder). Optionally, extracting the speech content of the source voice to obtain the source voice content includes: Extracting Mel spectrum information from the source speech information to obtain the source Mel spectrum; Perform dimension-changing processing on the source Mel spectrum to obtain the hidden state features corresponding to the source Mel spectrum; The hidden state features are extracted through a preset residual network to obtain the corresponding source speech content.

[0048] In this embodiment, the Mel spectrum information in the source voice information is extracted to obtain the source Mel spectrum. During the extraction, the source voice is first converted into the corresponding Fourier transform spectrum, and then a preset function is used to convert it into a Mel spectrum that is more in line with human hearing, so that the one-dimensional difficult-to-process time series signal can be converted into two-dimensional frequency domain data that is easy to process and richer in information. The Mel spectrum coefficients in the Mel spectrum are a group of key coefficients used to establish the Mel spectrum. From the fragments in the source voice, a group of inverse spectrums sufficient to represent the source voice can be obtained, and the Mel spectrum coefficients are the spectrum (that is, the spectrum of the spectrum) derived from this inverse spectrum. Different from the general spectrum, the biggest feature of the Mel spectrum is that the frequency bands on the Mel spectrum are evenly distributed on the Mel scale, so the Mel spectrum corresponding to the source voice is extracted and processed based on the Mel spectrum.

[0049] It should be noted that the spectrum of the source voice information is calculated by short-time Fourier transform to obtain a spectrogram. Specifically, the source voice information is subjected to signal frame windowing to obtain multiple frames of voice segments, and each frame of the voice segment is subjected to short-time Fourier transform, and the time domain features of the voice segment are converted into frequency domain features. Finally, the frequency domain features of each frame are stacked in the time dimension to obtain a spectrogram, wherein the frame length of each frame of the voice segment is 20ms, the frame shift is 10ms, and the number of points of the short-time Fourier transform is 512. The spectrogram is filtered by a 64-dimensional Mel spectrum filter group, and the spectrogram is first logarithmically operated to obtain a logarithmic spectrum, and then the logarithmic spectrum is subjected to inverse Fourier transform processing to obtain a source Mel spectrum. Feature extraction is performed on the source Mel spectrum to obtain Mel spectrum coefficients, wherein the feature dimension of the source Mel spectrum coefficients is T*64, and T is the number of frames of the source voice information.

[0050] The source Mel spectrum is processed by dimension change to obtain the hidden state features corresponding to the source Mel spectrum. The source Mel spectrum is processed by dimension change through a convolutional network, wherein the convolutional network includes at least one convolutional layer, the convolution kernel size of the convolutional layer is 3, and the number of channels is 512. The source Mel spectrum can be processed by the convolutional network, and the feature dimension of the source Mel spectrum is adjusted to 512, so as to filter out irrelevant information and obtain the hidden state of the Mel spectrum, that is, the hidden state features. The hidden state features are extracted by a preset residual network to obtain the corresponding source speech content. The residual network has a total of four layers, which are composed of four layers of the same network structure. The hidden state features are the input of the first layer, and the output of the first layer is the input of the second layer, and so on. Finally, the speech features are output through the fourth layer to obtain the corresponding source speech content.

[0051] Optionally, feature extraction is performed on the hidden state features through a preset residual network to obtain corresponding source speech content, including: The hidden state features are convolved through the residual network to obtain the convolution feature vector; The hidden state features are activated through the activation function of the residual network to obtain the activated feature vector; The convolution feature and the activation feature vector are summed according to the preset weight parameters to obtain the target feature, and the source speech content is determined according to the target feature.

[0052] In this embodiment, the residual network is used to perform convolution processing on the hidden state features to obtain a convolution feature vector. The residual network has four layers in total, which are composed of four layers of the same network structure. The hidden state features are the input of the first layer, the output of the first layer is the input of the second layer, and so on. Finally, the speech features are output through the fourth layer. The convolution kernels of each layer of the network structure are 3, 7, 11 and 15 respectively, wherein each layer of the network structure includes two convolution layers (convolution layer A and convolution layer B) and an activation layer. The hidden state features are convoluted by the convolution layer A of the residual network to capture the spatial features of the hidden state features and obtain the convolution feature vector, wherein the size of the convolution layer A is 1*1. The activation layer of the residual network includes two activation functions, and the activation function is used to activate the hidden state features to obtain the activation feature vector, wherein the target activation feature vector can be obtained by performing point multiplication processing on the activation vector output by each activation function and convolution processing on the vector obtained by the point multiplication processing through the convolution layer B, wherein the size of the convolution layer A is 1*1. The preset weight parameters can be set according to actual business needs without restriction. The convolution feature and the activation feature vector are summed according to the preset weight parameters to obtain the speech feature.

[0053] It should be noted that the activation layer of the residual network includes two activation functions. The first activation function is used to activate the hidden state feature to obtain the first activation feature vector. The second activation function is used to activate the hidden state feature to obtain the second activation feature vector. The first activation feature vector and the second activation feature vector are point-multiplied to obtain the target activation feature vector. The first activation function is the tanh function, through which the element value of the input hidden state feature can be transformed to between -1 and 1 to obtain the first activation feature vector. The second activation function is the sigmoid function, through which the element value of the input hidden state feature can be transformed to between 0 and 1 to obtain the second activation feature vector.

[0054] Perform matrix dot multiplication on the first activated eigenvector and the second activated eigenvector to obtain a dot product vector, and then perform one-dimensional convolution on the dot product vector to obtain a target activated eigenvector.

[0055] In this embodiment, feature extraction is performed on the Mel spectrum features corresponding to the source speech information to extract Mel spectrum latent states of variable length, and activation processing is performed on the Mel spectrum latent states to obtain speech features, thereby enriching the semantic information of the speech features and improving the accuracy of speech content recognition.

[0056] The source speech of the source speaker is obtained, the fundamental frequency in the source speech is extracted, the pitch feature of the fundamental frequency is extracted to obtain the pitch feature, the frame number of the pitch feature is determined, the target speech of the target speaker is obtained, the Mel spectrum information in the target speech is extracted to obtain the target Mel spectrum, the timbre feature of the target Mel spectrum is extracted to obtain the timbre feature, the emotion feature of the target Mel spectrum is extracted to obtain the emotion feature, the frame number of the timbre feature and the emotion feature are expanded to be equal to the frame number of the pitch feature, the expanded timbre feature and the expanded emotion feature are obtained, the pitch feature, the expanded timbre feature and the expanded emotion feature are aligned and then fused to obtain a first fused feature, the speech content of the source speech is extracted to obtain the source speech content, the source speech content is fused with the first fused feature to obtain a second fused feature, the spectrum of the second fused feature is reconstructed to obtain a reconstructed Mel spectrum, and the converted speech is obtained according to the reconstructed Mel spectrum. In the present application, during the speech conversion process, the emotion information and pitch information of the corresponding target speaker are integrated into the content information of the source speaker, so that the converted target speech can better reflect the voice of the target speaker, thereby improving the speech conversion effect.

[0057] See also Figure 3 , Figure 3 The structure block diagram of the sound conversion device based on artificial intelligence provided by the embodiment of the present application is shown, and the above-mentioned sound conversion device is applied to the above-mentioned server. For the convenience of explanation, only the part related to the embodiment of the present application is shown.

[0058] See also Figure 3 The sound conversion device 30 includes: a first acquisition module 31, a second acquisition module 32, an expansion module 33, a fusion module 34, and a reconstruction module 35.

[0059] The first acquisition module 31 is used to acquire the source speech of the source speaker, extract the fundamental frequency in the source speech, perform pitch feature extraction on the fundamental frequency to obtain the pitch feature, and determine the frame number of the pitch feature.

[0060] The second acquisition module 32 is used to acquire the target speech of the target speaker, extract the Mel spectrum information in the target speech to obtain the target Mel spectrum, perform timbre feature extraction on the target Mel spectrum to obtain timbre features, and perform emotion feature extraction on the target Mel spectrum to obtain emotion features.

[0061] The expansion module 33 is used to expand the frame numbers of the timbre feature and the emotional feature to be equal to the frame number of the pitch feature, so as to obtain the expanded timbre feature and the expanded emotional feature.

[0062] The fusion module 34 is used to align and fuse the pitch feature, the expanded timbre feature and the expanded emotion feature to obtain a first fusion feature.

[0063] The reconstruction module 35 is used to extract the speech content of the source speech to obtain the source speech content, fuse the source speech content with the first fusion feature to obtain the second fusion feature, perform spectrum reconstruction on the second fusion feature to obtain a reconstructed Mel spectrum, and obtain the converted speech according to the reconstructed Mel spectrum.

[0064] Optionally, the reconstruction module 35 includes: An extraction unit, used for extracting Mel spectrum information from the source speech information to obtain a source Mel spectrum; The dimension changing unit is used to perform dimension changing processing on the source Mel spectrum to obtain the hidden state features corresponding to the source Mel spectrum.

[0065] The obtaining unit is used to extract the hidden state features through a preset residual network to obtain the corresponding source speech content.

[0066] Optionally, the obtaining unit includes: The convolution subunit is used to perform convolution processing on the hidden state features through the residual network to obtain the convolution feature vector.

[0067] The activation subunit is used to activate the hidden state features through the activation function of the residual network to obtain an activation feature vector.

[0068] The summing subunit is used to sum the convolution feature and the activation feature vector according to a preset weight parameter to obtain a target feature, and determine the source speech content according to the target feature.

[0069] Optionally, the first acquisition module 31 includes: The fundamental frequency extraction unit is used to extract the fundamental frequency in the source speech using a preset vocoder.

[0070] The normalization unit is used to perform normalization processing after taking the logarithm of the fundamental frequency to obtain a normalized feature, and determine the normalized feature as a pitch feature.

[0071] Optionally, the expansion module 33 includes: The timbre feature is copied and expanded to obtain an expanded timbre feature with a frame number equal to that of the pitch feature.

[0072] The emotional features are copied and expanded to obtain expanded emotional features with the same number of frames as the pitch features.

[0073] Optionally, the reconstruction module 35 includes: The division unit is used to divide the source speech content into frames to obtain the divided source speech content with the same number of frames as the pitch feature.

[0074] The fusion unit is used to align the divided source speech content with the first fusion feature and then fuse them to obtain a second fusion feature.

[0075] Figure 4 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application. Figure 4 As shown, the terminal device of this embodiment includes: at least one processor ( Figure 4 Only one is shown), a memory, and a computer program stored in the memory and executable on at least one processor, wherein when the processor executes the computer program, the steps in any of the above-mentioned embodiments of the sound conversion method based on artificial intelligence are implemented.

[0076] The terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that Figure 4 It is only an example of a terminal device and does not constitute a limitation on the terminal device. The terminal device may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include a network interface, a display screen, and an input device.

[0077] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0078] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory may be the memory of the terminal device, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The readable storage medium may be a hard disk of the terminal device, and in other embodiments, it may also be an external storage device of the terminal device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal device. Further, the memory may also include both an internal storage unit of the terminal device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of a computer program, etc. The memory may also be used to temporarily store data that has been output or is to be output.

[0079] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiment can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0080] The present application implements all or part of the processes in the above-mentioned embodiment method, and may also be completed through a computer program product. When the computer program product runs on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiment when executing the computer program product.

[0081] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0082] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0083] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0084] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0085] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A sound conversion method based on artificial intelligence, characterized in that: The sound conversion method comprises: Acquire a source speech of a source speaker, extract a fundamental frequency in the source speech, perform pitch feature extraction on the fundamental frequency to obtain a pitch feature, and determine the number of frames of the pitch feature; Acquire a target speech of a target speaker, extract Mel spectrum information in the target speech to obtain a target Mel spectrum, perform timbre feature extraction on the target Mel spectrum to obtain a timbre feature, perform emotion feature extraction on the target Mel spectrum to obtain an emotion feature; Expanding the number of frames of the timbre feature and the emotional feature to be equal to the number of frames of the pitch feature, to obtain an expanded timbre feature and an expanded emotional feature; Aligning the pitch feature, the expanded timbre feature, and the expanded emotion feature, and fusing them to obtain a first fused feature; Speech content is extracted from the source speech to obtain source speech content, the source speech content is fused with the first fusion feature to obtain a second fusion feature, spectrum is reconstructed on the second fusion feature to obtain a reconstructed Mel spectrum, and converted speech is obtained according to the reconstructed Mel spectrum.

2. The sound conversion method according to claim 1, characterized in that: The extracting the speech content of the source voice to obtain the source voice content includes: Extracting Mel spectrum information from the source voice information to obtain a source Mel spectrum; Performing dimension conversion processing on the source Mel spectrum to obtain hidden state features corresponding to the source Mel spectrum; The hidden state features are extracted through a preset residual network to obtain corresponding source speech content.

3. The sound conversion method according to claim 2, characterized in that: The step of extracting the hidden state features through a preset residual network to obtain corresponding source speech content includes: Performing convolution processing on the hidden state feature through the residual network to obtain a convolution feature vector; Activate the hidden state feature through the activation function of the residual network to obtain an activated feature vector; The convolution feature and the activation feature vector are summed according to a preset weight parameter to obtain a target feature, and the source speech content is determined according to the target feature.

4. The sound conversion method according to claim 1, characterized in that: The step of extracting the pitch feature of the fundamental frequency to obtain the pitch feature comprises: Extracting the fundamental frequency in the source speech using a preset vocoder; The fundamental frequency is logarithmically processed and then normalized to obtain a normalized feature, which is then determined as the pitch feature.

5. The sound conversion method according to claim 1, characterized in that: The step of expanding the number of frames of the timbre feature and the emotional feature to be equal to the number of frames of the pitch feature to obtain the expanded timbre feature and the expanded emotional feature comprises: Copying and expanding the timbre feature to obtain an expanded timbre feature having a frame number equal to that of the pitch feature; The emotional feature is copied and expanded to obtain an expanded emotional feature having the same number of frames as the pitch feature.

6. The sound conversion method according to claim 1, characterized in that: The fusing the source speech content with the first fusion feature to obtain a second fusion feature includes: Dividing the source speech content by the number of frames to obtain the divided source speech content having the same number of frames as the pitch feature; The divided source speech content is aligned with the first fusion feature and then fused to obtain a second fusion feature.

7. A sound conversion device based on artificial intelligence, characterized in that: The sound conversion device comprises: A first acquisition module is used to acquire a source speech of a source speaker, extract a fundamental frequency in the source speech, perform pitch feature extraction on the fundamental frequency to obtain a pitch feature, and determine a frame number of the pitch feature; The second acquisition module is used to acquire the target speech of the target speaker, extract the Mel spectrum information in the target speech to obtain the target Mel spectrum, perform timbre feature extraction on the target Mel spectrum to obtain the timbre feature, and perform emotion feature extraction on the target Mel spectrum to obtain the emotion feature; An expansion module, used for expanding the number of frames of the timbre feature and the emotional feature to be equal to the number of frames of the pitch feature, so as to obtain an expanded timbre feature and an expanded emotional feature; A fusion module, used for aligning and fusing the pitch feature, the expanded timbre feature and the expanded emotion feature to obtain a first fusion feature; The reconstruction module is used to extract the speech content of the source speech to obtain the source speech content, fuse the source speech content with the first fusion feature to obtain the second fusion feature, perform spectrum reconstruction on the second fusion feature to obtain a reconstructed Mel spectrum, and obtain the converted speech according to the reconstructed Mel spectrum.

8. The sound conversion device according to claim 7, characterized in that: The reconstruction module includes: An extraction unit, used to extract Mel spectrum information from the source voice information to obtain a source Mel spectrum; A dimension conversion unit, used for performing dimension conversion processing on the source Mel spectrum to obtain a hidden state feature corresponding to the source Mel spectrum; The obtaining unit is used to extract the hidden state features through a preset residual network to obtain the corresponding source speech content.

9. A terminal device, characterized in that: The terminal device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the sound conversion method as claimed in any one of claims 1 to 6 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the sound conversion method according to any one of claims 1 to 6 is implemented.

Images