Method and device for migrating voice styles

Abstract

The invention discloses a method and a device for migrating voice styles. The method comprises the following steps: performing feature extraction on a source voice sequence to obtain source voice style features; the source speech style features are input into an implicit neural representation module to obtain target speech style features, and the implicit neural representation module is configured to generate the target speech style features based on the target implicit vector and the source speech style features, the target implicit vector is configured to be an implicit vector used for representing a migration relation between the source voice style feature and the target voice style feature; and generating a target voice sequence based on the target voice style feature.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computers, and in particular to a method and device for migrating speech style. BACKGROUND

[0002] With the help of large model technology, it has been possible to synthesize speech with similar timbre according to a given reference speech. In order to make the synthesized speech more natural, more speech style details such as pronunciation habits and speaking rhythm should also be considered in the performance of the speech, that is, the speech style is migrated to the synthesized speech. At present, when migrating the speech style, a large amount of cost analysis is required to analyze the details of the speech style, but such details are easily lost during the migration process, resulting in difficulty in guaranteeing the migration effect of the speech style in the synthesized speech. SUMMARY

[0003] The method for migrating speech style according to the embodiment of the present application comprises: performing feature extraction on a source speech sequence to obtain source speech style features; inputting the source speech style features into an implicit neural representation module to obtain target speech style features, wherein the implicit neural representation module is configured to generate the target speech style features based on a target implicit vector and the source language style features, and the target implicit vector is configured to be an implicit vector for representing the migration relationship between the source speech style features and the target speech style features; and generating a target speech sequence based on the target speech style features.

[0004] The device for migrating speech style according to the embodiment of the present application comprises: a processor; and a memory having computer executable instructions stored thereon, wherein the computer executable instructions, when executed by the processor, cause the processor to perform the above-mentioned method for migrating speech style.

[0005] The computer readable storage medium according to the embodiment of the present application has computer executable instructions stored thereon, wherein the computer executable instructions, when executed by the processor, cause the processor to perform the above-mentioned method for migrating speech style.

[0006] The computer program product according to the embodiment of the present application comprises computer executable instructions, wherein the computer executable instructions, when executed by the processor, cause the processor to perform the above-mentioned method for migrating speech style. BRIEF DESCRIPTION OF DRAWINGS

[0007] The present application can be better understood from the following description of specific embodiments thereof, given by way of example only and with reference to the accompanying drawings in which:

[0008] Figure 1 A schematic flowchart of the method for migrating speech style according to the embodiment of the present application is shown.

[0009] Figure 2A schematic diagram showing information flow of a method for migrating speech style according to an embodiment of the present application is shown.

[0010] Figure 3 A schematic diagram showing an example structure of an implicit neural representation module according to an embodiment of the present application is shown.

[0011] Figure 4 A schematic diagram showing a computer system that can implement a method and apparatus for migrating speech style according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0012] Features and exemplary embodiments of various aspects of the present application will be described below in detail. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without some or all of these specific details. The description of the embodiments is merely intended to provide a better understanding of the present application by showing examples of the present application. The present application is not in any way limited to the specific configurations and algorithms set forth below, but covers any modifications, alternatives, and improvements of elements, components, and algorithms without departing from the spirit of the present application. In the drawings and the following description, well-known structures and techniques are not shown in order to avoid unnecessary obscuring of the present application.

[0013] Currently, when migrating the speech style of a source speech into a target speech, the representation of a specific style can be obtained through an attention mechanism for style migration. The speech style details include a large amount of high-dimensional information or high-frequency information. The weight calculation of the attention mechanism uses the fitting principle of the probability description distribution and will have random errors, and the loss of speech style details is prone to occur, so that the style represented by each attention head will be randomly weakened or strengthened, such as missing words or repeated expressions, and it is difficult to guarantee the control of the details of the complex style. At the same time, the combination of attention heads is limited, and before combination, the style of each attention head needs to be understood, which requires a large amount of cost to analyze the single attention head. With the improvement of speech resolution, the required memory resources for style migration will be greatly increased.

[0014] In view of the above problems, the method and device for migrating speech style according to the embodiments of the present application are proposed, which inputs the source speech style feature into an implicit neural representation (INR) module based on a target implicit vector to obtain a target speech style feature. The implicit vector is a kind of high-resolution, continuous and differentiable data representation realized by a neural network parameterized continuous function, and the required parameters occupy a small amount of space regardless of the resolution. Therefore, the target implicit vector is set based on the characteristics of the implicit vector to represent the migration relationship between the source speech style feature and the target speech style feature, which can capture the speech style details of high latitude information and high frequency information through a continuous function, ensure the complexity and diversity of the speech style details, and accurately realize the style migration between the source speech style feature and the target speech style feature at a low cost.

[0015] Figure 1 A schematic flowchart of the method for migrating speech style according to the embodiments of the present application is shown. As shown in the figure, the method 100 for migrating speech style according to the embodiments of the present application includes: S101: performing feature extraction on a source speech sequence to obtain a source speech style feature; S102: inputting the source speech style feature into an implicit neural representation module to obtain a target speech style feature, wherein the implicit neural representation module is configured to generate the target speech style feature based on a target implicit vector and the source language style feature, and the target implicit vector is configured as an implicit vector for representing the migration relationship between the source speech style feature and the target speech style feature; and S103: generating a target speech sequence based on the target speech style feature. Figure 1

[0016] In some embodiments, obtaining the source speech style feature can include: performing feature extraction on the source speech sequence in multiple style dimensions to obtain multiple source speech style feature vectors; and performing preprocessing and arrangement on the multiple source speech style feature vectors to obtain a feature matrix as the source speech style feature. Generally, the style dimensions of the speech sequence include multiple style dimensions such as emotion, prosody, pronunciation habit, etc. The feature extraction of different style dimensions can be realized through corresponding speech analysis tools. Here, the feature matrix is filled in the order of time frame x style dimension to form a two-dimensional matrix as the source speech style feature.

[0017] ​In some embodiments, the preprocessing of the plurality of source speech style feature vectors can include: unifying the plurality of source speech style feature vectors to a preset length; and normalizing each of the plurality of source speech style feature vectors. In this way, all source language style feature vectors can be processed into equal length vectors and the feature vector energy can be ensured to be constant. In some embodiments, unifying the plurality of source speech style feature vectors to a preset length can include: performing a discrete cosine transform on a source speech style feature vector in the plurality of source speech style feature vectors whose vector length is less than the preset length to obtain a transformed frequency domain sequence, zero padding the transformed frequency domain sequence to obtain a zero padded frequency domain sequence whose sequence length is equal to the preset length, and performing a discrete cosine inverse transform on the zero padded frequency domain sequence to obtain a source speech style feature vector whose vector length is equal to the preset length. The preset length is usually the maximum value among the vector lengths of the source speech style feature vectors. This operation can adjust the dimension of the source speech feature vector while ensuring that the vector length is constant.

[0018] In some embodiments, the preprocessing can further include data operations such as denoising, dimensionality reduction, etc. In the method according to the present embodiment, the object of preprocessing is the source speech style feature vector, which is different from the traditional method of preprocessing the spectral feature. Figure 2 A schematic flowchart of migrating information of a speech style is shown according to an embodiment of the present application. As shown in Figure 2 The feature extraction is performed on the source speech sequence x2 to obtain a plurality of source speech style feature vectors v x2i , i = 1, …, p, p is a positive integer; the plurality of source speech style feature vectors are preprocessed and arranged to obtain a feature matrix X2 as a source speech style feature; and the feature matrix X2 is input into the implicit neural representation module to obtain a feature matrix Y2 as a target speech style feature.

[0019] In some embodiments, the implicit neural representation module determines the parameters, which can output a target speech style feature corresponding to a target speech style according to an input source speech style feature corresponding to an initial speech style, that is, the style migration from the initial speech style to the target speech style is realized. As shown in Figure 2 In some embodiments, the training phase of the implicit neural representation module can include: obtaining an initial speech sequence x1 corresponding to an initial speech style and an expected speech sequence y1 corresponding to a target speech style; performing feature extraction on the initial speech sequence x1 to obtain a plurality of initial speech style feature vectors v x1i , i = 1, …, p, and arranging the plurality of initial speech style feature vectors to obtain a feature matrix X1; performing feature extraction on the expected speech sequence y1 to obtain a plurality of expected speech style feature vectors v y1i, i = 1, …, p, the feature matrix Y1 is obtained by preprocessing and arranging a plurality of expected speech style feature vectors; and the implicit neural representation module is trained using the feature matrices X1 and Y1. The speech style feature vector v x1i and v y1i The feature matrices X1 and Y1 both represent the style features of the respective speech styles. The implicit neural representation module is trained based on the feature matrix X1 of the initial speech style and the feature matrix Y1 of the target speech style to obtain a target hidden vector representing the migration relationship between the feature matrices X1 and Y1, so that the style relationship between the style features of the two speech styles is established through the target hidden vector. In addition, training and learning using style features can reduce the training dimension explosion, noise influence, and time sequence inconsistency caused by the sampling inconsistency of the initial speech sequence and the expected speech sequence when training and learning using speech sequences.

[0020] In some embodiments, the speech style of the speech sequence is determined by the speaking subject. For example, the speaking subject of the initial speech style can be a tool for generating speech based on text, which can directly generate an initial speech sequence x1 of the corresponding initial speech style based on a preset standard text through the tool; the speaking subject of the target speech style can be a natural person, who can record the speech of the natural person about the preset standard text as the expected speech sequence y1 of the target speech style. In order to fully learn the style details of a certain speech style, the corresponding speech sequence should include as much rich style details as possible.

[0021] In some embodiments, the implicit neural representation module can be trained by a contrastive learning loss function to constrain the alignment of the feature matrices X1 and Y1 in each feature dimension, so as to find the mapping relationship (i.e., the migration relationship between the source speech style features and the target speech style features) of the initial speech style and the target speech style in the style details, which is represented in the target hidden vector. Figure 3 An example structure diagram of an implicit neural representation module (i.e., INR module) according to an embodiment of the present application is shown. As Figure 3 shown, in some embodiments, the implicit neural representation module can be configured as a style encoder E φ combined with a generative adversarial network structure. The style encoder E φ is configured to generate a target hidden vector, and the generative adversarial network structure can include a generator G θ that generates target speech style features based on source speech style features and the target hidden vector, and a discriminator D ψ that determines the authenticity probability of the target speech style features.

[0022] When training the implicit neural representation module, the style encoder E φ inputs Y1 and outputs a target hidden vector zY = E φ (Y1) = σ L (W L …σ1(W1·vec(Y1) + b1)… + b L ). Where vec(·) represents the matrix flattening into a vector, W i and b i represent the weight and bias of the i-th layer, respectively, and σ i represents the activation function. Since the target hidden vector needs to be generated according to the target hidden vector and the generator in the application stage, the Coiflet wavelet can be selected to design the activation function. The symmetry of the Coiflet wavelet reduces the phase distortion when reconstructing the speech, and can ensure the fidelity of the speech reconstruction. Since the Coiflet wavelet has no closed form, it is approximated here using a wavelet function, that is, coiflet(x) = ∑ k c k ·β(x-k), where c k is the discrete sampling coefficient of the Coiflet wavelet, which can be obtained by querying, and β is the basis function, which can be set to B-spline. Therefore where a is used to control the frequency resolution of the wavelet, and the larger the value is, the more it tends to capture smooth trends (such as vowel formants in speech), and the smaller the value is, the more it tends to capture transient features (such as speech boundary transients); for the generator G θ , the input X1 and the target hidden vector z Y , the output is the predicted feature matrix where the activation function is designed to be consistent with the style encoder E φ to ensure the speech generation effect in the application stage; for the discriminator D ψ , the input is Y1 or and the conditional information (for example, z Y or X1), and the output is D ψ (Y1) or , which represents the authenticity probability, and the activation function thereof can be selected from traditional activation functions, such as sigmoid, ReLU, and the like.

[0023] In some embodiments, the implicit neural representation module is trained based on a discriminator-based adversarial loss function, a reconstruction loss function of the generator, and a consistency loss function of the style encoder. Regarding the loss function of the implicit neural representation module, the total loss function can be represented as: where λ1 and λ2 are hyperparameters. is the adversarial loss function, which forces the generator G θ to capture high-frequency style features, improves the realism of style generation, and makes it difficult for the discriminator D ψ to distinguish between true and false, specifically To reconstruct the loss function, ensure that the generator G θ Preserve the structure of X1, such as the coupling relationship between prosody and pronunciation habits, specifically The consistency loss function of the style encoder is

[0024] In some embodiments, the training process of the implicit neural representation module can include: first fixing G θ , updating D ψ , Then fix D ψ , update G θ and E φ , Alternating iterations until convergence, η represents the convergence step size. After the module converges, the training is complete. The target latent vector z Y = E φ (Y1) can accurately control the speech style details of the output results of the implicit neural representation module, and can be applied to subsequent style transfer.

[0025] In the method according to the embodiments of the application, the style encoder compresses high-dimensional features into low-dimensional target latent vectors to extract the essential information of the style, and then performs high-fidelity mapping based on the target latent vectors through the generative adversarial network structure, overcoming the tendency of the output results of the style encoder to be excessively linear and smooth. At the same time, in order to control the scale of the detailed feature encoding of the style encoder, the activation functions in the style encoder and the generator are set to wavelet functions.

[0026] In the method according to the embodiments of the application, the target latent vector can efficiently encode the continuous features of the speech signal with compact neural network parameters, realizing fine modeling of acoustic details. At the same time, the generative adversarial network structure is continuously optimized for the naturalness and expressiveness of the synthesized speech through the adversarial training of the generator and the discriminator.

[0027] In the method according to the embodiments of the application, a long enough reference audio can be used as a training sample to ensure the accuracy of style transfer. Regardless of how large the training sample is or how high the resolution requirement is, the amount of storage space occupied by the weights of the neural network required by the implicit neural representation module is always very small, usually a few hundred KB.

[0028] In the method according to the embodiments of the present application, the target latent vector is combined with the generative adversarial network structure, so that a small amount of speech samples (for example, 1-2 minutes of recording) can be used to generate highly realistic and emotionally expressive personalized speech. Compared with the traditional speech transfer technology which relies on a large amount of training data and must be operated in the cloud, the method according to the embodiments of the present application requires less sample data, the storage space required by the target latent vector and the generative adversarial network structure is smaller, and the user data does not need to be uploaded to the cloud for processing, which can be applied to various scenarios, such as toys providing personalized voice interaction, companion robots providing emotional services, etc.

[0029] In some embodiments, the method can further include adjusting the numerical value of the target latent vector based on the adjustment instruction. The numerical value of the target latent vector determines the speech style details of the output result of the implicit neural representation module. By adjusting the numerical value of the target latent vector, the output speech style details can be changed. Since the target latent vector can use the relationship between features for global adjustment, any adjustment of the numerical value in the target latent vector can ensure that the output result of the implicit neural representation module has overall coordination in the speech style. Specifically, the adjustment manner of the numerical value of the target latent vector is not limited, so as to improve the style richness and flexibility of the output result.

[0030] Figure 4 A schematic diagram of a computer system that can implement the speech recognition model training apparatus and method according to the embodiments of the present application is shown. It should be understood that, Figure 4 The computer system 400 shown is only one example of a device that can be used to implement the apparatus and method for transferring speech style according to the embodiments of the present application, and should not be taken as limiting the functionality or use of the apparatus and method for transferring speech style according to the embodiments of the present application.

[0031] As Figure 4 shown, the computer system 400 can include a processing device (for example, a central processor, a graphics processor, etc.) 401, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 402 or loaded from a storage device 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the computer system 400 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other through a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0032] Generally, the following devices can be connected to the I / O interface 405: input devices 406, including, for example, a touch screen, a touch pad, a camera, an accelerometer, a gyroscope, a sensor, etc.; output devices 407, including, for example, a liquid crystal display (LCD), a speaker, a vibrator, a motor, an electronic speed regulator, etc.; storage devices 408, including, for example, a flash card, etc.; and communication devices 409. The communication devices 409 can allow the computer system 400 to exchange data with other devices via wireless communication or wired communication. Although Figure 4 The computer system 400 is shown with various devices, but it is understood that not all of the devices shown need be implemented or present. More or less devices can alternatively be implemented or present. Figure 4 Each block shown in the flowcharts can represent a device or multiple devices as needed.

[0033] In particular, the processes described above with reference to the flowcharts can be implemented as a computer program according to some embodiments of the present application. For example, a computer readable medium is provided, on which a computer program is stored, the computer program containing instructions for performing the above-mentioned functional units defined in the device for migrating speech style according to embodiments of the present application. Figure 1 The program code shown for the method for migrating speech style is executed by the processing device 401. In such embodiments, the computer program can be downloaded and installed from a network via the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functional units defined in the device for migrating speech style according to embodiments of the present application are implemented.

[0034] Note that a computer readable medium storing the computer program code according to the embodiments of the present application can be a computer readable signal medium or a computer readable storage medium, or any combination thereof. The computer readable storage medium may, for example, be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. The computer readable storage medium according to the embodiments of the present application can be any tangible medium that contains or stores a program, which can be used by or in connection with an instruction execution system, apparatus, or device. In addition, the computer readable signal medium according to the embodiments of the present application can include a computer readable program code that is propagated in or on a carrier medium, which can be a computer readable storage medium or a computer readable signal medium. The propagated computer readable program code can take any of a variety of forms, including but not limited to, radio frequency (RF) signals, light signals, or any suitable combination thereof. The computer readable signal medium can be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0035] Computer program code for carrying out operations of the embodiments of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++, or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0036] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions, and operations of the systems, methods, and computer program products according to the various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0037] The present invention may be implemented in other specific forms without departing from its spirit and essential characteristics. For example, the algorithms described in the specific embodiments may be modified without departing from the basic spirit of the present invention. Therefore, the present embodiments are to be considered in all respects as illustrative and not restrictive, the scope of the invention is defined by the appended claims rather than the foregoing description, and all modifications coming within the meaning and scope of equivalents of the claims are intended to be included within the scope of the present invention.

Claims

1. A method for transferring speech style, comprising: Extract features from the source speech sequence to obtain the source speech style features; Inputting the source speech style feature into an implicit neural representation module to obtain a target speech style feature, wherein the implicit neural representation module is configured to generate the target speech style feature based on a target latent vector and the source speech style feature, and the target latent vector is configured as a latent vector for representing a transfer relationship between the source speech style feature and the target speech style feature; and A target speech sequence is generated based on the target speech style feature.

2. The method according to claim 1, wherein The implicit neural representation module is configured as a model architecture that combines a style encoder with a generative adversarial network structure. The style encoder and the generator in the generative adversarial network structure use wavelet functions as activation functions to control the scale of detail feature encoding.

3. The method according to claim 2, wherein: The style encoder is configured to generate the target latent vector, the generative adversarial network structure includes the generator that generates the target speech style feature based on the source speech style feature and the target latent vector, and the discriminator that determines the authenticity probability of the target speech style feature, and training the implicit neural representation module includes training the implicit neural representation module based on the adversarial loss function of the discriminator, the reconstruction loss function of the generator, and the consistency loss function of the style encoder.

4. The method according to claim 1, wherein Acquiring the source speech style feature includes: Extracting features of multiple style dimensions from the source speech sequence to obtain multiple source speech style feature vectors; and The plurality of source speech style feature vectors are preprocessed and arranged to obtain a feature matrix as the source speech style feature.

5. The method according to claim 4, wherein Preprocessing the plurality of source speech style feature vectors includes: unifying the plurality of source speech style feature vectors into a preset length; and Normalization is performed on each of the plurality of source speech style feature vectors.

6. The method according to claim 5, wherein: Unifying the multiple source speech style feature vectors into a preset length includes: performing discrete cosine transform on the source speech style feature vectors whose vector length is less than the preset length among the multiple source speech style feature vectors to obtain a transformed frequency domain sequence, padding the transformed frequency domain sequence with zeros to obtain a zero-padded frequency domain sequence whose frequency domain sequence length is equal to the preset length, and performing inverse discrete cosine transform on the zero-padded frequency domain sequence to obtain a source speech style feature vector whose vector length is the preset length.

7. The method according to claim 1, wherein The method further includes adjusting the value of the target latent vector based on the adjustment instruction.

8. A device for migrating speech style, comprising: processor; as well as A memory having computer-executable instructions stored thereon, wherein when the computer-executable instructions are executed by the processor, the processor is prompted to perform the method for transferring speech style according to any one of claims 1 to 7.

9. A computer-readable storage medium having computer-executable instructions stored thereon, wherein: When the computer-executable instructions are executed by a processor, the processor is prompted to perform the method for transferring speech style according to any one of claims 1 to 7.

10. A computer program product comprising computer-executable instructions, wherein: When the computer-executable instructions are executed by a processor, the processor is prompted to perform the method for transferring speech style according to any one of claims 1 to 7.

Images