Voice style conversion method and device, electronic equipment and storage medium

By extracting the characteristics of speech and speakers and training the speech style conversion model, the problem of speech style conversion deviation in the existing technology is solved, and higher accuracy and nature are achieved.

CN119007733APending Publication Date: 2024-11-22PING AN TECH (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411412866.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-10
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

In the existing pronunciation style conversion technology, it is difficult for text to describe the pronunciation style of a specific speaker in a comprehensive and precise manner, resulting in a deviation between the converted pronunciation style and the real style, affecting accuracy.

Method used

By obtaining speech synthesis samples, including speech data and speaker description text, extracting speech features and speaker characteristics, training speech style conversion models, using the model to style the target speech data, and generating speech data with target speaker characteristics.

Benefits of technology

It improves the accuracy and naturalness of speech style conversion, enhances the conversion efficiency, and the generated speech is closer to the style expression of the target speaker.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007733A_ABST
    Figure CN119007733A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a voice style conversion method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence and finance. The method comprises the steps of performing feature extraction on sample source voice data, sample reference voice data and a sample speaker description text to obtain a sample source voice embedded feature, a sample speaker embedded feature and a sample speaker description embedded feature; training an original voice style conversion model by using the sample source voice embedded features, the sample speaker embedded features and the sample speaker description embedded features to obtain a target voice style conversion model; performing feature extraction on the target reference voice data and the target source voice data to obtain a target speaker embedding feature and a target source voice embedding feature; and performing style conversion on the target speaker embedded feature and the target source voice embedded feature by using the target voice style conversion model to obtain the target synthetic voice data, thereby improving the accuracy of voice style conversion.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence and financial technology, and in particular to a method and device for voice style conversion, an electronic device, and a storage medium. Background Art

[0002] Speech style transfer is a technology that converts the speaker style of a speech into the style of another speaker. It can be applied to a variety of scenarios. For example, in the financial field, it is used for speech style transfer in scenarios such as virtual anchors, intelligent customer service, and tutorial video production.

[0003] At present, the speaker style of speech is mainly adjusted by using style description text of a specific speaker. However, in actual application scenarios, it is often difficult for text to fully and accurately describe the unique speech style of a specific speaker, which leads to a deviation between the converted speech style and the real style of the specific speaker, affecting the accuracy of speech style conversion.

[0004] Therefore, how to improve the accuracy of speech style transfer has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to propose a speech style conversion method and device, an electronic device and a storage medium, aiming to improve the accuracy of speech style conversion.

[0006] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a method for voice style conversion, the method comprising:

[0007] Acquire a speech synthesis sample; wherein the speech synthesis sample includes sample source speech data, sample reference speech data and sample speaker description text; wherein the sample reference speech data has sample reference speaker characteristics, and the sample speaker description text is used to describe the sample reference speaker characteristics;

[0008] Extracting speech features from the sample source speech data to obtain sample source speech embedding features;

[0009] Performing speaker extraction on the sample reference speech data to obtain sample speaker embedding features;

[0010] Performing text feature extraction on the sample speaker description text to obtain sample speaker description embedding features;

[0011] Training a preset original speech style transfer model based on the sample source speech embedding feature, the sample speaker embedding feature and the sample speaker description embedding feature to obtain a target speech style transfer model;

[0012] Performing speaker extraction on the pre-acquired target reference speech data to obtain target speaker embedding features; wherein the target reference speech data has target reference speaker features;

[0013] Extracting speech features from the pre-acquired target source speech data to obtain target source speech embedding features;

[0014] The target speech style conversion model is used to perform style conversion on the target speaker embedding features and the target source speech embedding features to obtain target synthesized speech data; wherein the target synthesized speech data has the target reference speaker features.

[0015] In some embodiments, the training of a preset original speech style conversion model based on the sample source speech embedding feature, the sample speaker embedding feature, and the sample speaker description embedding feature to obtain a target speech style conversion model includes:

[0016] Extracting speech features from the sample reference speech data to obtain sample target speech embedding features;

[0017] Performing feature transformation on the sample source speech embedding feature and the sample speaker embedding feature through the original speech style conversion model to obtain a sample transformed speech embedding feature;

[0018] Performing comparative learning based on the sample speaker embedding feature and the sample speaker description embedding feature to obtain comparative loss data;

[0019] The original speech style conversion model is optimized based on the sample target speech embedding feature, the sample transformed speech embedding feature and the contrast loss data to obtain the target speech style conversion model.

[0020] In some embodiments, the step of performing feature transformation on the sample source speech embedding feature and the sample speaker embedding feature through the original speech style conversion model to obtain the sample transformed speech embedding feature includes:

[0021] Performing feature mapping on the sample source speech embedding features through the original speech style conversion model to obtain sample speech mapping features;

[0022] Performing feature adjustment on the sample speech mapping feature based on the sample speaker embedding feature to obtain the original sample speech embedding feature;

[0023] The original sample speech embedding feature is sampled to obtain the sample transformed speech embedding feature.

[0024] In some embodiments, performing comparative learning based on the sample speaker embedding feature and the sample speaker description embedding feature to obtain comparative loss data includes:

[0025] Performing negative sampling on the sample speaker description embedding feature to obtain a negative sample speaker embedding feature;

[0026] A contrast loss calculation is performed based on the negative sample speaker embedding feature and the sample speaker embedding feature to obtain contrast loss data.

[0027] In some embodiments, the optimizing the original speech style conversion model based on the sample target speech embedding feature, the sample transformed speech embedding feature and the contrast loss data to obtain the target speech style conversion model includes:

[0028] Performing divergence loss calculation on the sample target speech embedding feature and the sample transformed speech embedding feature to obtain sample divergence loss data;

[0029] The parameters of the original speech style transfer model are adjusted based on the sample divergence loss data and the contrast loss data to obtain the target speech style transfer model.

[0030] In some embodiments, performing style conversion on the target speaker embedding feature and the target source speech embedding feature through the target speech style conversion model to obtain target synthesized speech data includes:

[0031] Performing feature transformation on the target speaker embedding feature and the target source speech embedding feature through the target speech style conversion model to obtain a target speech synthesis embedding feature;

[0032] The target speech synthesis embedded features are decoded to obtain target synthesized speech data.

[0033] In some embodiments, the extracting the speaker of the sample reference speech data to obtain the sample speaker embedding feature includes:

[0034] Performing spectrum conversion on the sample reference speech data to obtain sample speech spectrum data;

[0035] Style features are extracted from the sample speech spectrum data to obtain embedded features of the sample speaker.

[0036] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a speech style conversion device, the device comprising:

[0037] A sample data acquisition module, used to acquire a speech synthesis sample; wherein the speech synthesis sample includes sample source speech data, sample reference speech data and sample speaker description text; wherein the sample reference speech data has sample reference speaker characteristics, and the sample speaker description text is used to describe the sample reference speaker characteristics;

[0038] A sample speech feature extraction module is used to extract speech features from the sample source speech data to obtain sample source speech embedding features;

[0039] A sample speaker extraction module, used to extract the speaker of the sample reference speech data to obtain a sample speaker embedding feature;

[0040] A sample text feature extraction module is used to extract text features from the sample speaker description text to obtain sample speaker description embedding features;

[0041] An original model training module, used to train a preset original speech style transfer model based on the sample source speech embedding feature, the sample speaker embedding feature and the sample speaker description embedding feature to obtain a target speech style transfer model;

[0042] A target speaker extraction module is used to perform speaker extraction on the pre-acquired target reference speech data to obtain a target speaker embedding feature; wherein the target reference speech data has a target reference speaker feature;

[0043] A target speech feature extraction module is used to extract speech features from the pre-acquired target source speech data to obtain target source speech embedding features;

[0044] A target speech style conversion module is used to perform style conversion on the target speaker embedding features and the target source speech embedding features through the target speech style conversion model to obtain target synthesized speech data; wherein the target synthesized speech data has the target reference speaker features.

[0045] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0046] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0047] The speech style conversion method and device, electronic device and storage medium proposed in the present application respectively extract features from the acquired sample source speech data, sample reference speech data and sample speaker description text, convert them into numerical representations, and obtain sample source speech embedding features, sample speaker embedding features and sample speaker description embedding features; then, the sample source speech embedding features, sample speaker embedding features and sample speaker description embedding features are used to train the original speech style conversion model to improve the model's learning ability for features, thereby improving the accuracy of speech style conversion and obtaining a target speech style conversion model; finally, feature extraction is performed on the pre-acquired target reference speech data and target source speech data respectively to obtain target speaker embedding features and target source speech embedding features; the target speech style conversion model is used to perform style conversion on the target speaker embedding features and the target source speech embedding features to generate target synthetic speech data that contains both the target source speech data content and the target reference speaker features, which can improve the naturalness of speech and the efficiency and accuracy of speech style conversion. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flow chart of a method for voice style conversion provided by an embodiment of the present application;

[0049] Figure 2 yes Figure 1 Flow chart of step S102 in FIG.

[0050] Figure 3 yes Figure 1 Flow chart of step S105 in FIG.

[0051] Figure 4 yes Figure 3 Flow chart of step S302 in FIG.

[0052] Figure 5 yes Figure 3 Flow chart of step S304 in FIG.

[0053] Figure 6 yes Figure 5 Flow chart of step S502 in FIG.

[0054] Figure 7 yes Figure 1 Flow chart of step S106 in FIG.

[0055] Figure 8 is a structural schematic diagram of a speech style conversion device provided in an embodiment of the present application;

[0056] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0060] First, some nouns involved in this application are analyzed:

[0061] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0062] Voice Conversion (VC) is an artificial intelligence technology that can convert the speaker style or sound characteristics of a speech into another speaker style or characteristics while keeping the content unchanged. Voice style conversion technology involves the deep understanding and processing of speech signals, including but not limited to the conversion of acoustic features, prosody, emotional expression and individual characteristics. It is widely used in speech synthesis, virtual character design, personalized voice assistants and services.

[0063] Speech style transfer technology can be applied to a variety of scenarios. For example, in the financial field, it can be used to perform speech style transfer in scenarios such as virtual anchors, intelligent customer service, and tutorial video production.

[0064] At present, the speaker style of speech is mainly adjusted by using style description text of a specific speaker. However, in actual application scenarios, it is often difficult for text to fully and accurately describe the unique speech style of a specific speaker, which leads to a deviation between the converted speech style and the real style of the specific speaker, affecting the accuracy of speech style conversion.

[0065] Based on this, the embodiments of the present application provide a speech style conversion method and device, an electronic device and a storage medium, aiming to improve the accuracy of speech style conversion.

[0066] The speech style conversion method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the speech style conversion method in the embodiments of the present application is described.

[0067] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0068] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0069] The speech style conversion method provided in the embodiment of the present application relates to the field of artificial intelligence and financial technology. The speech style conversion method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the speech style conversion method, etc., but is not limited to the above forms.

[0070] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0071] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0072] Figure 1 is an optional flowchart of the speech style conversion method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S108.

[0073] Step S101, obtaining a speech synthesis sample; wherein the speech synthesis sample includes sample source speech data, sample reference speech data and sample speaker description text; wherein the sample reference speech data has sample reference speaker characteristics, and the sample speaker description text is used to describe the sample reference speaker characteristics;

[0074] Step S102, extracting speech features from the sample source speech data to obtain embedded features of the sample source speech;

[0075] Step S103, performing speaker extraction on the sample reference speech data to obtain sample speaker embedding features;

[0076] Step S104, extracting text features from the sample speaker description text to obtain embedded features of the sample speaker description;

[0077] Step S105, training a preset original speech style transfer model based on the sample source speech embedding features, the sample speaker embedding features and the sample speaker description embedding features to obtain a target speech style transfer model;

[0078] Step S106, performing speaker extraction on the pre-acquired target reference speech data to obtain target speaker embedding features; wherein the target reference speech data has target reference speaker features;

[0079] Step S107, extracting speech features from the pre-acquired target source speech data to obtain target source speech embedding features;

[0080] Step S108, performing style conversion on the target speaker embedding features and the target source speech embedding features through a target speech style conversion model to obtain target synthesized speech data; wherein the target synthesized speech data has target reference speaker features.

[0081] In steps S101 to S108 shown in the embodiment of the present application, feature extraction is performed on the acquired sample source speech data, sample reference speech data and sample speaker description text respectively, and the features are converted into numerical representations to obtain sample source speech embedding features, sample speaker embedding features and sample speaker description embedding features; then, the original speech style conversion model is trained using the sample source speech embedding features, sample speaker embedding features and sample speaker description embedding features to improve the model's learning ability for features, thereby improving the accuracy of speech style conversion and obtaining a target speech style conversion model; finally, feature extraction is performed on the pre-acquired target reference speech data and target source speech data respectively to obtain target speaker embedding features and target source speech embedding features; the target speech style conversion model is used to perform style conversion on the target speaker embedding features and the target source speech embedding features to generate target synthesized speech data that contains both the target source speech data content and the target reference speaker features, which can improve the naturalness of the speech and the efficiency and accuracy of speech style conversion.

[0082] In step S101 of some embodiments, the speech synthesis sample is collected, processed and labeled in advance by a technician, wherein the speaker of the sample reference speech data may be referred to as the sample reference speaker, and the speaker features of the sample reference speaker are the sample reference speaker features; and the sample speaker description text is used to describe the sample reference speaker features corresponding to the sample reference speaker;

[0083] Specifically, the sample speaker description text can be:

[0084] The sample reference speaker's voice sounds young and energetic, with a clear hint of laughter.

[0085] The sample reference speaker's voice appears warm and friendly, with a low, soothing rhythm.

[0086] The sample reference speaker's tone of voice appears firm and confident, with a hint of seriousness and authority.

[0087] In step S102 of some embodiments, speech features are extracted from the sample source speech data using a pre-trained speech coding model, speech content is extracted from the sample source speech data, and sample source speech embedding features are obtained, which can characterize the specific meaning of the sample source speech data.

[0088] Specifically, the speech coding model may be a WavLM model, a FastSpeech model, etc., but is not limited thereto.

[0089] See also Figure 2 In some embodiments, step S103 may include but is not limited to steps S201 to S202:

[0090] Step S201, performing spectrum conversion on the sample reference speech data to obtain sample speech spectrum data;

[0091] Step S202: extract style features from the sample speech spectrum data to obtain sample speaker embedding features.

[0092] In steps S201 to S202 shown in the embodiment of the present application, by performing spectrum conversion on the sample reference speech data to obtain sample speech spectrum data, and then performing style feature extraction on the sample speech spectrum data, it is possible to accurately extract the embedded features of the sample speaker, provide accurate data support for speech style conversion, and thus improve the accuracy of speech style conversion.

[0093] Before step S201 in some embodiments, the speech style conversion method may also include but is not limited to:

[0094] Performing denoising processing on the sample reference speech data to obtain denoised speech sample data;

[0095] Performing voice detection on the denoised speech sample data to obtain sample voice segment information;

[0096] Based on the sample vocal segment information, voice extraction is performed on the denoised voice sample data to obtain the final sample reference voice data.

[0097] It can effectively eliminate background noise in sample reference speech data and accurately identify the effective human voice part in the speech, thereby enhancing the reliability of the data and improving the processing efficiency of speech data.

[0098] In step S201 of some embodiments, the sample reference speech data can be converted from the time domain to the frequency domain by performing a short-time Fourier transform on the sample reference speech data, and then the frequency domain representation is filtered through a Mel filter to obtain the sample speech spectrum data.

[0099] Specifically, the sample speech spectrum data may be mel-cephalogram data, mel-spectrogram data, etc., but is not limited thereto.

[0100] In step S202 of some embodiments, style features are extracted from the sample speech spectrum data using a preset original speaker coding model to obtain sample speaker embedding features; wherein the sample speaker embedding features include but are not limited to: voiceprint features, timbre features, tone features, intonation features, etc., which can reflect the style features of the sample reference speaker in the sample reference speech data.

[0101] It is understandable that since the sample speaker embedding features are extracted through the original speaker encoding model, the expression accuracy of the sample speaker embedding features depends on the ability of the original speaker encoding model. Therefore, there may be differences between the sample speaker embedding features and the sample reference speaker features.

[0102] In some embodiments, the voice style switching method may also include but is not limited to the following steps:

[0103] Calculate speaker loss for sample speaker embedding features and sample reference speaker features to obtain speaker extraction loss data;

[0104] The original speaker encoding model is optimized based on the speaker extraction loss data to obtain the target speaker encoding model.

[0105] Among them, the speaker loss calculation can be calculated using loss functions such as mean square error loss function, cross entropy loss function, likelihood loss function, etc., but is not limited to these.

[0106] This can improve the accuracy of speaker feature extraction by the speaker encoding model.

[0107] In step S104 of some embodiments, text features may be extracted from the sample speaker description text using a pre-trained text encoding model, and the text may be converted into a numerical representation to obtain embedded features of the sample speaker description;

[0108] Among them, the text encoding model can be an embedding model based on deep learning, a Word2Vec model, a BERT model, a GloVe model, etc., but is not limited to these.

[0109] See also Figure 3In some embodiments, step S105 may include but is not limited to steps S301 to S304:

[0110] Step S301, extracting speech features from sample reference speech data to obtain embedded features of sample target speech;

[0111] Step S302, performing feature transformation on the sample source speech embedding features and the sample speaker embedding features through the original speech style conversion model to obtain the sample transformed speech embedding features;

[0112] Step S303, performing comparative learning based on the sample speaker embedding features and the sample speaker description embedding features to obtain comparative loss data;

[0113] Step S304, optimizing the original speech style transfer model based on the sample target speech embedding features, the sample transformed speech embedding features and the contrast loss data to obtain the target speech style transfer model.

[0114] In steps S301 to S304 shown in the embodiment of the present application, by extracting the sample target speech embedding features from the sample reference speech data, it is possible to ensure that the converted speech is highly consistent with the target style: the sample source speech embedding features and the sample speaker embedding features are transformed by the original speech style conversion model to obtain the sample transformed speech embedding features; and then by comparing the difference between the sample speaker embedding features and the sample speaker description embedding features, the loss function is calculated to obtain the contrast loss data, thereby optimizing the original speech style conversion model based on the sample target speech embedding features, the sample transformed speech embedding features and the contrast loss data to obtain the target speech style conversion model, which can effectively improve the accuracy and personalization ability of the speech style conversion model.

[0115] It should be noted that the sample target speech embedding features are used to determine whether there is a large gap in the speech embedding features after the speech style switching, so as to adjust the original speech style conversion model and improve the model's capabilities.

[0116] In step S301 of some embodiments, speech features are extracted from sample reference speech data using a pre-trained speech coding model, speech content is extracted from the sample reference speech data, and sample target speech embedding features are obtained, which can characterize the specific meaning of the sample reference speech data.

[0117] See also Figure 4 In some embodiments, step S302 may include but is not limited to steps S401 to S403:

[0118] Step S401, feature mapping is performed on the embedded features of the sample source speech through the original speech style conversion model to obtain the sample speech mapping features;

[0119] Step S402, adjusting the sample speech mapping feature based on the sample speaker embedding feature to obtain the original sample speech embedding feature;

[0120] Step S403, sampling and processing are performed on the original sample speech embedding features to obtain sample transformed speech embedding features.

[0121] In steps S401 to S403 shown in the embodiment of the present application, feature mapping is performed on the embedded features of the sample source speech through the original speech style conversion model to obtain more accurate sample speech mapping features, thereby providing an accurate benchmark for subsequent model optimization; then, feature adjustment is performed on the sample speech mapping features based on the sample speaker embedding features to ensure that the converted speech not only retains the speech content of the sample source speech data, but also incorporates the sample speaker embedding features contained in the sample reference speech data; finally, the sample transformed speech embedding features are obtained by sampling the adjusted original sample speech embedding features, thereby further enhancing the fluency and naturalness of the speech, thereby making the final speech style conversion result closer to the style expression of the sample speaker.

[0122] In some embodiments, the original speech style conversion model may be a pre-trained normalized stream model that can map the sample source speech embedding features to a preset latent expression space, and adjust the latent expression space based on the sample speaker embedding features, thereby completing the speech style conversion.

[0123] In step S403 of some embodiments, the original sample speech embedding features can be upsampled or downsampled to reduce the amount of calculation, and at the same time, the abruptness of the speech expression after feature adjustment can be flattened to improve the fluency and naturalness of speech synthesis.

[0124] See also Figure 5 In some embodiments, step S303 may also include but is not limited to steps S501 to S502:

[0125] Step S501, performing negative sampling on the sample speaker description embedding feature to obtain a negative sample speaker embedding feature;

[0126] Step S502, performing contrast loss calculation based on the negative sample speaker embedding features and the sample speaker embedding features to obtain contrast loss data.

[0127] In steps S501 to S502 shown in the embodiment of the present application, negative sample sampling is performed on the sample speaker description embedding feature to obtain the negative sample speaker embedding feature, and contrast loss calculation is performed based on the negative sample speaker embedding feature and the sample speaker embedding feature to obtain contrast loss data, which can provide optimization direction for model optimization.

[0128] In step S501 of some embodiments, a sample having a different embedded feature from the sample speaker description, i.e., a negative sample, is obtained to obtain a speaker embedded feature of the negative sample;

[0129] For example, suppose the sample speaker description text is: The sample reference speaker's voice sounds young and energetic, with a clear hint of laughter.

[0130] The negative sample can be speech data with the description text "the tone of the sample reference speaker appears firm and confident, with a hint of seriousness and authority". By extracting the speaker of the negative sample, the speaker embedding feature of the negative sample is obtained.

[0131] In some embodiments, step S502 may include but is not limited to the following steps:

[0132] Calculate the similarity between the negative sample speaker embedding features and the sample speaker embedding features to obtain sample similarity data;

[0133] Calculate the sample loss of the negative sample speaker embedding features and the sample speaker embedding features to obtain the sample loss data;

[0134] Based on the combination of sample similarity data and sample loss data, comparative loss data is obtained.

[0135] Specifically, the sample loss calculation can be performed using loss functions such as mean square error loss function, cross entropy loss function, likelihood loss function, etc., but is not limited thereto.

[0136] The similarity calculation may be performed using a metric such as cosine similarity or Euclidean distance, but is not limited thereto.

[0137] See also Figure 6 In some embodiments, step S304 includes but is not limited to steps S601 to S602:

[0138] Step S601, performing divergence loss calculation on the sample target speech embedding features and the sample transformed speech embedding features to obtain sample divergence loss data;

[0139] Step S602: adjusting parameters of the original speech style transfer model based on the sample divergence loss data and the contrast loss data to obtain a target speech style transfer model.

[0140] In steps S601 to S602 shown in the embodiment of the present application, the divergence loss between the sample target speech embedding features and the sample transformed speech embedding features is calculated, and the parameters of the original speech style conversion model are adjusted together in combination with the contrast loss data. This can significantly improve the accuracy and robustness of the model in the speech style conversion task, and can ensure that the speech embedding features of the speech data after style conversion are as close as possible to the speech embedding features of the sample reference speech data, thereby improving the accuracy of speech style conversion.

[0141] In step S601 of some embodiments, since the sample target speech embedding feature is used to determine whether there is a large gap in the speech embedding feature after the speech style switching, the divergence loss between the sample target speech embedding feature and the sample transformed speech embedding feature can be calculated to obtain the gap between the two and obtain the sample divergence loss data.

[0142] In step S602 of some embodiments, back propagation method, gradient descent method, momentum update method and other methods may be used to adjust parameters of the original speech style conversion model based on sample divergence loss data and contrast loss data.

[0143] In some embodiments, step S106 may include but is not limited to the following steps:

[0144] Performing spectrum conversion on the target reference speech data to obtain target speech spectrum data;

[0145] The target speaker encoding model is used to extract style features from the target speech spectrum data to obtain the target speaker embedding features.

[0146] It should be noted that the specific implementation of the above step S106 is basically the same as the implementation illustrated in the above steps S201 to S202, and will not be repeated here.

[0147] It is understandable that since the target speaker encoding model is trained, its feature extraction capability and accuracy meet the preset requirements. Therefore, it can be determined that the speaker represented by the target speaker embedded features is equivalent to the speaker represented by the target reference speaker features, and is the same speaker.

[0148] In step S107 of some embodiments, speech features are extracted from the target source speech data through a pre-trained speech coding model, speech content is extracted from the target source speech data, and target source speech embedding features are obtained, which can characterize the specific meaning of the target source speech data.

[0149] In one embodiment, in step S102, step S301 and step S107, the same speech coding model may be used to extract speech features, or different speech coding models may be used to extract speech features, but the invention is not limited thereto.

[0150] See also Figure 7 In some embodiments, step S108 may include but is not limited to steps S701 to S702:

[0151] Step S701, performing feature transformation on the target speaker embedding feature and the target source speech embedding feature through a target speech style conversion model to obtain a target speech synthesis embedding feature;

[0152] Step S702: Decode the target speech synthesis embedded features to obtain target synthesized speech data.

[0153] In the steps S701 to S702 shown in the embodiment of the present application, the target speaker embedding features and the target source speech embedding features are transformed by the target speech style conversion model to obtain the target speech synthesis embedding features; the target speech synthesis embedding features are decoded to obtain the target synthesized speech data. The speech style conversion is realized, and the obtained target speech synthesis data contains both the target source speech data content and the target synthesized speech data with the characteristics of the target reference speaker, thereby improving the accuracy of speech style conversion.

[0154] In some embodiments, step S701 may include but is not limited to the following steps:

[0155] Perform feature mapping on the target source speech embedding features through the target speech style conversion model to obtain the target speech mapping features;

[0156] Based on the target speaker embedding feature, the target speech mapping feature is adjusted to obtain the synthetic speech embedding feature;

[0157] The synthetic speech embedding features are sampled to obtain the target speech synthesis embedding features.

[0158] It should be noted that the specific implementation of the above step S701 is basically the same as the implementation illustrated in the above steps S401 to S403, and will not be repeated here.

[0159] In step S702 of some embodiments, the target speech synthesis embedded features are decoded by a pre-trained speech decoding model to obtain target synthesized speech data. The speech decoding model may be a WaveNet model, a Seq2Seq model, a Transformer model, etc., but is not limited thereto.

[0160] In some embodiments, before step S106, the speech style conversion method further includes:

[0161] Perform text feature extraction on the pre-acquired target speaker description text to obtain the target speaker description embedding feature;

[0162] Step S106 may include but is not limited to the following steps:

[0163] Performing spectrum conversion on the target reference speech data to obtain target speech spectrum data;

[0164] The target speaker encoding model is used to extract the style features of the target speech spectrum data to obtain the initial speaker embedding features;

[0165] The style of the initial speaker embedding features is adjusted based on the target speaker description embedding features to obtain the target speaker embedding features.

[0166] Specifically, the style adjustment can be achieved by comparing the target speaker description embedding features with the initial speaker embedding features to obtain the target speaker embedding features.

[0167] See also Figure 8 The embodiment of the present application also provides a speech style conversion device, which can implement the above-mentioned speech style conversion method, and the device includes:

[0168] The sample data acquisition module 801 is used to acquire a speech synthesis sample; wherein the speech synthesis sample includes sample source speech data, sample reference speech data and sample speaker description text; wherein the sample reference speech data has sample reference speaker characteristics, and the sample speaker description text is used to describe the sample reference speaker characteristics;

[0169] The sample speech feature extraction module 802 is used to extract speech features from the sample source speech data to obtain the sample source speech embedding features;

[0170] A sample speaker extraction module 803 is used to extract the speaker of the sample reference speech data to obtain the sample speaker embedding feature;

[0171] A sample text feature extraction module 804 is used to extract text features from the sample speaker description text to obtain sample speaker description embedding features;

[0172] The original model training module 805 is used to train the preset original speech style transfer model based on the sample source speech embedding features, the sample speaker embedding features and the sample speaker description embedding features to obtain the target speech style transfer model;

[0173] The target speaker extraction module 806 is used to perform speaker extraction on the pre-acquired target reference speech data to obtain a target speaker embedding feature; wherein the target reference speech data has a target reference speaker feature;

[0174] The target speech feature extraction module 807 is used to extract speech features from the pre-acquired target source speech data to obtain target source speech embedding features;

[0175] The target speech style conversion module 808 is used to perform style conversion on the target speaker embedding features and the target source speech embedding features through a target speech style conversion model to obtain target synthesized speech data; wherein the target synthesized speech data has target reference speaker features.

[0176] The specific implementation of the speech style conversion device is basically the same as the specific implementation of the above-mentioned speech style conversion method, and will not be repeated here.

[0177] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned voice style conversion method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0178] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0179] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0180] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the speech style conversion method of the embodiment of this application;

[0181] Input / output interface 903, used to implement information input and output;

[0182] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0183] A bus 905 that transmits information between the various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0184] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0185] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned speech style conversion method is implemented.

[0186] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0187] The speech style conversion method and device, electronic device and storage medium provided in the embodiments of the present application respectively extract features from the acquired sample source speech data, sample reference speech data and sample speaker description text, convert them into numerical representations, and obtain sample source speech embedding features, sample speaker embedding features and sample speaker description embedding features; then, the original speech style conversion model is trained using the sample source speech embedding features, sample speaker embedding features and sample speaker description embedding features to improve the model's learning ability for features, thereby improving the accuracy of speech style conversion and obtaining a target speech style conversion model; finally, feature extraction is performed on the pre-acquired target reference speech data and target source speech data respectively to obtain target speaker embedding features and target source speech embedding features; the target speech style conversion model is used to perform style conversion on the target speaker embedding features and the target source speech embedding features to generate target synthetic speech data that contains both the target source speech data content and the target reference speaker features, which can improve the naturalness of speech and the efficiency and accuracy of speech style conversion.

[0188] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0189] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0190] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0191] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0192] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0193] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0194] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0195] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0196] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0197] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0198] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for voice style transfer, It is characterized in that The method comprises: Acquire a speech synthesis sample; wherein the speech synthesis sample includes sample source speech data, sample reference speech data and sample speaker description text; wherein the sample reference speech data has sample reference speaker characteristics, and the sample speaker description text is used to describe the sample reference speaker characteristics; Extracting speech features from the sample source speech data to obtain sample source speech embedding features; Performing speaker extraction on the sample reference speech data to obtain sample speaker embedding features; Performing text feature extraction on the sample speaker description text to obtain sample speaker description embedding features; Training a preset original speech style conversion model based on the sample source speech embedding feature, the sample speaker embedding feature and the sample speaker description embedding feature to obtain a target speech style conversion model; Performing speaker extraction on the pre-acquired target reference speech data to obtain target speaker embedding features; wherein the target reference speech data has target reference speaker features; Extracting speech features from the pre-acquired target source speech data to obtain target source speech embedding features; The target speech style conversion model is used to perform style conversion on the target speaker embedding features and the target source speech embedding features to obtain target synthesized speech data; wherein the target synthesized speech data has the target reference speaker features.

2. The method according to claim 1, It is characterized in that The method of training a preset original speech style conversion model based on the sample source speech embedding feature, the sample speaker embedding feature and the sample speaker description embedding feature to obtain a target speech style conversion model includes: Extracting speech features from the sample reference speech data to obtain sample target speech embedding features; Performing feature transformation on the sample source speech embedding feature and the sample speaker embedding feature through the original speech style conversion model to obtain a sample transformed speech embedding feature; Performing comparative learning based on the sample speaker embedding feature and the sample speaker description embedding feature to obtain comparative loss data; The original speech style conversion model is optimized based on the sample target speech embedding feature, the sample transformed speech embedding feature and the contrast loss data to obtain the target speech style conversion model.

3. The method according to claim 2, It is characterized in that The step of performing feature transformation on the sample source speech embedding feature and the sample speaker embedding feature through the original speech style conversion model to obtain the sample transformed speech embedding feature includes: Performing feature mapping on the sample source speech embedding features through the original speech style conversion model to obtain sample speech mapping features; Performing feature adjustment on the sample speech mapping feature based on the sample speaker embedding feature to obtain the original sample speech embedding feature; The original sample speech embedding feature is sampled to obtain the sample transformed speech embedding feature.

4. The method according to claim 2, It is characterized in that The performing comparative learning based on the sample speaker embedding feature and the sample speaker description embedding feature to obtain comparative loss data includes: Performing negative sampling on the sample speaker description embedding feature to obtain a negative sample speaker embedding feature; A contrast loss calculation is performed based on the negative sample speaker embedding feature and the sample speaker embedding feature to obtain contrast loss data.

5. The method according to claim 2, It is characterized in that The method of optimizing the original speech style conversion model based on the sample target speech embedding feature, the sample transformed speech embedding feature and the contrast loss data to obtain the target speech style conversion model includes: Performing divergence loss calculation on the sample target speech embedding feature and the sample transformed speech embedding feature to obtain sample divergence loss data; The parameters of the original speech style transfer model are adjusted based on the sample divergence loss data and the contrast loss data to obtain the target speech style transfer model.

6. The method according to any one of claims 1 to 5, It is characterized in that The method of performing style conversion on the target speaker embedding feature and the target source speech embedding feature through the target speech style conversion model to obtain target synthesized speech data includes: Performing feature transformation on the target speaker embedding feature and the target source speech embedding feature through the target speech style conversion model to obtain a target speech synthesis embedding feature; The target speech synthesis embedded features are decoded to obtain target synthesized speech data.

7. The method according to any one of claims 1 to 5, It is characterized in that The extracting the speaker from the sample reference speech data to obtain the sample speaker embedding feature includes: Performing spectrum conversion on the sample reference speech data to obtain sample speech spectrum data; Style features are extracted from the sample speech spectrum data to obtain embedded features of the sample speaker.

8. A speech style conversion device, It is characterized in that The device comprises: A sample data acquisition module, used to acquire a speech synthesis sample; wherein the speech synthesis sample includes sample source speech data, sample reference speech data and sample speaker description text; wherein the sample reference speech data has sample reference speaker characteristics, and the sample speaker description text is used to describe the sample reference speaker characteristics; A sample speech feature extraction module is used to extract speech features from the sample source speech data to obtain sample source speech embedding features; A sample speaker extraction module, used to extract the speaker of the sample reference speech data to obtain a sample speaker embedding feature; A sample text feature extraction module is used to extract text features from the sample speaker description text to obtain sample speaker description embedding features; An original model training module, used to train a preset original speech style transfer model based on the sample source speech embedding feature, the sample speaker embedding feature and the sample speaker description embedding feature to obtain a target speech style transfer model; A target speaker extraction module is used to perform speaker extraction on the pre-acquired target reference speech data to obtain a target speaker embedding feature; wherein the target reference speech data has a target reference speaker feature; A target speech feature extraction module is used to extract speech features from the pre-acquired target source speech data to obtain target source speech embedding features; A target speech style conversion module is used to perform style conversion on the target speaker embedding features and the target source speech embedding features through the target speech style conversion model to obtain target synthesized speech data; wherein the target synthesized speech data has the target reference speaker features.

9. An electronic device, It is characterized in that The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.