Voice conversion method and device, electronic equipment and storage medium

By conducting Mel spectrogram conversion and image transformation training on the voice conversion model of smart phone customer service, the problem of unstable content extraction under the influence of different tones is solved, the accuracy and stability of customer service audio content extraction is improved, and maintenance is reduced cost.

CN118942482AActive Publication Date: 2024-11-12PING AN TECH (SHENZHEN) CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202411013012.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-12
Estimated Expiration
2044-07-25

AI Technical Summary

Technical Problem

When faced with customer service audio with different tones, the content extraction capability of the voice conversion model is unstable, affecting the customer service experience of customers.

Method used

By obtaining the pending customer service audio, using a pre-trained conversion model for speech conversion processing. The model is trained through Mel spectrogram conversion and image transformation, introducing timbre changes factors, adjusting the parameter content encoder and speaker encoder to improve content Robustness of extraction capability.

Benefits of technology

With the addition of customer service audio with different tones, the conversion model can extract content more accurately and stably, reducing the impact of tone on extraction capabilities, reducing the negative impact on customer experience, and reducing the maintenance cost of smart seats.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942482A_ABST
    Figure CN118942482A_ABST
Patent Text Reader

Abstract

The invention provides a voice conversion method and device, electronic equipment and a storage medium, and belongs to the field of financial science and tech, and the method comprises the steps: obtaining a to-be-processed customer service audio, and obtaining a target customer service audio through the voice conversion of a pre-trained conversion model; the conversion model training comprises the following steps: acquiring an audio training set comprising a plurality of training audios and label content features thereof; obtaining a target Mel spectrogram of each training audio based on Mel spectrogram conversion and image conversion, wherein the training audios in the audio training set alternately select one of image stretching or image compression for image conversion; obtaining a target content feature of each training audio through the initial model; when the preset training ending condition is not met, performing first parameter adjustment on a content encoder of the initial model according to the multiple target content features and the label content features thereof; and continuing to train the initial model after parameter adjustment based on the audio training set until a conversion model is obtained. The stability of voice conversion can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of financial technology, and in particular to a voice conversion method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of artificial intelligence technology, the traditional financial industry is gradually transforming to financial technology, and various financial service systems are being continuously upgraded and updated through artificial intelligence technology. In the financial industry, there is a customer service link. Due to the special attributes of the financial industry, it is necessary to provide telephone customer service for a large number of users. In the past, telephone customer service required manual communication with customers. After the telephone customer service was upgraded with artificial intelligence technology, it can automatically answer relevant consultation information based on the communication information with customers.

[0003] Existing intelligent telephone customer service services (also known as smart agents) generally store different customer service audios recorded by a customer service staff according to different business scenarios in a preset database in advance. Then, in the actual customer service process, the content spoken by the customer service is recognized through a speech recognition model to obtain the recognized content. Then, according to the speech recognition content, the corresponding customer service audio is extracted from the database. Then, the customer service audio is converted into speech through a speech conversion model to generate audio to reply to the customer.

[0004] With the development of financial services, different customer service personnel may be asked to record more complex customer service audio to update the preset database. However, the content extraction ability of the speech conversion model will be affected by the timbre. When the speech conversion model performs speech conversion on the newly added customer service audio in the preset database, the extracted content quality is not stable and it is impossible to output good quality customer service audio for replying to customers, thus affecting the customer service experience. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to propose a voice conversion method, device, electronic device and storage medium, aiming to improve the content extraction capability of customer service audio, thereby reducing the impact of timbre on content extraction capability when adding customer service audio with different timbre.

[0006] To achieve the above object, a first aspect of an embodiment of the present application provides a voice conversion method, the method comprising:

[0007] Get the customer service audio to be processed;

[0008] Performing speech conversion processing on the customer service audio through a pre-trained conversion model, and outputting target customer service audio;

[0009] The conversion model is trained by the following steps:

[0010] Acquire an audio training set, wherein the audio training set includes a plurality of training audios and a label content feature corresponding to each of the training audios;

[0011] Performing mel-spectrogram conversion processing on the plurality of training audios respectively to obtain a mel-spectrogram corresponding to each of the training audios;

[0012] For each of the training audios, performing image transformation on the mel-spectrogram corresponding to the training audio to obtain a target mel-spectrogram, wherein the image transformation is image stretching or image compression, and the training audios in the audio training set alternately select one of image stretching and image compression for image transformation;

[0013] Inputting the plurality of target mel-spectrograms into an initial model respectively to obtain target content features corresponding to each of the training audios, wherein the initial model includes a content encoder;

[0014] When the current initial model does not meet the preset training end condition, first adjusting the parameters of the content encoder according to the target content features and label content features corresponding to the plurality of training audios;

[0015] The initial model after parameter adjustment is continuously trained based on the audio training set until the preset training end condition is met to obtain the conversion model.

[0016] In some embodiments, the mel-spectrogram corresponding to each of the training audios includes a plurality of segmented mel-spectrograms; the audio training set also includes a label timbre feature corresponding to each of the training audios; the initial model also includes a speaker encoder, and the speaker encoder is parallel to the content encoder;

[0017] The step of performing mel-spectrogram conversion on the audio training set to obtain a plurality of mel-spectrograms includes:

[0018] Performing random audio segmentation processing on each of the training audios to obtain a plurality of segmented audios corresponding to each of the training audios;

[0019] Performing mel-spectrogram conversion processing on the plurality of segmented audios respectively to obtain the segmented mel-spectrogram corresponding to each segmented audio;

[0020] After inputting the plurality of target mel-spectrograms into the initial model respectively, the method further comprises:

[0021] Using the speaker encoder, respectively extract the timbre features of the target mel-spectrogram corresponding to each of the segmented mel-spectrograms to obtain the segmented timbre features corresponding to each of the segmented mel-spectrograms;

[0022] For each of the training audios, according to a preset combination ratio coefficient, a feature combination process is performed on a plurality of the segmented timbre features corresponding to the training audio to obtain a target timbre feature;

[0023] When the current initial model does not meet the preset training end condition, the method further includes:

[0024] The speaker encoder is subjected to a second parameter adjustment according to the target timbre feature and the label timbre feature corresponding to each of the training audios.

[0025] In some embodiments, when the current image transformation includes the image compression, performing image transformation on the corresponding Mel-spectrogram to obtain a target Mel-spectrogram includes:

[0026] Get the current compression ratio;

[0027] According to the compression ratio coefficient, image compression is performed on the mel-spectrogram corresponding to the training audio in the frequency domain dimension to obtain a first mel-spectrogram;

[0028] The first mel-spectrogram is predicted and completed to obtain the target mel-spectrogram having the same size as the mel-spectrogram.

[0029] In some embodiments, when the image transformation includes the image stretching, performing image transformation on the corresponding Mel-spectrogram to obtain a target Mel-spectrogram includes:

[0030] Get the current stretch ratio;

[0031] According to the stretching ratio coefficient, the mel-spectrogram corresponding to the training audio is stretched in the frequency domain dimension to obtain a second mel-spectrogram;

[0032] The second mel-spectrogram is cropped according to the size of the mel-spectrogram to obtain the target mel-spectrogram with the same size as the mel-spectrogram.

[0033] In some embodiments, the audio training set further includes a label audio corresponding to each of the training audios; the initial model further includes a speaker encoder; the target Mel spectrogram corresponding to each of the training audios includes the target Mel spectrogram for timbre feature extraction;

[0034] The obtaining of the current compression ratio coefficient comprises:

[0035] Determining the compression ratio coefficient according to the fusion coefficient and the current preset compression ratio coefficient, wherein the fusion coefficient of the training audio is obtained by extracting the timbre feature of the target mel-spectrogram for timbre feature extraction corresponding to the previous training audio by the speaker encoder;

[0036] After obtaining the target content feature corresponding to each training audio, the method further includes:

[0037] Performing audio conversion processing on the plurality of target content features respectively to obtain a target audio corresponding to each of the training audios;

[0038] When the current initial model does not meet the preset training end condition, the method further includes:

[0039] The initial model is subjected to a third parameter adjustment according to the target audio and the label audio corresponding to each of the training audios.

[0040] In some embodiments, the first parameter adjustment of the content encoder according to the target content features and the label content features corresponding to the plurality of training audios includes:

[0041] Perform a minimization loss analysis based on target content features and label content features corresponding to the plurality of training audios to determine a first loss value;

[0042] Perform first parameter adjustment on the content encoder according to the first loss value.

[0043] In some embodiments, performing a second parameter adjustment on the speaker encoder according to the target timbre feature and the label timbre feature corresponding to each of the training audios includes:

[0044] Performing angular edge loss analysis according to the plurality of segmented timbre features corresponding to each of the training audios to determine a second loss value;

[0045] Performing a maximization loss analysis based on the target timbre feature and the label timbre feature corresponding to each of the training audios to determine a third loss value;

[0046] A second parameter adjustment is performed on the speaker encoder with respect to the second loss value and the third loss value.

[0047] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a speech conversion device, the device comprising:

[0048] A customer service audio acquisition module is used to acquire customer service audio to be processed;

[0049] A target customer service audio output module is used to perform speech conversion processing on the customer service audio through a pre-trained conversion model and output the target customer service audio;

[0050] The conversion model is trained by the following steps:

[0051] Acquire an audio training set, wherein the audio training set includes a plurality of training audios and a label content feature corresponding to each of the training audios;

[0052] Performing mel-spectrogram conversion processing on the plurality of training audios respectively to obtain a mel-spectrogram corresponding to each of the training audios;

[0053] For each of the training audios, performing image transformation on the mel-spectrogram corresponding to the training audio to obtain a target mel-spectrogram, wherein the image transformation is image stretching or image compression, and the training audios in the audio training set alternately select one of image stretching and image compression for image transformation;

[0054] Inputting the plurality of target mel-spectrograms into an initial model respectively to obtain target content features corresponding to each of the training audios, wherein the initial model includes a content encoder;

[0055] When the current initial model does not meet the preset training end condition for being determined as the conversion model, first adjusting the parameters of the content encoder according to the target content features and label content features corresponding to the plurality of training audios;

[0056] The initial model after parameter adjustment is continuously trained based on the audio training set until the current initial model is determined as the conversion model.

[0057] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0058] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0059] The speech conversion method, device, electronic device and storage medium proposed in the present application obtain customer service audio to be processed; perform speech conversion processing on the customer service audio through a pre-trained conversion model, and output a target customer service audio, and the conversion model is trained by the following steps: obtain an audio training set, the audio training set includes multiple training audios and label content features corresponding to each training audio; perform mel-spectrogram conversion processing on multiple training audios respectively to obtain a mel-spectrogram corresponding to each training audio; for each training audio, perform image transformation on the mel-spectrogram corresponding to the training audio to obtain a target mel-spectrogram, and the image transformation is image stretching or image compression, and the training audio in the audio training set alternately selects one of image stretching or image compression for image transformation; input multiple target mel-spectrograms into an initial model respectively to obtain a target content feature corresponding to each training audio, and the initial model includes a content encoder; when the current initial model does not meet the preset training end condition, the content encoder is first adjusted according to the target content features and label content features corresponding to the multiple training audios; the initial model after parameter adjustment is continuously trained based on the audio training set until the preset training end condition is met to obtain a conversion model. By performing image transformation on the Mel-spectrogram corresponding to the training audio, the original timbre characteristics of the training audio are changed, thereby introducing timbre change factors in the training process. Therefore, performing the first parameter adjustment on the label content characteristics through the target content characteristics corresponding to multiple training audios is helpful to improve the content encoder's ability to extract content information that remains unchanged before and after the timbre change characteristics in the Mel-spectrogram, thereby reducing the impact of timbre on the content extraction capability of the conversion model; and, since the training audio in the audio training set alternately selects one of image stretching or image compression for image transformation, the timbre feature changes of high-frequency timbre and low-frequency timbre can be introduced alternately during the training process, so that the first parameter adjustment becomes a parameter adjustment for enhancing the content extraction capability for diversified timbre, thereby making the content extraction capability enhanced by the content encoder more robust relative to the customer service audio with diversified timbre, reducing the impact of timbre on the content extraction capability. In this way, when adding customer service audio with different timbre, the conversion model can also extract content from the newly added customer service audio more accurately and stably, thereby reducing the impact of the newly added customer service audio on the customer's customer service experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a flow chart of a voice conversion method provided by an embodiment of the present application;

[0061] Figure 2 yes Figure 1 A schematic diagram of an embodiment of a training step of a conversion model used in the process shown;

[0062] Figure 3 yes Figure 1 A schematic diagram of another embodiment of a training step of a conversion model used in the process shown;

[0063] Figure 4 yes Figure 2 A schematic diagram of an embodiment of a sub-step of step 230 in FIG.

[0064] Figure 5 yes Figure 2 A schematic diagram of another sub-step embodiment of step 230 in FIG.

[0065] Figure 6 yes Figure 1 A schematic diagram of another embodiment of a training step of a conversion model used in the process shown;

[0066] Figure 7 yes Figure 1 A schematic diagram of another embodiment of a training step of a conversion model used in the process shown;

[0067] Figure 8 yes Figure 2 A schematic diagram of an embodiment of a sub-step of step 250;

[0068] Fig. 9 yes Figure 3 A schematic diagram of an embodiment of a sub-step of step 350 in FIG.

[0069] Fig.10 is a structural diagram of a speech conversion device provided in an embodiment of the present application;

[0070] Fig.11 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0072] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0073] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0074] First, some nouns involved in this application are analyzed:

[0075] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0076] With the rapid development of artificial intelligence technology, the traditional financial industry is gradually transforming to financial technology, and various financial service systems are being continuously upgraded and updated through artificial intelligence technology. In the financial industry, there is a customer service link. Due to the special attributes of the financial industry, it is necessary to provide telephone customer service for a large number of users. In the past, telephone customer service required manual communication with customers. After the telephone customer service was upgraded with artificial intelligence technology, it can automatically answer relevant consultation information based on the communication information with customers.

[0077] Existing intelligent telephone customer service services (also known as smart agents) generally store different customer service audios recorded by a customer service staff according to different business scenarios in a preset database in advance. Then, in the actual customer service process, the content spoken by the customer service is recognized through a speech recognition model to obtain the recognized content. Then, according to the speech recognition content, the corresponding customer service audio is extracted from the database. Then, the customer service audio is converted into speech through a speech conversion model to generate audio to reply to the customer.

[0078] With the development of financial services, different customer service personnel may be asked to record more complex customer service audio to update the preset database. However, the content extraction ability of the speech conversion model will be affected by the timbre. When the speech conversion model performs speech conversion on the newly added customer service audio in the preset database, the extracted content quality is not stable and it is impossible to output good quality customer service audio for replying to customers, thus affecting the customer service experience.

[0079] Based on this, the embodiments of the present application provide a speech conversion method, device, electronic device and storage medium, which aim to improve the content extraction capability of customer service audio, thereby reducing the impact of timbre on content extraction capability when adding customer service audio with different timbre. Specifically, the method obtains customer service audio to be processed; performs speech conversion processing on the customer service audio through a pre-trained conversion model, and outputs a target customer service audio, and the conversion model is trained through the following steps: obtaining an audio training set, the audio training set includes multiple training audios and label content features corresponding to each training audio; performing mel-spectrogram conversion processing on multiple training audios respectively to obtain a mel-spectrogram corresponding to each training audio; for each training audio, performing image transformation on the mel-spectrogram corresponding to the training audio to obtain a target mel-spectrogram, and the image transformation is image stretching or image compression, and the training audio in the audio training set alternately selects one of image stretching or image compression for image transformation; inputting multiple target mel-spectrograms into an initial model respectively to obtain a target content feature corresponding to each training audio, and the initial model includes a content encoder; when the current initial model does not meet the preset training end condition for being determined as a conversion model, the content encoder is first adjusted according to the target content features and label content features corresponding to the multiple training audios; the initial model after parameter adjustment is continuously trained based on the audio training set until the preset training end condition is met to obtain a conversion model. By performing image transformation on the Mel-spectrogram corresponding to the training audio, the original timbre characteristics of the training audio are changed, thereby introducing timbre change factors in the training process. Therefore, performing the first parameter adjustment on the label content characteristics through the target content characteristics corresponding to multiple training audios is helpful to improve the content encoder's ability to extract content information that remains unchanged before and after the timbre change characteristics in the Mel-spectrogram, thereby reducing the impact of timbre on the content extraction capability of the conversion model; and, since the training audio in the audio training set alternately selects one of image stretching or image compression for image transformation, the timbre feature changes of high-frequency timbre and low-frequency timbre can be introduced alternately during the training process, so that the first parameter adjustment becomes a parameter adjustment for enhancing the content extraction capability for diversified timbre, thereby making the content extraction capability enhanced by the content encoder more robust relative to the customer service audio with diversified timbre, reducing the impact of timbre on the content extraction capability. In this way, when adding customer service audio with different timbre, the conversion model can also extract content from the newly added customer service audio more accurately and stably, thereby reducing the impact of the newly added customer service audio on the customer's customer service experience. Moreover, since the content extraction capability of the conversion model is enhanced, when new customer service audio with different tones is added, the probability of retraining the conversion model can be effectively reduced, thereby reducing the maintenance cost of the intelligent agent.

[0080] The speech conversion method, device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the speech conversion method in the embodiments of the present application is described.

[0081] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0082] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0083] The speech conversion method provided in the embodiment of the present application relates to the field of financial technology. The speech conversion method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the speech conversion method, etc., but is not limited to the above forms.

[0084] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0085] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0086] Reference Figure 1 As shown, Figure 1 is a flow chart of a voice conversion method provided by an embodiment of the present application, Figure 1 The method may include but is not limited to steps 110 to 120.

[0087] Step 110: Obtain customer service audio to be processed;

[0088] Step 120: Perform speech conversion processing on the customer service audio through the pre-trained conversion model to output the target customer service audio.

[0089] In one embodiment, the customer service audio refers to a template audio used for voice reply based on the information provided by the customer. The customer service audio can be stored in a preset audio database, wherein the timbres of the multiple customer service audios stored in the preset audio database can be the same or different, which is not specifically limited here.

[0090] In one embodiment, in the process of obtaining the customer service audio to be processed, the user's interactive voice can be obtained, and voice recognition can be performed on the interactive voice to obtain voice recognition information. According to the voice recognition information, the customer service audio corresponding to the voice recognition information is called in a preset audio database to obtain the customer service audio to be processed.

[0091] In one embodiment, the voice conversion process refers to the operation of performing a timbre conversion on the customer service audio to be processed and outputting a target customer service audio under the premise of retaining the content represented by the customer service audio to be processed. The target customer service audio refers to the audio that expresses the content represented by the customer service audio through the target timbre. In addition, the target timbre can be called from a preset timbre library, which is not limited here.

[0092] In one embodiment, the pre-trained conversion model may include a content encoder, a speaker encoder, a decoder, and an audio generation network, wherein the content encoder is parallel to the speaker encoder and connected to the decoder in parallel, and the decoder is connected to the content encoder. In the process of voice conversion processing, the customer service audio may be firstly converted to a mel-spectrogram to obtain a mel-spectrogram corresponding to the customer service audio, and then the mel-spectrogram may be input into the pre-trained conversion model, the content encoder in the model may extract content features from the mel-spectrogram to obtain content features, the speaker encoder in the model may extract timbre features from the mel-spectrogram to obtain timbre features, the decoder in the model may perform feature fusion processing on the content features and timbre features to obtain target audio features, and then the audio generation network in the model may perform audio conversion according to the target audio features, so that the target audio features may be converted into target customer service audio. Among them, the pre-trained conversion model may be an autoencoder-based voice conversion (AutoVC) model, a VoiceCycleGAN model, etc., which are not specifically limited here. In addition, the audio generation network can be a generative adversarial network (GAN), a convolutional neural network (CNN) or a recurrent neural network (RNN), etc., which is not specifically limited here.

[0093] Reference Figure 2 As shown, Figure 2 Shows Figure 1The training steps of the conversion model used in the process are shown. In one embodiment, the conversion model can be obtained by training the initial model through the following steps, wherein the initial model includes a content encoder, a speaker encoder, a decoder, and an audio generation network.

[0094] Step 210: Obtain an audio training set, where the audio training set includes multiple training audios and label content features corresponding to each training audio;

[0095] Step 220: performing mel-spectrogram conversion processing on multiple training audios respectively to obtain a mel-spectrogram corresponding to each training audio;

[0096] Step 230: for each training audio, perform image transformation on the corresponding Mel-spectrogram to obtain a target Mel-spectrogram, the image transformation is image stretching or image compression, and the training audio in the audio training set alternately selects one of image stretching or image compression for image transformation;

[0097] Step 240: Input multiple target mel-spectrograms into the initial model respectively to obtain target content features corresponding to each training audio;

[0098] Step 250: When the current initial model does not meet the preset training end condition, the content encoder is first adjusted according to the target content features and the label content features corresponding to the plurality of training audios;

[0099] Step 260: Continue to train the adjusted initial model based on the audio training set until the preset training end conditions are met to obtain a conversion model.

[0100] In one embodiment, the training audio refers to the customer service audio used to train the initial model. The training audio may be a customer service audio in a preset audio database. In addition, the plurality of training audios may be training audios with the same timbre and different contents, training audios with different timbres and the same contents, training audios with different timbres and different contents, etc., which are not specifically limited here.

[0101] In one embodiment, the label content feature refers to the content feature obtained by extracting features from the mel-spectrogram by a content encoder after the training audio is converted into a mel-spectrogram without image transformation of the mel-spectrogram. The label content feature may be the label content feature obtained before starting the initial model training, or the label content feature updated by the content encoder after adjusting parameters based on multiple training audios and the previous training round, etc., which is not specifically limited here.

[0102] In one embodiment, when the current initial model is not determined as the conversion model, after a training round is completed, the label content features corresponding to each training audio in the audio training set can be updated: first, the mel-spectrogram conversion process is performed on multiple training audios respectively to obtain the mel-spectrogram corresponding to each training audio, and then, the content feature extraction is performed on the mel-spectrogram corresponding to each training audio according to the content encoder after parameter adjustment to obtain the updated label content features corresponding to each training audio. Since the content encoder is adjusted in each training round (that is, the first parameter adjustment mentioned below), the label content features are updated by using the content encoder after parameter adjustment in the previous training round, so that the loss value corresponding to the content feature extraction process of the current training round can be more accurately reflected, so that the parameter adjustment of the content encoder in the current training round can be more accurate, and then the content extraction capability of the conversion model can be made stronger.

[0103] In one embodiment, image transformation refers to an operation of performing image processing on a mel-spectrogram to obtain a target mel-spectrogram during the model training phase. The target mel-spectrogram refers to a mel-spectrogram after the mel-spectrogram has been image transformed. In addition, in addition to alternately selecting one of image stretching and image compression for the training audio, image transformation may also include specific image processing operations such as sharpening, smoothing, and segmentation, which are not specifically limited here. It should be noted that image transformation is only used to assist in model parameter adjustment during the model training phase. Therefore, in the actual process of voice conversion processing of the customer service audio, the mel-spectrogram corresponding to the customer service audio is not subjected to image transformation.

[0104] In one embodiment, image stretching refers to changing the distribution of pixels or pixel blocks in the mel-spectrogram to achieve the size extension of the mel-spectrogram in a certain direction; image stretching refers to changing the distribution of pixels or pixel blocks in the mel-spectrogram to achieve the size contraction of the mel-spectrogram in a certain direction.

[0105] In one embodiment, the training audio alternately selects one of image stretching or image compression for image transformation, which means that one training audio selects an operation different from the processing selected by the previous training audio in image stretching and image compression. For example, assuming that an audio training set includes three training audios A, B and C in sequence, training audio A selects image stretching, and the image transformation of the Mel-spectrogram corresponding to the training audio A includes image stretching, then training audio B will alternately select image compression, and the image transformation of the Mel-spectrogram corresponding to the training audio B alternately includes image compression but does not include image stretching. Similarly, relative to training audio B, training audio B will alternately select image stretching, and the image transformation of the Mel-spectrogram corresponding to the training audio C alternately includes image stretching but does not include image compression.

[0106] In one embodiment, at the beginning of each training round, a random selection is made from image stretching and image compression to determine which of the image stretching and image compression is performed on the mel-spectrogram corresponding to the first training audio in the audio training set.

[0107] In one embodiment, when the current training round is not the first training round, according to the one selected by the first training audio in the audio training set in the previous training round between image stretching and image compression, it is determined that the first training audio in the audio training set in the current training round alternately selects one of image stretching and image compression. For example, assuming that the image transformation of the mel-spectrogram corresponding to the first training audio in the audio training set in the previous training round includes image compression, then in the current training round, the first training audio in the audio training set alternately selects image stretching. By allowing the first training audio to alternately select one of image stretching and image compression relative to the previous training round in each training round, the mel-spectrogram corresponding to each training audio can undergo image stretching and image compression during the entire model training process, so that the content encoder can perform at least one content feature on different mel-spectrograms corresponding to each training audio, thereby reducing the biased influence caused by a single image transformation method, so that the content extraction capability of the content encoder can be improved, thereby improving the quality of the target customer service audio.

[0108] In one embodiment, since the initial model includes a content encoder, a speaker encoder, a decoder and an audio generation network, for a target mel-spectrogram, after the target mel-spectrogram is input into the output model, in addition to outputting the target content features corresponding to the training audio through the content encoder, the target timbre features corresponding to the training audio can also be output through the speaker encoder. In addition, the target fusion features corresponding to the training audio can also be output through the decoder. In addition, the target audio corresponding to the training audio can also be output through the audio generation network.

[0109] In one embodiment, the preset training end condition may include the accuracy of content feature extraction of the content encoder, or may include the accuracy of the output target audio, the accuracy of timbre feature extraction of the speaker encoder, etc. on the basis of the accuracy of content feature extraction of the content encoder, and is not specifically limited here. Specifically, after the current initial model completes the conversion of multiple training audios in the audio training set, the current training round ends. At this time, it is necessary to determine whether the current initial model meets the preset training end condition based on the data output by the initial model (such as the target content feature). If not, it is necessary to adjust the hyperparameters of the current initial model. If it is satisfied, the current initial model is determined as the conversion model.

[0110] In one embodiment, the first parameter adjustment refers to an operation of adjusting the parameters of the content encoder based on the hyperparameters determined based on the target content features and label content features corresponding to the multiple training audios. The hyperparameters determined based on the target content features and label content features corresponding to the multiple training audios can be determined by a preset parameter adjustment function, or can be hyperparameters input by a technician based on the target content features and label content features corresponding to the multiple training audios, etc., which are not specifically limited here.

[0111] In one embodiment, in the process of continuing to train the adjusted initial model based on the audio training set, multiple additional training audios and label content features corresponding to each additional training audio can be first obtained, and then the audio training set is updated according to the multiple additional training audios and the label content features corresponding to each additional training audio, and the adjusted initial model is continued to be trained based on the updated audio training set until the current initial model is determined as a conversion model. By updating the audio training set, the scenario of adding customer service audio to the preset database can be simulated, so that the content encoder can be adjusted for the scenario of adding customer service audio to the preset database, and the robustness of content feature extraction can be improved.

[0112] In one embodiment, the audio training set may further include a label timbre feature corresponding to each training audio. After the content encoder is first tuned according to the target content features and label content features corresponding to the plurality of training audios, the speaker encoder may be second tuned according to the target timbre features and label timbre features corresponding to the plurality of training audios. Since the target Mel spectrogram is affected by the image transformation, the frequency spectrum displayed by the target Mel spectrogram also changes, and the timbre is related to the frequency. Therefore, when the frequency spectrum displayed by the target Mel spectrogram changes, the timbre represented by the target Mel spectrogram will also change. In theory, the timbre represented by the target Mel spectrogram is different from the timbre of the original training audio. Based on this, by performing a second parameter adjustment on the speaker encoder, the target timbre features output by the speaker encoder after the second parameter adjustment can be distanced from the label timbre features, thereby enhancing the speaker encoder's ability to distinguish timbre differences, and further enabling the speaker encoder to better distinguish the timbre of different customer service audios in the actual customer service business process, which is beneficial to improving the robustness and generalization ability of the speaker encoder. Therefore, when different customer service personnel record customer service audio and store it in a preset database, the speaker encoder can better extract the timbre features of the customer service audio recorded by different customer service personnel, so that the conversion model can more accurately perform speech conversion of the customer service audio.

[0113] In one embodiment, the second parameter adjustment refers to an operation of adjusting the parameters of the speaker encoder based on the hyperparameters determined based on the target timbre features and the label timbre features corresponding to the multiple training audios. The hyperparameters determined based on the target timbre features and the label timbre features corresponding to the multiple training audios can be determined by a preset parameter adjustment function, or can be hyperparameters input by a technician based on the target timbre features and the label timbre features corresponding to the multiple training audios, etc., which are not specifically limited here.

[0114] In one embodiment, after the content encoder is first adjusted according to the target content features and label content features corresponding to the multiple training audios, the speaker encoder may also be second adjusted according to the target timbre features and label timbre features corresponding to the multiple training audios. The second parameter adjustment refers to the operation of adjusting the parameters of the speaker encoder based on the hyperparameters determined based on the target timbre features and label timbre features corresponding to the multiple training audios. Among them, the hyperparameters determined based on the target timbre features and label timbre features corresponding to the multiple training audios can be determined by a preset parameter adjustment function, or can be hyperparameters input by a technician based on the target timbre features and label timbre features corresponding to the multiple training audios, etc., which are not specifically limited here.

[0115] The following examples illustrate the application scenarios of the voice conversion method provided in the embodiments of the present application. Assume that the current preset audio database includes multiple fund customer service audios A, and the multiple fund customer service audios A are all recorded by customer service personnel B. The current fund intelligent agent function can perform voice conversion on the multiple fund customer service audios A through the current conversion model. When the preset audio database adds multiple fund customer service audios D recorded by customer service personnel C, since the conversion model adjusts the parameters of the content encoder based on the content features that introduce image transformation factors during the training process, the content extraction capability of the conversion model is enhanced. When voice conversion is performed on the fund customer service audio D, the conversion model can reduce the impact caused by the different timbre of customer service personnel C and customer service personnel B, and accurately and stably extract the content in the fund customer service audio D, so that the fund customer service audio D can be accurately output to correspond to the target customer service audio. Therefore, the technical personnel do not need to retrain the conversion model, which reduces the maintenance cost.

[0116] See also Figure 3 , Figure 3 Shows Figure 1 Another training step of the conversion model used in the process. In one embodiment, the mel-spectrogram corresponding to each training audio includes a plurality of segmented mel-spectrograms, and the audio training set also includes a label timbre feature corresponding to each training audio.

[0117] Step 220 may include the following sub-steps:

[0118] Step 310: performing random audio segmentation processing on each training audio to obtain a plurality of segmented audios corresponding to each training audio;

[0119] Step 320: performing Mel-spectrogram conversion processing on the multiple segmented audios respectively to obtain a segmented Mel-spectrogram corresponding to each segmented audio;

[0120] After the multiple target mel-spectrograms are respectively input into the initial model, the method further includes the following steps:

[0121] Step 330: extracting timbre features from the target mel-spectrogram corresponding to each segmented mel-spectrogram through the speaker encoder to obtain segmented timbre features corresponding to each segmented mel-spectrogram;

[0122] Step 340: for each training audio, perform feature combination processing on a plurality of segmented timbre features corresponding to the training audio according to a preset combination ratio coefficient to obtain a target timbre feature;

[0123] When the current initial model does not meet the preset training end condition, the method further includes the following steps:

[0124] Step 350: Perform a second parameter adjustment on the speaker encoder according to the target timbre features and label timbre features corresponding to each training audio.

[0125] In one embodiment, the audio segmentation process refers to the operation of segmenting the audio into multiple segmented audios according to a specific segmentation ratio; the random audio segmentation process refers to the operation of segmenting the audio into multiple segmented audios according to a random segmentation ratio. The number of segmented audios obtained by the random audio segmentation process may be the same or different, which is not specifically limited here.

[0126] In one embodiment, the segmented Mel spectrogram corresponding to each segmented audio is used for timbre feature extraction, but not for content feature extraction. That is to say, for a training audio, the training audio is first subjected to random audio segmentation processing to obtain a plurality of corresponding segmented audios. Then, the training audio and the plurality of segmented audios corresponding to the training audio are respectively subjected to Mel spectrogram conversion processing to obtain a Mel spectrogram corresponding to the complete training audio and a Mel spectrogram corresponding to each segmented audio. Then, the Mel spectrogram corresponding to the complete training audio and the Mel spectrogram corresponding to each segmented audio are subjected to the same image transformation to obtain a target Mel spectrogram corresponding to the complete training audio and a target Mel spectrogram corresponding to each segmented audio (that is, the target Mel spectrogram corresponding to the segmented Mel spectrogram mentioned in step 330). The target Mel spectrogram corresponding to the complete training audio is used to input the content feature encoder, and the target content feature is obtained by the content feature encoder. The target Mel spectrogram corresponding to the segmented audio is used to input the speaker encoder, and the segmented timbre feature is obtained by the speaker encoder.

[0127] In one embodiment, feature combination processing refers to, for a plurality of segmented timbre features corresponding to a training audio, weighted averaging the values ​​of each segmented timbre feature through a preset combination ratio coefficient to form a timbre feature (a target timbre feature in this embodiment) that can represent the plurality of segmented timbre features. The preset combination ratio coefficient refers to a coefficient used for weighted calculation of each segmented timbre feature, and for each training audio, the preset combination ratio coefficient used for each training audio corresponds to the number of segmented audios corresponding to each training audio.

[0128] It should be noted that in the actual process of voice conversion of the customer service audio, the customer service audio will also be subjected to random audio segmentation processing to obtain multiple customer service segmented audios corresponding to the customer service audio. The timbre features of the Mel-spectrograms corresponding to the multiple customer service segmented audios are extracted through the speaker encoder to obtain the segmented timbre features corresponding to the multiple customer service segmented audios. The segmented timbre features corresponding to the multiple customer service segmented audios are subjected to feature combination processing to obtain the target timbre features corresponding to the customer service audio.

[0129] exist Figure 3In the illustrated embodiment, for an audio, since the audio contains multiple timbre information, after the audio is randomly segmented, different timbre information can be dispersed into different segmented audios. Therefore, the speaker encoder can extract more refined timbre features for each segmented audio respectively, so that the speaker encoder can more accurately capture the timbre information in the audio, and then output more precise timbre features, thereby reducing the situation where some timbre features are diluted or covered up due to the diversity of audio content. Therefore, in the process of voice conversion of customer service audio, more accurate timbre features can be extracted for different customer service audios with similar timbre, thereby improving the quality of the target customer service audio. In the training stage, the random segmentation process is introduced, so that the extracted target timbre features are more refined, so that the difference between the target timbre features and the standard timbre features can be more accurate, which is conducive to more precise second parameter adjustment of the speaker encoder.

[0130] See also Figure 4 , Figure 4 An embodiment of a sub-step of step 230 is shown. In one embodiment, for each training audio, when the current image transformation includes image compression, step 230 may include the following sub-steps.

[0131] Step 410: Obtain the current compression ratio coefficient;

[0132] Step 420: performing image compression on the mel-spectrogram corresponding to the training audio in the frequency domain dimension according to the compression ratio coefficient to obtain a first mel-spectrogram;

[0133] Step 430: Perform prediction and completion processing on the first mel-spectrogram to obtain a target mel-spectrogram of the same size as the mel-spectrogram.

[0134] In one embodiment, the compression ratio coefficient refers to the size ratio in one direction of the desired mel-spectrogram obtained by image compression of a mel-spectrogram to the original mel-spectrogram without image compression. For example, if the length of the desired mel-spectrogram is 80% of the length of the original mel-spectrogram without image compression in the length direction, the compression ratio coefficient is 80%. The compression ratio coefficient may be a preset ratio coefficient corresponding to the training round, for example, in the nth to 2n-1th training rounds, the preset ratio coefficient used is A, and in the 2nth to 3n-1th training rounds, the preset ratio coefficient used is B; the compression ratio coefficient may be a ratio coefficient obtained in the current training round based on the preset ratio coefficient corresponding to the training round, and so on, which is not specifically limited here.

[0135] In one embodiment, image compression of a mel-spectrogram in the frequency domain dimension refers to an operation of compressing data represented by a mel-spectrogram in the frequency domain and representing the compressed data as a first mel-spectrogram, wherein the first mel-spectrogram refers to a mel-spectrogram obtained after image compression of the mel-spectrogram in the frequency domain dimension, with the length unchanged and the height reduced. Generally speaking, the lateral direction of the mel-spectrogram is used to represent the time domain dimension, and the longitudinal direction represents the frequency domain dimension. When the mel-spectrogram is image compressed in the frequency domain dimension, the waveform displayed by the first mel-spectrogram is more compact and low compared to the waveform displayed by the original mel-spectrogram because the data represented in the frequency domain is affected by the compression process. In addition, there are various ways to compress the data represented by a mel-spectrogram in the frequency domain, which can be performed by discrete cosine transform (DCT) or wavelet transform, etc., which are not specifically limited here.

[0136] In one embodiment, the prediction completion process refers to an operation of supplementing the predicted data points in the frequency domain dimension based on the first mel-spectrogram by predicting data points to obtain a target mel-spectrogram of the same size as the mel-spectrogram. Specifically, based on the time domain dimension of the first mel-spectrogram, the data points corresponding to the frequencies whose frequency values ​​are higher than the frequency values ​​represented by the length of the first mel-spectrogram are predicted, and the predicted data points are compared with the first mel-spectrogram. Figure 1 The same is represented as a new mel-spectrogram (ie, the target mel-spectrogram). The prediction completion process can be performed by linear interpolation, or by generating a neural network to predict data points, and then interpolating the predicted data points, etc., which is not specifically limited here.

[0137] exist Figure 4In the illustrated embodiment, due to image compression, the size of the first mel-spectrogram after compression is different from the size of the mel-spectrogram required by the content encoder, so it is necessary to complete the first mel-spectrogram to a target mel-spectrogram of the same size as the mel-spectrogram required by the content encoder through prediction and completion processing. For a training audio, the semantics of the content expressed by it will not change with the image transformation, that is, in theory, regardless of whether the mel-spectrogram is subjected to image compression and prediction and completion processing, the target content features extracted by the content encoder should be the same as the corresponding standard content features. Therefore, training the content encoder through the target mel-spectrogram can effectively improve the training intensity of the content encoder extracting the content information that remains unchanged before and after image compression in the mel-spectrogram. Then, through the obvious difference between the target content features of the target mel-spectrogram and the corresponding standard content features, the content encoder is first adjusted, so that the content encoder is improved in its ability to extract the content information that remains unchanged before and after image compression in the mel-spectrogram, which helps to improve the quality of the audio content extracted by the conversion model, thereby improving the quality of the target customer service audio.

[0138] See also Figure 5 , Figure 5 Another embodiment of sub-steps of step 230 is shown. In one embodiment, for each training audio, when the current image transformation includes image stretching, step 230 may include the following sub-steps.

[0139] Step 510: Obtain the current stretching ratio coefficient;

[0140] Step 520: Perform image stretching on the mel-spectrogram corresponding to the training audio in the frequency domain according to the stretching ratio coefficient to obtain a second mel-spectrogram;

[0141] Step 530: Crop the second mel-spectrogram according to the size of the mel-spectrogram to obtain a target mel-spectrogram with the same size as the mel-spectrogram.

[0142] In one embodiment, the stretching ratio coefficient refers to the size ratio in one direction of the desired mel-spectrogram obtained by image stretching a mel-spectrogram to the original mel-spectrogram without image compression. For example, if the length of the desired mel-spectrogram is 120% of the length of the original mel-spectrogram without image stretching in the length direction, the compression ratio coefficient is 120%. The stretching ratio coefficient may be a preset ratio coefficient corresponding to the training round, for example, in the nth to 2n-1th training rounds, the preset ratio coefficient used is A, and in the 2nth to 3n-1th training rounds, the preset ratio coefficient used is B.

[0143] In one embodiment, image stretching of a mel-spectrogram in the frequency domain dimension refers to an operation of interpolating the data represented by a mel-spectrogram in the frequency domain, and representing the interpolated data together with the original data as a second mel-spectrogram, wherein the second mel-spectrogram refers to a mel-spectrogram with unchanged length and increased height obtained after the mel-spectrogram is image stretched in the frequency domain dimension. Generally speaking, the lateral direction of the mel-spectrogram is used to represent the time domain dimension, and the longitudinal direction represents the frequency domain dimension. When the mel-spectrogram is image stretched in the frequency domain dimension, the resolution of the data displayed by the second mel-spectrogram will increase due to the influence of the interpolation processing on the data represented in the frequency domain, and the waveform displayed by the second mel-spectrogram is more sparse and higher than the waveform displayed by the original mel-spectrogram. In addition, there are various ways to interpolate the data represented by a mel-spectrogram in the frequency domain, such as by linear interpolation, by generating a neural network to predict data points, and by interpolating the predicted data points, etc., which are not specifically limited here.

[0144] In one embodiment, the cropping process refers to an operation of obtaining a target mel-spectrogram of the same size as the mel-spectrogram by intercepting data points of a certain frequency span starting from the highest frequency and moving toward the lowest frequency on the basis of the second mel-spectrogram, and then deleting the intercepted data points.

[0145] exist Figure 5 In the illustrated embodiment, due to image stretching, the size of the stretched second mel-spectrogram is different from the size of the mel-spectrogram required by the content encoder, so it is necessary to crop the second mel-spectrogram to a target mel-spectrogram of the same size as the mel-spectrogram required by the content encoder. For a training audio, the semantics of the content expressed by it will not change with the image transformation, that is, in theory, regardless of whether the mel-spectrogram is subjected to image stretching and cropping, the target content features extracted by the content encoder should be the same as the corresponding standard content features. Therefore, training the content encoder through the target mel-spectrogram can effectively improve the training intensity of the content encoder extracting the content information that remains unchanged before and after the image stretching in the mel-spectrogram. Then, through the obvious difference between the target content features of the target mel-spectrogram and the corresponding standard content features, the content encoder is first adjusted, so that the content encoder is improved in its ability to extract the content information that remains unchanged before and after the image stretching in the mel-spectrogram, which helps to improve the quality of the audio content extracted by the conversion model, thereby improving the quality of the target customer service audio.

[0146] See also Figure 6 , Figure 6 Shows Figure 1Another training step of the conversion model used in the process. In one embodiment, the audio training set also includes a label audio corresponding to each training audio, and the target mel-spectrogram corresponding to each training audio includes a target mel-spectrogram for timbre feature extraction.

[0147] Step 410 may include the following sub-steps:

[0148] Step 610: determining a compression ratio coefficient according to the fusion coefficient and the current preset compression ratio coefficient, wherein the fusion coefficient of the training audio is obtained by extracting the timbre feature of a target mel-spectrogram corresponding to the previous training audio by the speaker encoder;

[0149] After obtaining the target content feature corresponding to each training audio, the method further includes:

[0150] Step 620: Perform audio conversion processing on multiple target content features respectively to obtain target audio corresponding to each training audio;

[0151] When the current initial model does not meet the preset training end condition, the method further includes the following steps:

[0152] Step 630: Perform a third parameter adjustment on the initial model according to the target audio and label audio corresponding to each training audio.

[0153] In one embodiment, when random segmentation processing is not involved on the mel-spectrogram, the target mel-spectrogram used for timbre feature extraction and the target mel-spectrogram used for content feature extraction are the same target mel-spectrogram; when random segmentation processing is involved on the mel-spectrogram, the target mel-spectrogram used for timbre feature extraction is the target mel-spectrogram corresponding to the segmented mel-spectrogram.

[0154] In one embodiment, the previous training audio refers to the previous training audio in the audio training set before the currently used training audio.

[0155] In one embodiment, the fusion coefficient refers to a coefficient used to represent the feature fusion process of the decoder for content features and timbre features. Specifically, after the speaker encoder extracts the timbre features, it will also output the fusion coefficient used for the next training audio. In the speaker encoder, the output fusion coefficient can be obtained according to a pre-constructed mapping function. Specifically, the initial model is a model with basic speech conversion capabilities. The function can be constructed according to the timbre features and audio output by the initial model to obtain the mapping function. Then, during the training process, the fusion coefficient can be obtained through the mapping function and the output target timbre features; the fusion coefficient can also be obtained through the target timbre features. Specifically, the target timbre features include frequency components corresponding to the timbre. The target timbre features can be reduced in dimension by pooling and other methods to obtain a single-dimensional value, which is the fusion coefficient; the fusion coefficient can also be a preset value, etc., which is not specifically limited here.

[0156] It should be noted that the fusion coefficient is only introduced in the process of training the initial model into a conversion model, and is not introduced in the actual process of voice conversion of customer service audio. That is to say, in the actual process, it is still necessary to fuse the content features and timbre features of the customer service audio through the decoder.

[0157] In one embodiment, audio conversion processing refers to the operation of converting the features of the input itself into simulated audio through an audio generation network. In the training process, the features of the input audio generation network are target content features; in the actual application process, the features of the input audio generation network are fusion features obtained by the decoder fusing the content features and timbre features of the customer service audio. It should be noted that during the training process, due to the use of fusion coefficients, the features input to the decoder are actually only the target content features.

[0158] In one embodiment, the third parameter adjustment refers to determining a third loss value based on the target audio and the label audio corresponding to each training audio, and adjusting the parameters of the initial model components (such as the decoder, audio generation network, etc. mentioned above) in the initial model except the content encoder and the speaker encoder according to the third loss value.

[0159] exist Figure 6 In the illustrated embodiment, a fusion coefficient is obtained through a speaker encoder, so that a preset compression ratio coefficient can be adjusted through the fusion coefficient to obtain the compression ratio coefficient, and then the feature fusion process of the subsequent decoder can be omitted through image compression, which helps to speed up the operation process of the initial model on the training audio, thereby improving the training efficiency of the initial model.

[0160] See also Figure 7 , Figure 7 Shows Figure 1Another training step of the transformation model used in the process. In one embodiment, when the image change correspondence includes image stretching, step 510 may include the following sub-steps:

[0161] Step 710: determining a stretching ratio coefficient according to the fusion coefficient and the current preset stretching ratio coefficient, wherein the fusion coefficient of the training audio is obtained by extracting the timbre feature of a target mel-spectrogram corresponding to the previous training audio by a speaker encoder;

[0162] exist Figure 7 In the illustrated embodiment, a fusion coefficient is obtained through a speaker encoder, so that a preset stretching ratio coefficient can be adjusted through the fusion coefficient to obtain the stretching ratio coefficient, and then the feature fusion process of the subsequent decoder can be omitted through image stretching, which helps to speed up the operation process of the initial model on the training audio, thereby improving the training efficiency of the initial model.

[0163] See also Figure 8 , Figure 8 Shows Figure 2 In one embodiment, step 250 may include the following sub-steps.

[0164] Step 810: performing a minimization loss analysis based on target content features and label content features corresponding to a plurality of training audios to determine a first loss value;

[0165] Step 820: Perform a first parameter adjustment on the content encoder according to the first loss value.

[0166] In one embodiment, the initial model can introduce a reward function, and then introduce a reinforcement learning mechanism into the initial model. Based on this, in the process of performing a first parameter adjustment on the content encoder for a first loss value, the initial model can specifically perform a hyperparameter simulation according to the first loss value, determine the optimal hyperparameters corresponding to the content encoder in the current round, and perform a first parameter adjustment on the content encoder according to the optimal hyperparameters.

[0167] In one embodiment, during the process of performing the first parameter adjustment on the content encoder for the first loss value, the hyperparameters input by the technician for the first loss value may be first obtained, and the first parameter adjustment is performed on the content encoder according to the input hyperparameters.

[0168] exist Figure 8In the illustrated embodiment, since the target content features extracted by the content encoder should be the same as the corresponding standard content features, the difference between the content information in the mel-spectrogram actually extracted and remaining unchanged after image compression and the content information in the mel-spectrogram theoretically remaining unchanged before and after image compression can be determined by using the first loss value corresponding to the minimized loss between the target content features and the label content features. This helps to perform a first parameter adjustment on the content encoder to enhance the content extraction capability, thereby improving the quality of the content information extracted from the customer service audio by the content encoder in the conversion model, and improving the robustness of the process of extracting content information by the content encoder in the conversion model.

[0169] See also Fig. 9 , Fig. 9 Shows Figure 3 In one embodiment, step 350 may include the following sub-steps.

[0170] Step 910: performing angular edge loss analysis according to a plurality of segmented timbre features corresponding to each training audio to determine a second loss value;

[0171] Step 920: performing a maximum loss analysis based on the target timbre features and the label timbre features corresponding to each training audio to determine a third loss value;

[0172] Step 930: Perform a second parameter adjustment on the speaker encoder with respect to the second loss value and the third loss value.

[0173] In one embodiment, performing angular edge loss analysis based on multiple segmented timbre features corresponding to each training audio refers to a process of performing angular edge loss analysis between the corresponding multiple segmented timbre features for each training audio to obtain a second loss value. Specifically, for an audio, since this audio is recorded by a customer service staff, after the audio is processed by random audio segmentation, the timbre corresponding to each segmented audio is theoretically the same, that is, the multiple segmented timbre features corresponding to the audio should be the same. Therefore, the difference between the timbre features when the speaker encoder performs timbre features on the audio of the same timbre can be determined by introducing the second loss value, which is beneficial to reduce the difference between the segmented timbre features corresponding to the segmented audios of the same timbre when performing the second parameter adjustment on the speaker encoder, thereby making the timbre features output by the speaker encoder more accurate.

[0174] In one embodiment, the initial model can introduce a reward function, and then introduce a reinforcement learning mechanism into the initial model. Based on this, in the process of performing a second parameter adjustment on the speaker encoder for the second loss value and the third loss value, the initial model can specifically perform a hyperparameter simulation according to the second loss value and the third loss value, determine the optimal hyperparameters corresponding to the speaker encoder in the current round, and perform a second parameter adjustment on the speaker encoder according to the optimal hyperparameters.

[0175] In one embodiment, during the second parameter adjustment of the speaker encoder for the second loss value and the third loss value, the hyperparameters input by the technician for the second loss value and the third loss value can be first obtained, and the second parameter adjustment of the speaker encoder can be performed based on the input hyperparameters.

[0176] exist Fig. 9 In the illustrated embodiment, after the Mel-spectrogram undergoes image transformation, the timbre it represents is actually different from the timbre in the training audio, that is, the target timbre feature extracted by the speaker encoder should be different from the corresponding standard timbre feature. Therefore, the speaker encoder can be adjusted by a third loss value corresponding to the maximum loss between the target content feature and the label content feature, so that the speaker encoder can improve the discrimination between the target timbre feature and the label timbre feature. In this way, even if there are customer service audios with similar timbre in the preset database, the timbre feature extracted by the speaker encoder can better show the timbre difference between the two similar timbres, so that the timbre feature extraction capability of the speaker encoder is enhanced. In addition, the introduction of the second loss value in the process of adjusting the parameters of the speaker encoder can reduce the difference between the segmented timbre features corresponding to each segmented audio of the same timbre, so that the timbre feature extraction capability of the speaker encoder can be further enhanced. Therefore, when new customer service audio is added to the preset database, regardless of whether the timbre of the recorded customer service audio is the same as the previous timbre, the speaker encoder can extract the timbre features more accurately, thereby improving the speech conversion capability of the customer service audio.

[0177] See also Fig.10 The embodiment of the present application further provides a voice conversion device, which can implement the above method. The voice conversion device 1000 includes:

[0178] A customer service audio acquisition module 1010 is used to acquire customer service audio to be processed;

[0179] The target customer service audio output module 1020 is used to perform speech conversion processing on the customer service audio through a pre-trained conversion model and output the target customer service audio.

[0180] In one embodiment, the pre-trained conversion model used by the target customer service audio output module 1020 can be trained by the following steps: first, an audio training set including multiple training audios and label content features corresponding to each training audio is obtained, then, the multiple training audios are respectively subjected to mel-spectrogram conversion processing to obtain a mel-spectrogram corresponding to each training audio, then, for each training audio, the corresponding mel-spectrogram is subjected to image transformation to obtain a target mel-spectrogram, the image transformation is image stretching or image compression, the training audios in the audio training set alternately select one of image stretching or image compression for image transformation, then the multiple target mel-spectrograms are respectively input into an initial model including a content encoder to obtain a target content feature corresponding to each training audio, then, when the current initial model does not meet the preset training end condition, the content encoder is first parameterized according to the target content features and label content features corresponding to the multiple training audios, and the initial model after parameter adjustment is continuously trained based on the audio training set until the preset training end condition is met to obtain a conversion model.

[0181] The specific implementation of the voice conversion device is basically the same as the specific implementation of the above-mentioned voice conversion method, and will not be repeated here.

[0182] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned voice conversion method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0183] See also Fig.11 , Fig.11 The hardware structure of an electronic device of another embodiment is illustrated. The electronic device 1100 includes:

[0184] The processor 1101 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0185] The memory 1102 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1102 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1102, and the processor 1101 calls and executes the speech conversion method of the embodiment of this application;

[0186] Input / output interface 1103, used to implement information input and output;

[0187] The communication interface 1104 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WI FI, Bluetooth, etc.);

[0188] A bus 1105 that transmits information between various components of the device (e.g., the processor 1101, the memory 1102, the input / output interface 1103, and the communication interface 1104);

[0189] The processor 1101 , the memory 1102 , the input / output interface 1103 and the communication interface 1104 are connected to each other in communication within the device via the bus 1105 .

[0190] The embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned voice conversion method is implemented.

[0191] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0192] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0193] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0194] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0195] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0196] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0197] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0198] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0199] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0200] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0201] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0202] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A speech conversion method, characterized in that: The method comprises: Get the customer service audio to be processed; Performing speech conversion processing on the customer service audio through a pre-trained conversion model, and outputting target customer service audio; The conversion model is trained by the following steps: Acquire an audio training set, wherein the audio training set includes a plurality of training audios and a label content feature corresponding to each of the training audios; Performing mel-spectrogram conversion processing on the plurality of training audios respectively to obtain a mel-spectrogram corresponding to each of the training audios; For each of the training audios, performing image transformation on the mel-spectrogram corresponding to the training audio to obtain a target mel-spectrogram, wherein the image transformation is image stretching or image compression, and the training audios in the audio training set alternately select one of image stretching and image compression for image transformation; Inputting the plurality of target mel-spectrograms into an initial model respectively to obtain target content features corresponding to each of the training audios, wherein the initial model includes a content encoder; When the current initial model does not meet the preset training end condition, first adjusting the parameters of the content encoder according to the target content features and label content features corresponding to the plurality of training audios; The initial model after parameter adjustment is continuously trained based on the audio training set until the preset training end condition is met to obtain the conversion model.

2. The method according to claim 1, characterized in that The mel-spectrogram corresponding to each of the training audios includes a plurality of segmented mel-spectrograms; the audio training set also includes a label timbre feature corresponding to each of the training audios; the initial model also includes a speaker encoder, and the speaker encoder is parallel to the content encoder; The step of performing mel-spectrogram conversion on the audio training set to obtain a plurality of mel-spectrograms includes: Performing random audio segmentation processing on each of the training audios to obtain a plurality of segmented audios corresponding to each of the training audios; Performing mel-spectrogram conversion processing on the plurality of segmented audios respectively to obtain the segmented mel-spectrogram corresponding to each segmented audio; After inputting the plurality of target mel-spectrograms into the initial model respectively, the method further comprises: Using the speaker encoder, respectively extract the timbre features of the target mel-spectrogram corresponding to each of the segmented mel-spectrograms to obtain the segmented timbre features corresponding to each of the segmented mel-spectrograms; For each of the training audios, according to a preset combination ratio coefficient, a feature combination process is performed on a plurality of the segmented timbre features corresponding to the training audio to obtain a target timbre feature; When the current initial model does not meet the preset training end condition, the method further includes: The speaker encoder is subjected to a second parameter adjustment according to the target timbre feature and the label timbre feature corresponding to each of the training audios.

3. The method according to claim 1, characterized in that In the case where the current image transformation includes the image compression, performing image transformation on the corresponding Mel-spectrogram to obtain a target Mel-spectrogram includes: Get the current compression ratio; According to the compression ratio coefficient, image compression is performed on the mel-spectrogram corresponding to the training audio in the frequency domain dimension to obtain a first mel-spectrogram; The first mel-spectrogram is predicted and completed to obtain the target mel-spectrogram having the same size as the mel-spectrogram.

4. The method according to claim 1, characterized in that: When the image transformation includes the image stretching, performing image transformation on the corresponding Mel-spectrogram to obtain a target Mel-spectrogram includes: Get the current stretch ratio; According to the stretching ratio coefficient, the mel-spectrogram corresponding to the training audio is stretched in the frequency domain dimension to obtain a second mel-spectrogram; The second mel-spectrogram is cropped according to the size of the mel-spectrogram to obtain the target mel-spectrogram with the same size as the mel-spectrogram.

5. The method according to claim 3, characterized in that: The audio training set also includes a label audio corresponding to each of the training audios; the initial model also includes a speaker encoder; the target Mel spectrogram corresponding to each of the training audios includes the target Mel spectrogram for timbre feature extraction; The obtaining of the current compression ratio coefficient comprises: Determining the compression ratio coefficient according to the fusion coefficient and the current preset compression ratio coefficient, wherein the fusion coefficient of the training audio is obtained by extracting the timbre feature of the target mel-spectrogram for timbre feature extraction corresponding to the previous training audio by the speaker encoder; After obtaining the target content feature corresponding to each training audio, the method further includes: Performing audio conversion processing on the plurality of target content features respectively to obtain the target audio corresponding to each of the training audios; When the current initial model does not meet the preset training end condition, the method further includes: The initial model is subjected to a third parameter adjustment according to the target audio and the label audio corresponding to each of the training audios.

6. The method according to claim 1, characterized in that The first parameter adjustment of the content encoder according to the target content features and the label content features corresponding to the plurality of training audios includes: Perform a minimization loss analysis based on target content features and label content features corresponding to the plurality of training audios to determine a first loss value; Perform first parameter adjustment on the content encoder according to the first loss value.

7. The method according to claim 2, characterized in that The second parameter adjustment of the speaker encoder according to the target timbre feature and the label timbre feature corresponding to each of the training audios includes: Performing angular edge loss analysis according to the plurality of segmented timbre features corresponding to each of the training audios to determine a second loss value; Performing a maximization loss analysis based on the target timbre feature and the label timbre feature corresponding to each of the training audios to determine a third loss value; A second parameter adjustment is performed on the speaker encoder with respect to the second loss value and the third loss value.

8. A speech conversion device, characterized in that: The device comprises: A customer service audio acquisition module is used to acquire customer service audio to be processed; A target customer service audio output module is used to perform speech conversion processing on the customer service audio through a pre-trained conversion model and output the target customer service audio; The conversion model is trained by the following steps: Acquire an audio training set, wherein the audio training set includes a plurality of training audios and a label content feature corresponding to each of the training audios; Performing mel-spectrogram conversion processing on the plurality of training audios respectively to obtain a mel-spectrogram corresponding to each of the training audios; For each of the training audios, performing image transformation on the mel-spectrogram corresponding to the training audio to obtain a target mel-spectrogram, wherein the image transformation is image stretching or image compression, and the training audios in the audio training set alternately select one of image stretching or image compression for image transformation; Inputting the plurality of target mel-spectrograms into an initial model respectively to obtain target content features corresponding to each of the training audios, wherein the initial model includes a content encoder; When the current initial model does not meet the preset training end condition, first adjusting the parameters of the content encoder according to the target content features and label content features corresponding to the plurality of training audios; The initial model after parameter adjustment is continuously trained based on the audio training set until the preset training end condition is met to obtain the conversion model.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the speech conversion method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the speech conversion method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Voice conversion model training method, voice conversion method, device and medium

    CN115171666A

  • Tone conversion method and device

    CN116312581A

  • Method and system for generating audio by text

    CN117854475A

  • Digital audio signal reproducing device, method and the media for general musics or musics for karaoke

    KR1019980023208A

  • Hybrid language models for conversational ai systems and applications

    US20240135920A1