Voice conversion method and device, electronic equipment and storage medium

Abstract

The application provides a voice conversion method and device, electronic equipment and storage medium, belonging to the field of financial technology, the method comprising: obtaining a to-be-processed customer service audio, and obtaining a target customer service audio through voice conversion of a pre-trained conversion model; the conversion model training steps are as follows: obtaining an audio training set comprising a plurality of training audios and their label content features; obtaining a target mel spectrum of each training audio based on mel spectrum conversion and image transformation, and alternately selecting one of image stretching or image compression for image transformation of the training audios in the audio training set; obtaining a target content feature of each training audio through an initial model; when a preset training end condition is not met, first adjusting the content encoder of the initial model according to the plurality of target content features and their label content features; continuing to train the adjusted initial model based on the audio training set until the conversion model is obtained. The application can improve the stability of voice conversion.

Description

Technical Field

[0008] , , , ,

[0009]

[0001] This application relates to the field of fintech, and in particular, to a voice conversion method, apparatus, electronic device, and storage medium. Background Art

[0002] With the rapid development of artificial intelligence technology, the traditional financial industry is gradually transforming into fintech, and various financial service systems are constantly being upgraded and updated through artificial intelligence technology. There is a customer service link in the financial industry. Due to the special nature of the financial industry, telephone customer service services need to be provided for a large number of users. In the past, telephone customer service services required manual communication with customers. After the telephone customer service service is upgraded through artificial intelligence technology, relevant consultation information can be automatically answered based on the communication information with customers.

[0003] Existing intelligent telephone customer service services (also known as intelligent seats) generally store different customer service audio recordings made by a certain customer service staff according to different business scenarios in a preset database in advance. Then, during the actual customer service process, the content of what the customer service says is recognized through a speech recognition model to obtain the recognized content. Then, the corresponding customer service audio is extracted from the database according to the content of the speech recognition, and then the customer service audio is converted through a voice conversion model to generate the audio for replying to the customer.

[0004] With the development of financial services, it may be necessary for different customer service staff to record more complex customer service audio to update the preset database. However, the content extraction ability of the voice conversion model is affected by the voice quality. When the voice conversion model converts the newly added customer service audio in the preset database, the quality of the extracted content is not stable, and it is impossible to output high-quality customer service audio for replying to customers, thus affecting the customer service experience. Summary of the Invention

[0005] The main purpose of the embodiments of this application is to propose a voice conversion method, apparatus, electronic device, and storage medium, aiming to improve the content extraction ability of customer service audio, so as to reduce the impact of voice quality on the content extraction ability in the case of newly added customer service audio with different voice qualities.

[0006] To achieve the above object, in the first aspect of the embodiments of this application, a voice conversion method is proposed, and the method includes:

[0007] Obtain the customer service audio to be processed;

[0008] Perform voice conversion processing on the customer service audio through a pre-trained conversion model, and output the target customer service audio;

[0009] Among them, the conversion model is trained through the following steps:

[0010] Obtain an audio training set, which includes multiple training audios and label content features corresponding to each training audio;

[0011] The Mel spectrogram transformation process is performed on the multiple training audios respectively to obtain the Mel spectrogram corresponding to each training audio;

[0012] For each training audio, an image transformation is performed on the Mel spectrogram corresponding to the training audio to obtain a target Mel spectrogram. The image transformation is either image stretching or image compression. The training audios in the audio training set are alternately transformed using either image stretching or image compression.

[0013] The multiple target Mel spectrograms are respectively input into the initial model to obtain the target content features corresponding to each training audio. The initial model includes a content encoder.

[0014] If the current initial model does not meet the preset training termination condition, the content encoder is subjected to a first parameter tuning based on the target content features and tag content features corresponding to multiple training audios;

[0015] The initial model after parameter tuning is trained on the audio training set until the preset training termination condition is met, and the transformed model is obtained.

[0016] In some embodiments, each training audio corresponds to a Mel spectrogram comprising multiple segmented Mel spectrograms; the audio training set also includes tagged timbre features corresponding to each training audio; the initial model further includes a speaker encoder, which runs in parallel with the content encoder;

[0017] The Mel spectrogram transformation process performed on the audio training set yields multiple Mel spectrograms, including:

[0018] Each training audio is subjected to random audio segmentation to obtain multiple segmented audios corresponding to each training audio.

[0019] The Mel spectrogram conversion process is performed on the multiple segmented audios to obtain the segmented Mel spectrogram corresponding to each segmented audio;

[0020] After inputting the multiple target Mel spectra into the initial model, the method further includes:

[0021] The speaker encoder extracts timbre features from the target Mel spectrogram corresponding to each segmented Mel spectrogram to obtain the segmented timbre features corresponding to each segmented Mel spectrogram.

[0022] For each training audio, according to a preset combination ratio coefficient, the multiple segmented timbre features corresponding to the training audio are subjected to feature combination processing to obtain the target timbre features;

[0023] If the current initial model does not meet the preset training termination condition, the method further includes:

[0024] The speaker encoder is subjected to a second parameter tuning based on the target timbre feature and the tag timbre feature corresponding to each training audio.

[0025] In some embodiments, where the current image transformation includes image compression, the step of performing image transformation on the corresponding Mel spectrogram to obtain the target Mel spectrogram includes:

[0026] Get the current compression ratio;

[0027] Based on the compression ratio coefficient, the Mel spectrogram corresponding to the training audio is compressed in the frequency domain to obtain a first Mel spectrogram;

[0028] The first Mel spectrum is subjected to prediction completion processing to obtain the target Mel spectrum with the same size as the first Mel spectrum.

[0029] In some embodiments, when the image transformation includes the image stretching, the step of performing image transformation on the corresponding Mel spectrogram to obtain the target Mel spectrogram includes:

[0030] Get the current stretch ratio coefficient;

[0031] Based on the stretching ratio coefficient, the Mel spectrogram corresponding to the training audio is stretched in the frequency domain dimension to obtain a second Mel spectrogram;

[0032] Based on the size of the Mel spectrum, the second Mel spectrum is cropped to obtain the target Mel spectrum with the same size as the first Mel spectrum.

[0033] In some embodiments, the audio training set further includes labeled audio corresponding to each training audio; the initial model further includes a speaker encoder; and the target Mel spectrogram corresponding to each training audio includes the target Mel spectrogram used for timbre feature extraction.

[0034] The process of obtaining the current compression ratio coefficient includes:

[0035] The compression ratio coefficient is determined based on the fusion coefficient and the current preset compression ratio coefficient. The fusion coefficient of the training audio is obtained by the speaker encoder extracting timbre features from the target Mel spectrogram corresponding to the previous training audio for timbre feature extraction.

[0036] After obtaining the target content features corresponding to each of the training audios, the method further includes:

[0037] The target content features are each subjected to audio conversion processing to obtain the target audio corresponding to each training audio.

[0038] If the current initial model does not meet the preset training termination condition, the method further includes:

[0039] The initial model is then subjected to a third parameter tuning based on the target audio and the label audio corresponding to each of the training audios.

[0040] In some embodiments, the first parameter tuning of the content encoder based on the target content features and tag content features corresponding to the plurality of training audios includes:

[0041] Based on the target content features and tag content features corresponding to multiple training audios, a minimization loss analysis is performed to determine the first loss value;

[0042] The content encoder is subjected to a first parameter tuning for the first loss value.

[0043] In some embodiments, the step of performing a second parameter tuning on the speaker encoder based on the target timbre feature and the labeled timbre feature corresponding to each of the training audios includes:

[0044] Based on the multiple segmentation timbre features corresponding to each training audio, angular edge loss analysis is performed to determine the second loss value;

[0045] A third loss value is determined by performing a maximum loss analysis based on the target timbre features and the tag timbre features corresponding to each training audio;

[0046] The speaker encoder is subjected to a second parameter tuning for the second loss value and the third loss value.

[0047] To achieve the above objectives, a second aspect of this application provides a speech conversion device, the device comprising:

[0048] The customer service audio acquisition module is used to acquire customer service audio files to be processed.

[0049] The target customer service audio output module is used to perform speech conversion processing on the customer service audio using a pre-trained conversion model and output the target customer service audio.

[0050] The conversion model is trained through the following steps:

[0051] Obtain an audio training set, which includes multiple training audios and label content features corresponding to each training audio;

[0052] The Mel spectrogram transformation process is performed on the multiple training audios respectively to obtain the Mel spectrogram corresponding to each training audio;

[0053] For each training audio, an image transformation is performed on the Mel spectrogram corresponding to the training audio to obtain a target Mel spectrogram. The image transformation is either image stretching or image compression. The training audios in the audio training set are alternately selected to undergo image transformation, either image stretching or image compression.

[0054] The multiple target Mel spectrograms are respectively input into the initial model to obtain the target content features corresponding to each training audio. The initial model includes a content encoder.

[0055] If the current initial model does not meet the preset training termination condition for being determined as the conversion model, the content encoder is subjected to a first parameter tuning based on the target content features and tag content features corresponding to multiple training audios;

[0056] The initial model, after parameter tuning, is trained again based on the audio training set until the current initial model is determined to be the transformed model.

[0057] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0058] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0059] The speech conversion method, apparatus, electronic device, and storage medium proposed in this application acquire customer service audio to be processed; perform speech conversion processing on the customer service audio using a pre-trained conversion model to output target customer service audio. The conversion model is trained through the following steps: acquiring an audio training set, which includes multiple training audios and corresponding label content features for each training audio; performing Mel spectrogram conversion processing on the multiple training audios to obtain a Mel spectrogram corresponding to each training audio; performing image transformation on the Mel spectrogram corresponding to each training audio to obtain a target Mel spectrogram, where the image transformation is either image stretching or image compression, and the training audios in the audio training set are alternately subjected to either image stretching or image compression for image transformation; inputting the multiple target Mel spectrograms into an initial model to obtain target content features corresponding to each training audio, where the initial model includes a content encoder; if the current initial model does not meet the preset training termination condition, performing a first parameter tuning on the content encoder based on the target content features and label content features corresponding to the multiple training audios; and continuing to train the parameter-tuned initial model based on the audio training set until the preset training termination condition is met to obtain the conversion model. By performing image transformation on the Mel spectrograms corresponding to the training audio, the original timbre characteristics of the training audio are altered, thus introducing timbre variation factors into the training process. Therefore, by performing the first parameter tuning on the label content features using the target content features corresponding to multiple training audios, it helps improve the content encoder's ability to extract content information that remains unchanged before and after the timbre variation features in the Mel spectrogram, thereby reducing the impact of timbre on the content extraction capability of the conversion model. Furthermore, since the training audios in the audio training set alternately undergo image transformation using either image stretching or image compression, timbre feature variations of high-frequency and low-frequency timbres can be introduced in turn during training. This transforms the first parameter tuning into a parameter tuning that enhances the content extraction capability for diverse timbres. As a result, the enhanced content extraction capability of the content encoder is more robust to customer service audio with diverse timbres, reducing the impact of timbre on content extraction capability. Thus, when new customer service audio with different timbres is added, the conversion model can also extract content from the new customer service audio more accurately and stably, thereby reducing the impact of the new customer service audio on the customer service experience. Attached Figure Description

[0060] Figure 1 This is a flowchart of a speech conversion method provided in an embodiment of this application;

[0061] Figure 2 yes Figure 1 A schematic diagram illustrating an embodiment of a training step of the transformation model used in the illustrated process;

[0062] Figure 3 yes Figure 1 A schematic diagram of another training step embodiment of the transformation model used in the illustrated process;

[0063] Figure 4 yes Figure 2 A schematic diagram of an embodiment of a sub-step of step 230 in the process;

[0064] Figure 5 yes Figure 2 A schematic diagram of another sub-step embodiment of step 230 in the diagram;

[0065] Figure 6 yes Figure 1 A schematic diagram of another training step embodiment of the transformation model used in the illustrated process;

[0066] Figure 7 yes Figure 1 A schematic diagram of another training step embodiment of the transformation model used in the illustrated process;

[0067] Figure 8 yes Figure 2 A schematic diagram of an embodiment of a sub-step of step 250 in the diagram;

[0068] Figure 9 yes Figure 3 A schematic diagram of an embodiment of a sub-step of step 350 in the process;

[0069] Figure 10 This is a schematic diagram of the structure of the speech conversion device provided in the embodiments of this application;

[0070] Figure 11 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0072] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0073] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs. The terms used herein are for the purpose of describing embodiments of this application only and are not intended to limit this application.

[0074] First, some nouns involved in this application are analyzed:

[0075] Artificial intelligence (AI): It is a new technical science that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence; artificial intelligence is a branch of computer science. Artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a way similar to human intelligence. The research in this field includes robots, speech recognition, image recognition, natural language processing, and expert systems, etc. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence also refers to the theory, method, technology, and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results.

[0076] With the rapid development of artificial intelligence technology, the traditional financial industry has gradually transformed into fintech, and various financial service systems are constantly being upgraded and updated through artificial intelligence technology. And there is a customer service link in the financial industry. Due to the special nature of the financial industry, it is necessary to provide telephone customer service for a large number of users. In the past, the telephone customer service service required manual communication with customers. After the telephone customer service service is upgraded by artificial intelligence technology, it can automatically answer relevant consultation information based on the communication information with customers.

[0077] Existing intelligent telephone customer service services (also known as intelligent seats) generally store different customer service audio recordings made by a certain customer service staff according to different business scenarios in a preset database in advance. Then, during the actual customer service process, the speech recognition model is used to perform speech recognition on the content spoken by the customer service, and the recognized content is obtained. Then, the corresponding customer service audio is extracted from the database according to the speech recognition content, and then the speech conversion model is used to perform speech conversion on this customer service audio, so as to generate the audio for replying to customers.

[0078] With the development of financial business, it may be necessary for different customer service staff to record more complex customer service audio to update the preset database. However, the content extraction ability of the speech conversion model is affected by the voice quality. When the speech conversion model performs speech conversion on this newly added customer service audio in the preset database, the quality of the extracted content is not stable, and it is impossible to output high-quality customer service audio for replying to customers, thus affecting the customer service experience.

[0079] Based on this, embodiments of this application provide a voice conversion method, apparatus, electronic device, and storage medium, aiming to improve the content extraction capability of customer service audio, thereby reducing the impact of timbre on content extraction capability when adding customer service audio with different timbres. Specifically, the method involves acquiring customer service audio to be processed; performing speech conversion processing on the customer service audio using a pre-trained conversion model to output target customer service audio. The conversion model is trained through the following steps: acquiring an audio training set, which includes multiple training audios and corresponding label content features for each training audio; performing Mel spectrogram conversion processing on each of the multiple training audios to obtain a Mel spectrogram corresponding to each training audio; performing image transformation on the Mel spectrogram corresponding to each training audio to obtain a target Mel spectrogram, where the image transformation is either image stretching or image compression, and the training audios in the audio training set are alternately subjected to either image stretching or image compression for image transformation; inputting multiple target Mel spectrograms into an initial model to obtain target content features corresponding to each training audio, where the initial model includes a content encoder; if the current initial model does not meet the preset training termination condition for being determined as a conversion model, performing the first parameter tuning on the content encoder based on the target content features and label content features corresponding to the multiple training audios; and continuing to train the parameter-tuned initial model based on the audio training set until the preset training termination condition is met to obtain the conversion model. By performing image transformation on the Mel spectrograms corresponding to the training audio, the original timbre characteristics of the training audio are altered, thus introducing timbre variation factors into the training process. Therefore, by performing the first parameter tuning on the label content features using the target content features corresponding to multiple training audios, it helps improve the content encoder's ability to extract content information that remains unchanged before and after the timbre variation features in the Mel spectrogram, thereby reducing the impact of timbre on the content extraction capability of the conversion model. Furthermore, since the training audios in the audio training set alternately undergo image transformation using either image stretching or image compression, timbre feature variations of high-frequency and low-frequency timbres can be introduced in turn during training. This transforms the first parameter tuning into a parameter tuning that enhances the content extraction capability for diverse timbres. As a result, the enhanced content extraction capability of the content encoder is more robust to customer service audio with diverse timbres, reducing the impact of timbre on content extraction capability. Thus, when new customer service audio with different timbres is added, the conversion model can also extract content from the new customer service audio more accurately and stably, thereby reducing the impact of the new customer service audio on the customer service experience. Furthermore, because the content extraction capability of the conversion model is enhanced, the probability of retraining the conversion model can be effectively reduced when adding customer service audio with different timbres, thereby reducing the maintenance cost of the smart agent.

[0080] The speech conversion method, apparatus, electronic device, and storage medium provided in this application are specifically described through the following embodiments. First, the speech conversion method in the embodiments of this application is described.

[0081] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0082] Foundational artificial intelligence technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0083] The speech conversion method provided in this application relates to the field of financial technology. The speech conversion method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the speech conversion method, but is not limited to the above forms.

[0084] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0085] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0086] Reference Figure 1 As shown, Figure 1 This is a flowchart of a speech conversion method provided in an embodiment of this application. Figure 1 The method may include, but is not limited to, steps 110 to 120.

[0087] Step 110: Obtain the customer service audio to be processed;

[0088] Step 120: Perform speech conversion processing on the customer service audio using a pre-trained conversion model to output the target customer service audio.

[0089] In one embodiment, customer service audio refers to template audio used to provide voice responses based on the information provided by the customer. The customer service audio can be stored in a preset audio database. The timbre of multiple customer service audio recordings stored in the preset audio database can be the same or different; no specific limitation is made here.

[0090] In one embodiment, the process of acquiring customer service audio to be processed can specifically involve acquiring the user's interactive voice, performing speech recognition on the interactive voice to obtain speech recognition information, and then calling the customer service audio corresponding to the speech recognition information from a preset audio database to obtain the customer service audio to be processed.

[0091] In one embodiment, speech conversion processing refers to the operation of changing the timbre of the customer service audio while preserving the content represented by the audio to be processed, and then outputting the target customer service audio. The target customer service audio refers to the audio that expresses the content represented by the customer service audio through the target timbre. Furthermore, the target timbre can be called from a preset timbre library, which is not limited here.

[0092] In one embodiment, the pre-trained conversion model may include a content encoder, a speaker encoder, a decoder, and an audio generation network. The content encoder and speaker encoder run in parallel and are connected in parallel to the decoder, which is in turn connected to the content encoder. During the speech conversion process, the customer service audio may first undergo Mel spectrogram conversion to obtain the corresponding Mel spectrogram. Then, the Mel spectrogram can be input into the pre-trained conversion model. The content encoder in the model can extract content features from the Mel spectrogram to obtain content features. The speaker encoder in the model can extract timbre features from the Mel spectrogram to obtain timbre features. The decoder in the model can fuse the content features and timbre features to obtain the target audio features. Finally, the audio generation network in the model can perform audio conversion based on the target audio features, thereby converting the target audio features into the target customer service audio. The pre-trained conversion model may be an autoencoder-based speech conversion (AutoVC) model, a VoiceCycleGAN model, etc., and is not specifically limited here. In addition, the audio generation network can be a Generative Adversarial Network (GAN), a Convolutional Neural Network (CNN), or a Recurrent Neural Network (RNN), etc., without being limited to any specific type here.

[0093] Reference Figure 2 As shown, Figure 2 It shows Figure 1The training steps of the conversion model used in the illustrated process, in one embodiment, the conversion model can be trained by the following steps to obtain an initial model, wherein the initial model includes a content encoder, a speaker encoder, a decoder, and an audio generation network.

[0094] Step 210: Obtain the audio training set, which includes multiple training audios and the label content features corresponding to each training audio.

[0095] Step 220: Perform Mel spectrogram transformation on multiple training audios to obtain the Mel spectrogram corresponding to each training audio;

[0096] Step 230: For each training audio, perform image transformation on the corresponding Mel spectrogram to obtain the target Mel spectrogram. The image transformation is either image stretching or image compression. The training audios in the audio training set are alternately transformed by either image stretching or image compression.

[0097] Step 240: Input multiple target Mel spectrograms into the initial model to obtain the target content features corresponding to each training audio;

[0098] Step 250: If the current initial model does not meet the preset training termination condition, perform the first parameter tuning of the content encoder based on the target content features and label content features corresponding to multiple training audios;

[0099] Step 260: Continue training the initial model after parameter tuning based on the audio training set until the preset training termination condition is met, and obtain the transformed model.

[0100] In one embodiment, the training audio refers to customer service audio used to train the initial model. The training audio can be customer service audio from a pre-defined audio database. Furthermore, the multiple training audios can be training audios with the same timbre but different content, training audios with different timbres but the same content, or training audios with different timbres and different content, etc., and the specific method is not limited here.

[0101] In one embodiment, the labeled content features refer to the content features obtained by extracting features from the Mel spectrogram after the training audio is converted into a Mel spectrogram without performing image transformation on the Mel spectrogram. These labeled content features can be those obtained before training the initial model, or they can be updated by the content encoder based on multiple training audios and parameter tuning from the previous training round; the specific method is not limited here.

[0102] In one embodiment, when the initial model is not determined to be the conversion model, after a training round, the label content features corresponding to each training audio in the audio training set can be updated: First, Mel spectrogram conversion is performed on multiple training audios to obtain the Mel spectrogram corresponding to each training audio. Then, based on the parameter-tuned content encoder, content features are extracted from the Mel spectrogram corresponding to each training audio to obtain the updated label content features corresponding to each training audio. Since the content encoder is parameter-tuned in each training round (i.e., the first parameter tuning below), updating the label content features using the parameter-tuned content encoder from the previous training round allows the loss value corresponding to the content feature extraction process in the current training round to be more accurately reflected. This makes the parameter tuning of the content encoder in the current training round more accurate, thereby enhancing the content extraction capability of the conversion model.

[0103] In one embodiment, image transformation refers to the operation of processing the Mel spectrogram during the model training phase to obtain a target Mel spectrogram. The target Mel spectrogram is the Mel spectrogram after image transformation. In addition to alternating between image stretching and image compression for the training audio, image transformation may also include specific image processing operations such as sharpening, smoothing, and segmentation; these are not specifically limited here. It is important to note that image transformation is only used to assist in model parameter tuning during the model training phase. Therefore, in the actual process of speech-to-speech processing of customer service audio, image transformation is not performed on the Mel spectrogram corresponding to the customer service audio.

[0104] In one embodiment, image stretching refers to stretching the size of the Mel spectrogram in a certain direction by changing the distribution of pixels or pixel blocks in the Mel spectrogram; image stretching also refers to shrinking the size of the Mel spectrogram in a certain direction by changing the distribution of pixels or pixel blocks in the Mel spectrogram.

[0105] In one embodiment, alternating between image stretching and image compression for image transformation in training audio means that a training audio selects an operation different from the one selected by the previous training audio in image stretching and image compression. For example, suppose an audio training set includes three training audios A, B, and C in sequence. Training audio A selects image stretching, and the image transformation performed on the Mel spectrogram corresponding to training audio A includes image stretching. Then, training audio B will alternately select image compression, and the image transformation performed on the Mel spectrogram corresponding to training audio B will alternately include image compression but not image stretching. Similarly, relative to training audio B, training audio B will alternately select image stretching, and the image transformation performed on the Mel spectrogram corresponding to training audio C will alternately include image stretching but not image compression.

[0106] In one embodiment, at the beginning of each training round, a random selection is made from image stretching and image compression to determine one of the image stretching and image compression operations performed on the Mel spectrogram corresponding to the first training audio in the audio training set.

[0107] In one embodiment, when the current training round is not the first training round, based on the choice between image stretching and image compression made by the first training audio in the audio training set in the previous training round, it is determined that the first training audio in the audio training set will alternately choose between image stretching and image compression in this training round. For example, assuming that the image transformation of the Mel spectrogram corresponding to the first training audio in the audio training set in the previous training round included image compression, then the first training audio in the audio training set will alternately choose image stretching in this training round. By alternately choosing between image stretching and image compression for the first training audio relative to the previous training round in each training round, the Mel spectrogram corresponding to each training audio can undergo both image stretching and image compression throughout the model training process. This allows the content encoder to perform content feature extraction on different Mel spectrograms corresponding to each training audio at least once, thereby reducing the bias caused by a single image transformation method. This improves the content extraction capability of the content encoder and thus enhances the quality of the target customer audio.

[0108] In one embodiment, since the initial model includes a content encoder, a speaker encoder, a decoder, and an audio generation network, for a target Mel spectrogram, when the target Mel spectrogram is input to the output model, in addition to outputting the target content features corresponding to the training audio through the content encoder, the target timbre features corresponding to the training audio through the speaker encoder, the target fusion features corresponding to the training audio through the decoder, and the target audio corresponding to the training audio through the audio generation network.

[0109] In one embodiment, the preset training termination condition may include the accuracy of content feature extraction by the content encoder, or it may include the accuracy of the output target audio and the accuracy of timbre feature extraction by the speaker encoder, in addition to the accuracy of content feature extraction by the content encoder. The specific details are not limited here. Specifically, after the current initial model completes the conversion of multiple training audios in the audio training set, the current training round ends. At this point, it is necessary to determine whether the current initial model meets the preset training termination condition based on the data output by the initial model (such as target content features). If it does not meet the condition, hyperparameter adjustments need to be made to the current initial model; if it does meet the condition, the current initial model is determined as the conversion model.

[0110] In one embodiment, the first parameter tuning refers to the operation of adjusting the parameters of the content encoder based on hyperparameters determined by the target content features and tag content features corresponding to multiple training audios. The hyperparameters determined by the target content features and tag content features corresponding to multiple training audios can be determined by a preset parameter tuning function, or they can be hyperparameters input by a technician based on the target content features and tag content features corresponding to multiple training audios, etc., and are not specifically limited here.

[0111] In one embodiment, during the training of the initial model after parameter tuning based on the audio training set, multiple additional training audio files and corresponding label content features for each additional training audio file can be acquired. Then, the audio training set is updated based on these additional training audio files and their corresponding label content features. The initial model after parameter tuning is then trained again based on the updated audio training set until the current initial model is determined as the conversion model. By updating the audio training set, a scenario of adding customer service audio to a preset database can be simulated. This allows for parameter tuning of the content encoder for scenarios involving adding customer service audio to a preset database, thereby improving the robustness of content feature extraction.

[0112] In one embodiment, the audio training set may further include the timbre features corresponding to each training audio. After performing a first parameter tuning on the content encoder based on the target content features and timbre features corresponding to multiple training audios, a second parameter tuning on the speaker encoder may be performed based on the target timbre features and timbre features corresponding to multiple training audios. Since the target Mel spectrogram is affected by image transformation, the spectrum displayed by the target Mel spectrogram also changes. Because timbre is frequency-dependent, when the spectrum displayed by the target Mel spectrogram changes, the timbre represented by the target Mel spectrogram will also change. Theoretically, the timbre represented by the target Mel spectrogram is different from the timbre of the original training audio. Based on this, by performing a second parameter tuning on the speaker encoder, the target timbre features output by the speaker encoder after the second parameter tuning can be made more distant from the label timbre features, thereby enhancing the speaker encoder's ability to distinguish timbre differences. This enables the speaker encoder to better distinguish the timbre of different customer service audio in actual customer service operations, which is beneficial to improving the robustness and generalization ability of the speaker encoder. Therefore, when different customer service personnel record customer service audio and store it in a preset database, the speaker encoder can better extract the timbre features of the customer service audio recorded by different customer service personnel, thereby enabling the conversion model to perform speech conversion of customer service audio more accurately.

[0113] In one embodiment, the second parameter tuning refers to the operation of adjusting the parameters of the speaker encoder based on hyperparameters determined by the target timbre features and labeled timbre features corresponding to multiple training audios. The hyperparameters determined by the target timbre features and labeled timbre features corresponding to multiple training audios can be determined by a preset parameter tuning function, or they can be hyperparameters input by a technician based on the target timbre features and labeled timbre features corresponding to multiple training audios, etc., and are not specifically limited here.

[0114] In one embodiment, after performing a first parameter tuning on the content encoder based on the target content features and tag content features corresponding to multiple training audios, a second parameter tuning can be performed on the speaker encoder based on the target timbre features and tag timbre features corresponding to the multiple training audios. The second parameter tuning refers to adjusting the parameters of the speaker encoder based on the hyperparameters determined by the target timbre features and tag timbre features corresponding to the multiple training audios. Here, the hyperparameters determined by the target timbre features and tag timbre features corresponding to the multiple training audios can be determined by a preset parameter tuning function, or they can be hyperparameters input by a technician based on the target timbre features and tag timbre features corresponding to the multiple training audios, etc., and are not specifically limited here.

[0115] The following example illustrates the application scenario of the speech conversion method provided in this application. Assume that the current preset audio database includes multiple fund customer service audio recordings A, all recorded by customer service personnel B. The current intelligent fund agent function can perform speech conversion on these multiple recordings A using the current conversion model. When multiple fund customer service audio recordings D, recorded by customer service personnel C, are added to the preset audio database, the conversion model's content extraction capability is enhanced because the content encoder is tuned based on content features incorporating image transformation factors during training. When performing speech conversion on fund customer service audio recordings D, the conversion model can reduce the impact of the different timbre between customer service personnel C and B, accurately and stably extracting the content from the recordings D. This allows for accurate output of the target customer service audio corresponding to recording D. Therefore, technicians do not need to retrain the conversion model, reducing maintenance costs.

[0116] Please see Figure 3 , Figure 3 It shows Figure 1 Another training step of the transformation model used in the process. In one embodiment, the Mel spectrogram corresponding to each training audio includes multiple segmented Mel spectrograms, and the audio training set also includes the labeled timbre features corresponding to each training audio.

[0117] Step 220 may include the following sub-steps:

[0118] Step 310: Perform random audio segmentation on each training audio to obtain multiple segmented audios corresponding to each training audio;

[0119] Step 320: Perform Mel spectrogram transformation on multiple segmented audio files to obtain the segmented Mel spectrogram corresponding to each segmented audio file;

[0120] After multiple target Mel spectra are input into the initial model, the method also includes the following steps:

[0121] Step 330: Extract timbre features from the target Mel spectrogram corresponding to each segmented Mel spectrogram using the speaker encoder to obtain the segmented timbre features corresponding to each segmented Mel spectrogram;

[0122] Step 340: For each training audio, according to the preset combination ratio coefficient, perform feature combination processing on multiple segmented timbre features corresponding to the training audio to obtain the target timbre features;

[0123] If the current initial model does not meet the preset training termination conditions, the method also includes the following steps:

[0124] Step 350: Perform a second parameter tuning on the speaker encoder based on the target timbre features and label timbre features corresponding to each training audio.

[0125] In one embodiment, audio segmentation processing refers to the operation of dividing audio into multiple segmented audio files according to a specific segmentation ratio; random audio segmentation processing refers to the operation of dividing audio into multiple segmented audio files according to a random segmentation ratio. The number of segmented audio files obtained through random audio segmentation processing can be the same or different, and is not specifically limited here.

[0126] In one embodiment, the segmented Mel spectrogram corresponding to each segmented audio is used for timbre feature extraction, not for content feature extraction. That is, for a training audio, the training audio is first randomly segmented to obtain multiple segmented audios. Then, the training audio and its corresponding segmented audios are subjected to Mel spectrogram transformation to obtain the Mel spectrogram corresponding to the complete training audio and the Mel spectrogram corresponding to each segmented audio. Then, the same image transformation is performed on the Mel spectrogram corresponding to the complete training audio and the Mel spectrogram corresponding to each segmented audio to obtain the target Mel spectrogram corresponding to the complete training audio and the target Mel spectrogram corresponding to each segmented audio (i.e., the target Mel spectrogram corresponding to the segmented Mel spectrogram mentioned in step 330). The target Mel spectrogram corresponding to the complete training audio is input to the content feature encoder to obtain target content features, and the target Mel spectrogram corresponding to the segmented audio is input to the speaker encoder to obtain segmented timbre features.

[0127] In one embodiment, feature combination processing refers to, for multiple segmented timbre features corresponding to a training audio, weighting and averaging the values ​​of each segmented timbre feature using a preset combination ratio coefficient to form a timbre feature (in this embodiment, the target timbre feature) that can represent the multiple segmented timbre features. The preset combination ratio coefficient refers to the coefficient used to weight the various segmented timbre features. For each training audio, the preset combination ratio coefficient used for each training audio corresponds to the number of segmented audios corresponding to each training audio.

[0128] It is important to note that in the actual process of speech conversion of customer service audio, random audio segmentation is also performed on the customer service audio to obtain multiple customer service segment audios. The speaker encoder extracts the timbre features of the Mel spectrograms corresponding to the multiple customer service segment audios to obtain the segmented timbre features of the multiple customer service segment audios. By combining the segmented timbre features of the multiple customer service segment audios, the target timbre features corresponding to the customer service audio are obtained.

[0129] exist Figure 3In the illustrated embodiment, for a single audio file, since the audio contains various timbre information, random segmentation of the audio can distribute different timbre information across different segmented audio files. Therefore, the speaker encoder can extract more refined timbre features from each segmented audio file, enabling it to more accurately capture the timbre information in the audio and output more precise timbre features. This reduces the dilution or masking of certain timbre features due to the diversity of audio content. Thus, in the process of speech-to-speech conversion of customer service audio, more accurate timbre feature extraction can be performed on different customer service audio files with similar timbres, thereby improving the quality of the target customer service audio. Furthermore, the introduction of random segmentation during the training phase results in more refined target timbre features, making the difference between the target timbre features and the standard timbre features more accurate. This facilitates more precise second parameter tuning of the speaker encoder.

[0130] Please see Figure 4 , Figure 4 An embodiment of a sub-step of step 230 is shown. In one embodiment, for each training audio, where the current image transformation includes image compression, step 230 may include the following sub-step.

[0131] Step 410: Obtain the current compression ratio;

[0132] Step 420: Based on the compression ratio coefficient, perform image compression on the Mel spectrogram corresponding to the training audio in the frequency domain dimension to obtain the first Mel spectrogram;

[0133] Step 430: Perform prediction completion processing on the first Mel spectrum to obtain a target Mel spectrum with the same size as the Mel spectrum.

[0134] In one embodiment, the compression ratio refers to the size ratio in one direction between the desired Mel spectrogram obtained by image compression of a Mel spectrogram and the original uncompressed Mel spectrogram. For example, if the length ratio of the desired Mel spectrogram to the original uncompressed Mel spectrogram is 80% in the length direction, then the compression ratio is 80%. The compression ratio can be a preset ratio corresponding to a training round, for example, preset ratio A used in training rounds n to 2n-1, and preset ratio B used in training rounds 2n ​​to 3n-1. The compression ratio can also be a ratio obtained in the current training round based on the preset ratio corresponding to the training round, etc., and is not specifically limited here.

[0135] In one embodiment, image compression of a Mel spectrogram in the frequency domain refers to the operation of compressing the data represented by a Mel spectrogram in the frequency domain and representing the compressed data as a first Mel spectrogram. Here, the first Mel spectrogram refers to the Mel spectrogram with the same length but reduced height after image compression in the frequency domain. Generally, the horizontal direction of a Mel spectrogram represents the time domain dimension, and the vertical direction represents the frequency domain dimension. When the Mel spectrogram is compressed in the frequency domain, the waveform displayed by the first Mel spectrogram is more compact and shorter than the waveform displayed by the original Mel spectrogram due to the compression effect on the data represented in the frequency domain. Furthermore, there are various ways to compress the data represented by a Mel spectrogram in the frequency domain, such as through Discrete Cosine Transform (DCT) or Wavelet Transform, etc., which are not specifically limited here.

[0136] In one embodiment, the prediction completion process refers to the operation of supplementing the first Mel spectrogram with predicted data points in the frequency domain dimension to obtain a target Mel spectrogram with the same size as the first Mel spectrogram. Specifically, based on the time domain dimension of the first Mel spectrogram, data points corresponding to frequencies whose frequency values ​​are higher than the frequency values ​​represented by the length of the first Mel spectrogram are predicted, and the predicted data points are compared with the first Mel spectrogram. Figure 1 The same is represented as a new Mel spectrogram (i.e., the target Mel spectrogram). The prediction completion process can be performed by linear interpolation, or by generating neural network prediction data points and then interpolating the predicted data points, etc., etc., without being limited to one specific method here.

[0137] exist Figure 4In the illustrated embodiment, due to image compression, the size of the compressed first Mel spectrogram differs from the size required by the content encoder. Therefore, predictive completion processing is needed to complete the first Mel spectrogram into a target Mel spectrogram of the same size as the content encoder's required Mel spectrogram. For a training audio, the semantic content it expresses does not change with image transformations, meaning that theoretically, regardless of whether the Mel spectrogram undergoes image compression and predictive completion processing, the target content features extracted by the content encoder should be the same as the corresponding standard content features. Therefore, training the content encoder using the target Mel spectrogram can effectively improve the training strength of the content encoder in extracting content information in the Mel spectrogram that remains unchanged before and after image compression. Then, based on the significant difference between the target content features of the target Mel spectrogram and the corresponding standard content features, the content encoder undergoes first parameter tuning, which improves the content encoder's ability to extract content information in the Mel spectrogram that remains unchanged before and after image compression. This helps improve the quality of the audio content extracted by the conversion model, thereby improving the quality of the target customer audio.

[0138] Please see Figure 5 , Figure 5 Another embodiment of a sub-step of step 230 is shown. In one embodiment, for each training audio, if the current image transformation includes image stretching, step 230 may include the following sub-step.

[0139] Step 510: Obtain the current stretch ratio coefficient;

[0140] Step 520: Based on the stretching ratio coefficient, stretch the Mel spectrogram corresponding to the training audio in the frequency domain to obtain the second Mel spectrogram;

[0141] Step 530: Based on the size of the Mel spectrum, the second Mel spectrum is cropped to obtain a target Mel spectrum with the same size as the original Mel spectrum.

[0142] In one embodiment, the stretching ratio refers to the size ratio in one direction between the desired Mel spectrogram obtained by stretching a Mel spectrogram and the original Mel spectrogram without image compression. For example, if the length ratio of the desired Mel spectrogram to the original Mel spectrogram without image stretching is 120% in the length direction, then the compression ratio is 120%. The stretching ratio can be a preset ratio corresponding to the training rounds. For example, the preset ratio used in training rounds n to 2n-1 is A, and the preset ratio used in training rounds 2n ​​to 3n-1 is B.

[0143] In one embodiment, stretching a Mel spectrogram in the frequency domain refers to interpolating the data represented by a Mel spectrogram in the frequency domain and then combining the interpolated data with the original data to form a second Mel spectrogram. Here, the second Mel spectrogram is the Mel spectrogram with the same length but increased height after stretching in the frequency domain. Generally, the horizontal direction of a Mel spectrogram represents the time domain dimension, and the vertical direction represents the frequency domain dimension. When a Mel spectrogram is stretched in the frequency domain, the resolution of the data displayed in the second Mel spectrogram increases due to the interpolation process, resulting in a more diffuse and higher waveform compared to the original Mel spectrogram. Furthermore, there are various ways to interpolate the data represented by a Mel spectrogram in the frequency domain, such as linear interpolation, generating neural networks to predict data points and then interpolating using the predicted data points, etc. No specific method is limited here.

[0144] In one embodiment, the cropping process refers to the operation of cropping data points from the highest frequency to the lowest frequency in a certain frequency span based on the second Mel spectrogram, and deleting the cropped data points to obtain a target Mel spectrogram with the same size as the original Mel spectrogram.

[0145] exist Figure 5 In the illustrated embodiment, due to image stretching, the size of the stretched second Mel spectrogram differs from the size of the Mel spectrogram required by the content encoder. Therefore, cropping is necessary to trim the second Mel spectrogram to the same size as the Mel spectrogram required by the content encoder. For a training audio, the semantic content it expresses does not change with image transformations. In other words, theoretically, regardless of whether the Mel spectrogram undergoes image stretching and cropping, the target content features extracted by the content encoder should be the same as the corresponding standard content features. Therefore, training the content encoder using the target Mel spectrogram can effectively improve the training strength of the content encoder in extracting content information that remains unchanged before and after image stretching in the Mel spectrogram. Then, by leveraging the significant difference between the target content features of the target Mel spectrogram and the corresponding standard content features, the content encoder undergoes first parameter tuning. This improves the content encoder's ability to extract content information that remains unchanged before and after image stretching in the Mel spectrogram, which helps improve the quality of the audio content extracted by the conversion model, thereby improving the quality of the target customer audio.

[0146] Please see Figure 6 , Figure 6 It shows Figure 1Another training step of the transformation model used in the process. In one embodiment, the audio training set also includes labeled audio for each training audio, and the target Mel spectrogram for each training audio includes a target Mel spectrogram for timbre feature extraction.

[0147] Step 410 may include the following sub-steps:

[0148] Step 610: Determine the compression ratio coefficient based on the fusion coefficient and the current preset compression ratio coefficient. The fusion coefficient of the training audio is obtained by extracting timbre features from the target Mel spectrogram corresponding to the previous training audio using the speaker encoder.

[0149] After obtaining the target content features corresponding to each training audio, the method also includes:

[0150] Step 620: Perform audio conversion processing on multiple target content features respectively to obtain the target audio corresponding to each training audio;

[0151] If the current initial model does not meet the preset training termination conditions, the method also includes the following steps:

[0152] Step 630: Perform third parameter tuning on the initial model based on the target audio and label audio corresponding to each training audio.

[0153] In one embodiment, when random segmentation of the Mel spectrogram is not involved, the target Mel spectrogram for timbre feature extraction and the target Mel spectrogram for content feature extraction are the same target Mel spectrogram; when random segmentation of the Mel spectrogram is involved, the target Mel spectrogram for timbre feature extraction is the target Mel spectrogram corresponding to the segmented Mel spectrogram.

[0154] In one embodiment, the previous training audio refers to the previous training audio in the audio training set currently used for training audio.

[0155] In one embodiment, the fusion coefficient refers to the coefficient used to represent the feature fusion process of the decoder in performing content features and timbre features. Specifically, after the speaker encoder extracts timbre features, it also outputs the fusion coefficient used in the next training audio. In the speaker encoder, the output fusion coefficient can be obtained from a pre-constructed mapping function. Specifically, the initial model is a model with basic speech conversion capabilities. A mapping function can be constructed based on the timbre features output by the initial model and the audio. During training, the fusion coefficient can be obtained through the mapping function and the output target timbre features. The fusion coefficient can also be obtained from the target timbre features. Specifically, the target timbre features include frequency components corresponding to the timbre. Dimensionality reduction of the target timbre features can be performed using pooling or other methods to obtain a single-dimensional value, which is the fusion coefficient. The fusion coefficient can also be a preset value, etc., and is not specifically limited here.

[0156] It is important to note that the fusion coefficient is only introduced during the training of the initial model into a conversion model, and is not introduced during the actual speech conversion of customer service audio. In other words, in practice, the content features and timbre features of the customer service audio still need to be fused using a decoder.

[0157] In one embodiment, audio conversion processing refers to the operation of converting the features of the input audio into simulated audio using an audio generation network. During training, the features input to the audio generation network are the target content features; in practical applications, the features input to the audio generation network are the fused features obtained by the decoder by fusing the content features and timbre features of the input audio. It is important to note that during training, due to the use of fusion coefficients, the features input to the decoder are actually only the target content features.

[0158] In one embodiment, the third parameter tuning refers to determining a third loss value based on the target audio and the labeled audio corresponding to each training audio, and then performing parameter tuning on the initial model components (such as the decoder, audio generation network, etc. mentioned above) in the initial model based on the third loss value.

[0159] exist Figure 6 In the illustrated embodiment, the fusion coefficients are obtained through the speaker encoder, and the preset compression ratio coefficients can be adjusted using the fusion coefficients to obtain the compression ratio coefficients. In turn, the feature fusion process of the subsequent decoder can be omitted through image compression, which helps to speed up the initial model's computation process on the training audio, thereby improving the training efficiency of the initial model.

[0160] Please see Figure 7 , Figure 7 It shows Figure 1Another training step of the transformation model used in the process. In one embodiment, when the image change corresponds to image stretching, step 510 may include the following sub-steps:

[0161] Step 710: Determine the stretching ratio coefficient based on the fusion coefficient and the current preset stretching ratio coefficient. The fusion coefficient of the training audio is obtained by extracting timbre features from the target Mel spectrogram corresponding to the previous training audio using the speaker encoder.

[0162] exist Figure 7 In the illustrated embodiment, fusion coefficients are obtained through a speaker encoder, which can then be used to adjust a preset stretching ratio coefficient to obtain a stretching ratio coefficient. This allows the feature fusion process of the subsequent decoder to be omitted through image stretching, which helps to accelerate the initial model's computation process on the training audio and thus improves the training efficiency of the initial model.

[0163] Please see Figure 8 , Figure 8 It shows Figure 2 An embodiment of a sub-step of step 250. In one embodiment, step 250 may include the following sub-steps.

[0164] Step 810: Perform a loss minimization analysis based on the target content features and tag content features corresponding to multiple training audios to determine the first loss value;

[0165] Step 820: Perform the first parameter tuning on the content encoder for the first loss value.

[0166] In one embodiment, the initial model may introduce a reward function, and then introduce a reinforcement learning mechanism into the initial model. Based on this, during the first parameter tuning of the content encoder for the first loss value, the initial model can specifically perform hyperparameter simulation based on the first loss value to determine the optimal hyperparameters of the content encoder in the current round, and perform the first parameter tuning of the content encoder based on the optimal hyperparameters.

[0167] In one embodiment, during the first parameter tuning of the content encoder for the first loss value, the hyperparameters input by the technician for the first loss value can be obtained first, and the content encoder can be tuned according to the input hyperparameters.

[0168] exist Figure 8In the illustrated embodiment, since the target content features extracted by the content encoder should be the same as the corresponding standard content features, the difference between the content information that remains unchanged before and after image compression in the actual extracted Mel spectrogram and the content information that remains unchanged before and after image compression in the theoretical Mel spectrogram can be determined by minimizing the first loss value corresponding to the loss between the target content features and the tag content features. This helps to perform a first parameter tuning to enhance the content extraction capability of the content encoder, thereby improving the quality of the content information extracted from the customer service audio by the content encoder in the conversion model, and improving the robustness of the content information extraction process of the content encoder in the conversion model.

[0169] Please see Figure 9 , Figure 9 It shows Figure 3 An embodiment of a sub-step of step 350. In one embodiment, step 350 may include the following sub-steps.

[0170] Step 910: Perform angular edge loss analysis based on multiple segmentation timbre features corresponding to each training audio to determine the second loss value;

[0171] Step 920: Perform maximum loss analysis based on the target timbre features and tag timbre features corresponding to each training audio to determine the third loss value;

[0172] Step 930: Perform a second parameter tuning on the speaker encoder for the second and third loss values.

[0173] In one embodiment, angular edge loss analysis based on multiple segmented timbre features corresponding to each training audio refers to the process of performing angular edge loss analysis on the multiple segmented timbre features corresponding to each training audio to obtain a second loss value. Specifically, for an audio recording, since it was recorded by a customer service representative, after random audio segmentation, each segmented audio theoretically corresponds to the same timbre. That is, the multiple segmented timbre features corresponding to the audio should be identical. Therefore, by introducing a second loss value, the differences in timbre features when the speaker encoder performs timbre feature analysis on audio with the same timbre can be determined. This helps to reduce the differences in segmented timbre features corresponding to different segmented audio with the same timbre during the second parameter tuning of the speaker encoder, thereby making the timbre features output by the speaker encoder more accurate.

[0174] In one embodiment, the initial model may introduce a reward function, thereby introducing a reinforcement learning mechanism into the initial model. Based on this, during the second parameter tuning of the speaker encoder for the second and third loss values, the initial model may specifically perform hyperparameter simulation based on the second and third loss values ​​to determine the optimal hyperparameters of the speaker encoder in the current round, and then perform the second parameter tuning of the speaker encoder based on the optimal hyperparameters.

[0175] In one embodiment, during the second parameter tuning of the speaker encoder for the second loss value and the third loss value, the hyperparameters input by the technician for the second loss value and the third loss value can be obtained first, and the speaker encoder can be tuned for the second parameter tuning based on the input hyperparameters.

[0176] exist Figure 9 In the illustrated embodiment, after the Mel spectrogram undergoes image transformation, the timbre it represents is actually different from the timbre in the training audio. That is, the target timbre features extracted by the speaker encoder should be different from the corresponding standard timbre features. Therefore, the speaker encoder can be tuned by maximizing the third loss value corresponding to the loss between the target content features and the label content features. This allows the speaker encoder to improve the discriminative power between the target timbre features and the label timbre features. Thus, even if similar timbre customer service audio exists in the preset database, the timbre features extracted by the speaker encoder can better display the timbre differences between two similar timbres, thereby enhancing the speaker encoder's timbre feature extraction capability. Furthermore, introducing a second loss value during the speaker encoder tuning process can reduce the differences between the segmented timbre features corresponding to different segmented audio segments of the same timbre, further enhancing the speaker encoder's timbre feature extraction capability. Therefore, when new customer service audio is added to the preset database, regardless of whether the timbre of the recorded customer service audio is the same as the previous timbre, the speaker encoder can extract the timbre features more accurately, thereby improving the speech conversion capability of customer service audio.

[0177] Please see Figure 10 This application also provides a speech conversion device that can implement the above method. The speech conversion device 1000 includes:

[0178] The customer service audio acquisition module 1010 is used to acquire customer service audio to be processed.

[0179] The target customer service audio output module 1020 is used to perform speech conversion processing on customer service audio using a pre-trained conversion model and output the target customer service audio.

[0180] In one embodiment, the pre-trained conversion model used by the target customer service audio output module 1020 can be trained through the following steps: First, an audio training set including multiple training audios and the tag content features corresponding to each training audio is obtained. Then, Mel spectrogram conversion processing is performed on the multiple training audios to obtain the Mel spectrogram corresponding to each training audio. Then, for each training audio, image transformation is performed on the corresponding Mel spectrogram to obtain the target Mel spectrogram. The image transformation is either image stretching or image compression. The training audios in the audio training set are alternately transformed by either image stretching or image compression. Then, the multiple target Mel spectrograms are input into the initial model including the content encoder to obtain the target content features corresponding to each training audio. Then, if the current initial model does not meet the preset training termination condition, the content encoder is first tuned according to the target content features and tag content features corresponding to the multiple training audios. The tuned initial model is then trained based on the audio training set until the preset training termination condition is met to obtain the conversion model.

[0181] The specific implementation of this voice conversion device is basically the same as the specific embodiment of the above-described voice conversion method, and will not be repeated here.

[0182] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described voice conversion method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0183] Please see Figure 11 , Figure 11 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device 1100 includes:

[0184] The processor 1101 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0185] The memory 1102 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1102 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1102 and is called and executed by the processor 1101 using the speech conversion method of the embodiments of this application.

[0186] Input / output interface 1103 is used to implement information input and output;

[0187] The communication interface 1104 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0188] Bus 1105 transmits information between various components of the device (e.g., processor 1101, memory 1102, input / output interface 1103, and communication interface 1104);

[0189] The processor 1101, memory 1102, input / output interface 1103 and communication interface 1104 are connected to each other within the device via bus 1105.

[0190] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described speech conversion method.

[0191] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0192] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of this application, and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

[0193] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0194] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0195] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0196] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0197] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0198] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0199] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0200] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0201] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0202] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A speech conversion method, characterized in that, The method includes: Retrieve pending customer service audio; The customer service audio is processed by a pre-trained conversion model to output the target customer service audio. The conversion model is trained through the following steps: Obtain an audio training set, which includes multiple training audios and label content features corresponding to each training audio; The Mel spectrogram transformation process is performed on the multiple training audios respectively to obtain the Mel spectrogram corresponding to each training audio; For each training audio, an image transformation is performed on the Mel spectrogram corresponding to the training audio to obtain a target Mel spectrogram. The image transformation is either image stretching or image compression. The training audios in the audio training set are alternately transformed using either image stretching or image compression. The multiple target Mel spectrograms are respectively input into the initial model to obtain the target content features corresponding to each training audio. The initial model includes a content encoder. If the current initial model does not meet the preset training termination condition, the content encoder is subjected to a first parameter tuning based on the target content features and tag content features corresponding to multiple training audios; The initial model after parameter tuning is trained on the audio training set until the preset training termination condition is met, and the transformed model is obtained.

2. The method according to claim 1, characterized in that, Each training audio corresponds to a Mel spectrogram, which includes multiple segmented Mel spectrograms; the audio training set also includes tactile features labeled for each training audio; the initial model also includes a speaker encoder, which runs in parallel with the content encoder; The Mel spectrogram transformation process performed on the audio training set yields multiple Mel spectrograms, including: Each training audio is subjected to random audio segmentation to obtain multiple segmented audios corresponding to each training audio. The Mel spectrogram conversion process is performed on the multiple segmented audios to obtain the segmented Mel spectrogram corresponding to each segmented audio; After inputting multiple target Mel spectrograms into the initial model, the method further includes: extracting timbre features from the target Mel spectrogram corresponding to each segmented Mel spectrogram using the speaker encoder to obtain segmented timbre features corresponding to each segmented Mel spectrogram; For each training audio, according to a preset combination ratio coefficient, the multiple segmented timbre features corresponding to the training audio are subjected to feature combination processing to obtain the target timbre features; If the current initial model does not meet the preset training termination condition, the method further includes: performing a second parameter tuning on the speaker encoder based on the target timbre feature and the tag timbre feature corresponding to each training audio.

3. The method according to claim 1, characterized in that, In the case where the current image transformation includes image compression, the step of performing image transformation on the corresponding Mel spectrogram to obtain the target Mel spectrogram includes: Get the current compression ratio; Based on the compression ratio coefficient, the Mel spectrogram corresponding to the training audio is compressed in the frequency domain to obtain a first Mel spectrogram; The first Mel spectrum is subjected to prediction completion processing to obtain the target Mel spectrum with the same size as the first Mel spectrum.

4. The method according to claim 1, characterized in that, When the image transformation includes image stretching, the step of performing image transformation on the corresponding Mel spectrogram to obtain the target Mel spectrogram includes: Get the current stretch ratio coefficient; Based on the stretching ratio coefficient, the Mel spectrogram corresponding to the training audio is stretched in the frequency domain dimension to obtain a second Mel spectrogram; Based on the size of the Mel spectrum, the second Mel spectrum is cropped to obtain the target Mel spectrum with the same size as the first Mel spectrum.

5. The method according to claim 3, characterized in that, The audio training set also includes labeled audio for each training audio; the initial model also includes a speaker encoder; the target Mel spectrogram for each training audio includes the target Mel spectrogram used for timbre feature extraction; The process of obtaining the current compression ratio coefficient includes: The compression ratio coefficient is determined based on the fusion coefficient and the current preset compression ratio coefficient. The fusion coefficient of the training audio is obtained by the speaker encoder extracting timbre features from the target Mel spectrogram corresponding to the previous training audio for timbre feature extraction. After obtaining the target content feature corresponding to each training audio, the method further includes: performing audio conversion processing on the multiple target content features respectively to obtain the target audio corresponding to each training audio; If the current initial model does not meet the preset training termination condition, the method further includes: performing a third parameter tuning on the initial model based on the target audio and the label audio corresponding to each training audio.

6. The method according to claim 1, characterized in that, The first parameter tuning of the content encoder based on the target content features and tag content features corresponding to multiple training audios includes: Based on the target content features and tag content features corresponding to multiple training audios, a minimization loss analysis is performed to determine the first loss value; The content encoder is subjected to a first parameter tuning for the first loss value.

7. The method according to claim 2, characterized in that, The step of performing a second parameter tuning on the speaker encoder based on the target timbre feature and the labeled timbre feature corresponding to each training audio includes: Based on the multiple segmentation timbre features corresponding to each training audio, angular edge loss analysis is performed to determine the second loss value; A third loss value is determined by performing a maximum loss analysis based on the target timbre features and the tag timbre features corresponding to each training audio; The speaker encoder is subjected to a second parameter tuning for the second loss value and the third loss value.

8. A voice conversion device, characterized in that, The device includes: The customer service audio acquisition module is used to acquire customer service audio files to be processed. The target customer service audio output module is used to perform speech conversion processing on the customer service audio using a pre-trained conversion model and output the target customer service audio. The conversion model is trained through the following steps: Obtain an audio training set, which includes multiple training audios and label content features corresponding to each training audio; The Mel spectrogram transformation process is performed on the multiple training audios respectively to obtain the Mel spectrogram corresponding to each training audio; For each training audio, an image transformation is performed on the Mel spectrogram corresponding to the training audio to obtain a target Mel spectrogram. The image transformation is either image stretching or image compression. The training audios in the audio training set are alternately subjected to either image stretching or image compression for image transformation. The multiple target Mel spectrograms are respectively input into the initial model to obtain the target content features corresponding to each training audio. The initial model includes a content encoder. If the current initial model does not meet the preset training termination condition, the content encoder is subjected to a first parameter tuning based on the target content features and tag content features corresponding to multiple training audios; The initial model after parameter tuning is trained on the audio training set until the preset training termination condition is met, and the transformed model is obtained.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the speech conversion method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the speech conversion method according to any one of claims 1 to 7.

Images