A real-time voice conversion method and device, electronic equipment and medium

Abstract

The application provides a real-time voice conversion method and device, electronic equipment and medium. The method comprises the following steps: intercepting first voice data meeting voice segmentation conditions from voice data of a source speaking object recorded in real time; processing the first voice data to extract first semantic information; inputting the first semantic information into a pre-trained voice conversion model, and converting and processing effective information of historical voice data before the first semantic information and the first voice data through the voice conversion model to obtain target voice feature information corresponding to the first semantic information and a voice factor of a target speaking object; reconstructing the target voice feature information to obtain second voice data converted from the first voice data, thereby realizing low-delay streaming inference and low-delay and high-performance real-time voice conversion.

Description

Technical Field

[0001] This application relates to the field of speech conversion, and more specifically, to a real-time speech conversion method, apparatus, electronic device, and medium. Background Technology

[0002] Voice conversion is a technology that changes a speaker's voice to that of another person while preserving the original content of the speech. In other words, it alters certain features of the speech, such as the speaker's timbre, style, and accent, while maintaining the original semantic meaning.

[0003] In real life, speech conversion technology has already found numerous practical applications, such as voice changers, film and television dubbing, voice assistants, personalized conversion, and helping people with speech impairments recover their damaged speech, demonstrating broad development prospects. However, in practical applications, speech conversion is still limited by factors such as data volume, computing resources, real-time performance, and conversion quality. Specifically, existing technical solutions mainly rely on whole-segment conversion based on non-parallel corpora. Traditional speech conversion models require the acquisition of all input speech before conversion can begin, which cannot meet the low-latency, high-performance real-time requirements of real-time applications. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a real-time speech conversion method, apparatus, electronic device and medium that can achieve low-latency streaming inference, thereby achieving low-latency, high-performance real-time speech conversion.

[0005] This application provides a real-time speech conversion method, which includes the following steps:

[0006] Extract the first voice data that meets the voice segmentation conditions from the real-time recorded voice data of the source speaker;

[0007] Process the first speech data and extract the first semantic information from the first speech data;

[0008] The first semantic information is input into a pre-trained speech conversion model, and the effective information of the first semantic information and the historical speech data preceding the first speech data is converted by the speech conversion model to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker; wherein, the effective information is the information that affects the speech conversion of the first semantic information.

[0009] The target speech feature information is reconstructed to obtain the second speech data after the first speech data is converted.

[0010] In some embodiments, a real-time speech conversion device is also provided, the device comprising:

[0011] The extraction module is used to extract the first voice data that meets the voice segmentation conditions from the real-time recorded voice data of the source speaker.

[0012] The extraction module is used to process the first speech data and extract the first semantic information of the first speech data;

[0013] The conversion module is used to input the first semantic information into a pre-trained speech conversion model, and to convert the first semantic information and the effective information of the historical speech data preceding the first speech data through the speech conversion model to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker; wherein, the effective information is the information that affects the speech conversion of the first semantic information.

[0014] The reconstruction module is used to reconstruct the target speech feature information to obtain the second speech data after the first speech data is converted.

[0015] In some embodiments, an electronic device is also provided, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps of the real-time speech conversion method are performed.

[0016] In some embodiments, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, performs the steps of the real-time speech conversion method.

[0017] Based on this, embodiments of this application provide a real-time speech conversion method, apparatus, electronic device, and medium. Real-time recorded speech is input into a speech conversion model in segments, rather than as a whole segment, thereby reducing latency issues in real-time applications. Then, the first speech information of the segmented speech is identified. The first semantic information and valid information from historical speech data preceding the first speech data are processed by a pre-trained speech conversion model to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker. Finally, the target speech feature information is reconstructed to obtain second speech data converted from the first speech data. The use of valid information from historical speech data reduces the impact of speech segmentation on the continuity of the entire speech segment, enabling the speech conversion model to perform inference in a continuous, streaming manner, meeting the high-performance requirements of real-time applications. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart of the real-time speech conversion method according to an embodiment of this application is shown;

[0020] Figure 2 A flowchart of a method for processing the first speech data and extracting the first semantic information of the first speech data according to an embodiment of this application is shown.

[0021] Figure 3 A flowchart of the training method described in an embodiment of this application is shown;

[0022] Figure 4 A schematic diagram of the training phase of the training method described in an embodiment of this application is shown;

[0023] Figure 5 This document illustrates a flowchart of a method for converting the first semantic information and valid information from historical speech data preceding the first speech data using the speech conversion model, as described in an embodiment of this application.

[0024] Figure 6 A schematic diagram of the speech conversion model described in an embodiment of this application is shown;

[0025] Figure 7 This document illustrates a flowchart of a method described in this application for converting effective information from the first semantic information and the historical speech data preceding the first speech data using the speech conversion model to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker.

[0026] Figure 8 The flowchart of the method described in this application embodiment, which uses an encoder of a speech conversion model to generate target speech vector features corresponding to the first semantic information and the target speech factor vector, based on the target speech factor vector, the first semantic information, and the second effective information of historical speech data prior to the first speech data, is shown.

[0027] Figure 9 This document illustrates a flowchart of a method described in this application for processing the target speech vector features and third effective information from historical speech data prior to the first speech data using the decoder to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker.

[0028] Figure 10 A schematic diagram of the structure of the real-time speech conversion device according to an embodiment of this application is shown;

[0029] Figure 11 A schematic diagram of the structure of the electronic device is shown. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0031] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0032] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0033] Speech, as the external expression of language, is the most natural form of communication in our daily lives. Speech not only contains the content information of language but also conveys multi-dimensional information such as the speaker's timbre, emotion, and style. Voice conversion is a technology that changes a speaker's timbre to that of another person while preserving the original speech content. That is, it alters certain features of the speech, including the speaker's timbre, style, and accent, while maintaining the original semantic meaning.

[0034] With the development of deep learning, speech conversion technology has also made significant progress. Speech conversion models based on deep neural networks have significantly improved the similarity and naturalness of the converted speech. The emergence of neural vocoders has enabled the output sound quality of speech conversion to approach the level of real recordings.

[0035] In real life, speech conversion technology has already found numerous practical applications, such as voice changers, film and television dubbing, voice assistants, personalized conversion, and helping people with speech impairments recover their damaged speech, demonstrating broad development prospects. However, in practical applications, speech conversion is still limited by factors such as data volume, computing resources, real-time performance, and conversion quality. This application improves the real-time inference and optimization of the model, enabling the speech conversion model to possess streaming inference capabilities, thus improving model inference efficiency and allowing it to be used in scenarios such as real-time voice changing during calls and privacy protection.

[0036] Existing speech conversion technologies mainly consist of three steps: feature analysis and extraction, feature conversion, and speech synthesis. Feature analysis and extraction involves extracting feature information that characterizes the speech data from the source speaker's speech data, such as the Mel spectrum of the speech data. Feature conversion involves transforming the feature information of the speech data into feature information corresponding to the speech elements of the target speaker, such as the Mel spectrum of the speech elements of the target speaker. Speech synthesis involves recombining the converted feature information into acoustic audio features.

[0037] Based on the characteristics of the training data, speech conversion can be divided into methods based on parallel corpora and methods based on non-parallel corpora. Parallel corpus methods use data that includes recordings of the source and target speakers delivering the same semantic content. Conversely, non-parallel data does not contain recordings with identical content but different speakers. The improvement from parallel corpus methods to non-parallel corpus methods addresses the problems of difficulty in obtaining and limited volume of parallel corpus data.

[0038] Parallel corpus-based speech-to-speech methods use parallel recordings with identical text, including both source and target speakers, for training. The main approach involves using Dynamic Time Warping (DTW) to align the source and target speaker recordings at the frame level to obtain a training set. A speech-to-speech model is then trained on this set, modeling the feature mapping relationship between the source and target speakers. Commonly used methods include traditional Gaussian Mixture Models (GMMs), vector quantization, instance selection, nonnegative matrix factorization, and partial least squares regression. Since the attention mechanism was proposed in 2014, it has been widely used in image processing and natural language processing. In 2019, Google proposed the Parrotron model, which uses an attention mechanism to align parallel corpora. However, because parallel corpus methods are generally one-to-one speech-to-speech methods, their practicality is limited; and because they require parallel data from two or more speakers, the cost is too high, resulting in their limited use.

[0039] Non-parallel speech conversion refers to methods that do not require parallel corpus data for conversion. There are two common approaches: one is to align the source and target speaker audio in the non-parallel corpus at the phoneme level, transforming the problem into speech conversion using parallel corpus data; the other is to use audio data containing only the target speaker. Currently, mainstream methods for non-parallel speech conversion include methods using speech recognition features, VAE (Variational Auto-Encoder), and GAN (Generative Adversarial Network), among others, with significant differences in framework. A commonly used method is based on Phonetic Posterior Grams (PPG) features from speech recognition. Its core idea is to extract the posterior probability of each character, word, and triphone set from the audio using speech recognition techniques. This posterior probability is frame-level, preserving the duration information of the source speech while removing speaker-related information to some extent, thus decoupling the audio semantic content from the speaker's timbre. The speech conversion model learns the mapping relationship between this posterior probability and spectral features, ultimately generating audio through a vocoder. Compared to parallel corpus methods, speech conversion based on non-parallel corpora does not require hard-to-obtain parallel recordings, can widely use various datasets, and can meet conversion needs in various scenarios.

[0040] Traditional speech conversion models require all input speech before conversion, failing to meet the low latency and high performance requirements of real-time applications. Therefore, this application provides a real-time speech conversion method, apparatus, electronic device, and medium. It inputs real-time recorded speech into a speech conversion model in segments, rather than as a whole segment, thereby reducing latency in real-time applications. Then, it identifies the first semantic information of the segmented first speech data. A pre-trained speech conversion model processes the first semantic information and valid information from previous historical speech data to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker. Finally, the target speech feature information is reconstructed to obtain the second speech data converted from the first speech data. The use of valid information from historical speech data reduces the impact of speech segmentation on the continuity of the entire speech segment, enabling the speech conversion model to perform inference in a continuous, streaming manner, meeting the high performance requirements of real-time applications.

[0041] It should be noted that both the real-time speech conversion method and the virtual avatar driving method provided in this application embodiment can be implemented based on artificial intelligence. Artificial intelligence (AI) is a comprehensive discipline that utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. Basic AI technologies generally include technologies such as sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0042] In the embodiments of this application, the main artificial intelligence technologies involved include Automatic Speech Recognition (ASR) and Voice Conversion (VC).

[0043] The speech conversion provided in this application embodiment can be applied to a processing device, which can be a terminal device or a server. The processing device can have the capability to perform speech recognition and speech conversion. In this application embodiment, by implementing the above-mentioned speech conversion technology, the processing device can input real-time recorded speech into a speech conversion model in a segmented manner, thereby realizing functions such as recognizing the first semantic information of the segmented speech, obtaining target speech feature information corresponding to the first semantic information and the speech factors of the target speaker, processing the target speech feature information to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker, and processing the target speech feature information to obtain converted second speech data for playback.

[0044] The processing device can be a terminal device, such as a smart terminal, computer, personal digital assistant (PDA), tablet computer, etc.

[0045] The processing device can also be a server, such as a standalone server or a cluster server. When the server implements real-time speech conversion, it identifies the first speech information of the segmented first speech data, processes the first semantic information and the effective information of the historical speech data preceding the first speech data through a pre-trained speech conversion model, obtains target speech feature information corresponding to the first semantic information and the speech factors of the target speaker, and finally reconstructs the target speech feature information to obtain the second speech data after the first speech data is converted, and stores the second speech data or sends the second speech data to the terminal device for playback.

[0046] The real-time speech conversion method provided in this application can be applied to various applicable speech conversion scenarios, such as live streaming, real-time calls, and video recording, increasing the enjoyment of voice calls. In these scenarios, the method provided in this application can convert the real-time recorded voice data of the source speaker into the voice data of the target speaker with high quality and low latency.

[0047] The following is a detailed description of a real-time speech conversion method, apparatus, device, and medium provided in the embodiments of this application.

[0048] Reference Figure 1 As shown, Figure 1 A flowchart of the real-time speech conversion method according to an embodiment of this application is shown. The real-time speech conversion method includes the following steps S101-S104.

[0049] S101. Extract the first voice data that meets the voice segmentation conditions from the real-time recorded voice data of the source speaker.

[0050] S102. Process the first speech data and extract the first semantic information of the first speech data;

[0051] S103. Input the first semantic information into a pre-trained speech conversion model, and use the speech conversion model to convert the first semantic information and the effective information of the historical speech data before the first speech data to obtain the target speech feature information corresponding to the first semantic information and the speech factors of the target speaker; wherein, the effective information is the information that affects the speech conversion of the first semantic information.

[0052] S104. The target speech feature information is reconstructed to obtain the second speech data after the first speech data is converted.

[0053] This application provides a real-time speech conversion method. Real-time recorded speech is input into a speech conversion model in segments, rather than as a whole segment, thereby reducing latency in real-time applications. Then, the first semantic information of the segmented first speech data is identified. A pre-trained speech conversion model is used to process the first semantic information and valid information from previous historical speech data to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker. Finally, the target speech feature information is reconstructed to obtain the second speech data converted from the first speech data. The use of valid information from historical speech data reduces the impact of speech segmentation on the continuity of the entire speech segment. The speech conversion model can perform inference in a continuous, streaming manner, and the resulting segmented second speech data can be played continuously, smoothly, and with high quality, meeting the high-performance requirements of real-time applications.

[0054] Specifically, in step S101, first voice data that meets the voice segmentation conditions is extracted from the real-time recorded voice data of the source speaker; the first voice data that meets the voice segmentation conditions includes at least one of the following:

[0055] Record the first audio data in real time when the preset segment duration is reached;

[0056] Record the first audio data in real time when the frame rate reaches the preset frame threshold;

[0057] The first voice data captured upon receiving a preset segmentation command;

[0058] The first audio data captured upon completion of recording.

[0059] Step S101 involves segmenting the real-time recorded speech data of the source speaker. Specifically, there can be one or more speech segmentation conditions; for example, there can be one or more speech segmentation conditions.

[0060] For example, when extracting the first voice data of the source speaker in real-time recording, it can be set to extract a 5-second segment of the first voice data when the recording duration reaches 5 seconds.

[0061] Alternatively, in some embodiments, a set of recording durations of 5s, 8s, and 10s can be set; when the recording duration reaches 5s, a 5s long segment of the first voice data is extracted; the timing restarts, and when the recording duration reaches 8s, an 8s long segment of the first voice data is extracted; the timing restarts, and when the recording duration reaches 10s, a 10s long segment of the first voice data is extracted, and so on in a loop.

[0062] Alternatively, in some embodiments, it can be set to: when the recording duration reaches 5 seconds, extract a 5-second segment of first audio data; when the recording is completed, regardless of the recording duration, extract the first audio data from the last extraction time point to the recording end time point.

[0063] Alternatively, in some embodiments, it can be configured to: when the number of recorded frames reaches 5, extract a segment of 5 frames of first audio data; when recording is complete, regardless of the number of recorded frames, extract the first audio data from the last extraction time point to the recording end time point.

[0064] Alternatively, in some embodiments, a segmentation instruction can be generated through a user's segmentation operation, and the first voice data can be extracted upon receiving the preset segmentation instruction in real time. The user's segmentation operation can be a segmentation operation targeting the recording device of the source speaker; specifically, the segmentation operation can be a click, touch, shortcut key, etc.

[0065] In step S102, the first speech data is processed to extract the first semantic information of the first speech data.

[0066] In the embodiments of this application, reference is made to Figure 2 As shown, processing the first speech data and extracting the first semantic information of the first speech data includes the following steps S201-S202.

[0067] S201. Obtain a pre-trained speech recognition model;

[0068] S202. Input the first speech data into the speech recognition model, and use the speech recognition model to decouple the timbre of the first speech data, remove noise from the first speech data, and extract the semantic information of the first speech data.

[0069] The speech recognition model is used to identify the semantic information of the first speech data.

[0070] In this embodiment of the application, the speech recognition model is also known as the ASR (Automatic Speech Recognition) model.

[0071] In this embodiment of the application, during the training phase of the speech recognition model, a speaker-independent automatic speech recognition model (SI-ASR) is trained using multi-speaker data.

[0072] In other words, the speech recognition model described in this application is based on a speech conversion scheme using non-parallel corpora and employs an ASR model to extract semantic information. The output features of the ASR model only contain semantic information, which is equivalent to decoupling the semantics of the source audio from other information (including the speaker's timbre), thus improving the similarity of the conversion results. Therefore, speech conversion uses the semantic recognition results output by the speech recognition model as its input, which not only reduces the data requirements but also greatly improves the conversion effect.

[0073] It should be understood that the first semantic information in this embodiment is not text information, but a bottleneck feature represented in vector form. Because the most important aspect of the speech conversion method described in this embodiment is "real-time," the steps need to be simplified as much as possible to improve the real-time performance of speech conversion and reduce latency. Therefore, in the ASR part, the main steps are to extract Fbank features from the first speech data, then perform inference using the speech recognition model to obtain the bottleneck features of the penultimate layer, without needing to convert the first speech data into text information.

[0074] The speech recognition model can also remove noise from the first speech data. For example, during the training phase of the speech recognition model, the speech recognition model is trained using multi-speaker data carrying noise, so that the speech recognition model has noise removal capability, thereby removing noise from the first speech data, extracting more accurate semantic information from the first speech data, and improving the conversion effect.

[0075] In step S103, the first semantic information is input into a pre-trained speech conversion model, and the effective information of the first semantic information and the historical speech data before the first speech data is converted by the speech conversion model to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker; wherein, the effective information is the information that affects the speech conversion of the first semantic information.

[0076] In the embodiments of this application, reference is made to Figure 3 , Figure 3A flowchart of the training method according to an embodiment of this application is shown; specifically, the speech conversion model is trained through steps S301-S304:

[0077] S301. Obtain a pre-trained speech recognition model;

[0078] S302. Obtain the third speech data of the target speaker and extract the target speech feature information from the third speech data;

[0079] S303. Extract the semantic information of the third speech data using the pre-trained speech recognition model;

[0080] S304. Input the semantic information and target speech feature information of the third speech data into the pre-established speech conversion model, and train the speech conversion model until the speech conversion model meets the training completion conditions.

[0081] Here, the training completion conditions include reaching a preset number of training iterations, and the speech conversion model's detection value reaching a preset detection value. The detection value of the speech conversion model is determined by a loss function.

[0082] When the speech conversion model meets the training completion conditions, training stops, and the trained speech conversion model is obtained.

[0083] In this embodiment of the application, the target speech feature information is a Mel spectrum.

[0084] In other words, referencing Figure 4 A speech conversion system capable of implementing the speech conversion method described in this application requires training in two stages:

[0085] The first stage: Speech recognition model training: This stage uses multi-speaker data to train a speaker-independent automatic speech recognition model (SI-ASR).

[0086] The second stage: speech conversion model training: using the SI-ASR model trained in the first stage to extract BNFs (i.e. semantic information of audio) from the audio in the training set, and extracting the corresponding Mel spectrum features, the speech conversion model is trained to learn the relationship between BNFs and Mel spectrum.

[0087] Reference Figure 4After obtaining the trained speech recognition model and the trained speech conversion model through the first and second stages, the conversion stage begins. In the conversion stage, the first speech data to be converted is first input into the ASR model to obtain BNFs, then the BNFs are input into the speech conversion model to obtain the Mel spectrum, and finally the audio is reconstructed through the vocoder to obtain the converted speech.

[0088] To achieve streaming inference for the speech conversion model described in this application embodiment, the streaming capabilities of common deep learning neural network structures are analyzed as follows:

[0089] Linear Layer: The inference process of the linear layer is relatively simple, consisting of a matrix multiplication plus an optional bias. While the input feature dimension remains constant, the time dimension, although varying with the segment length of the input data, does not affect the inference result. In other words, the result of concatenating the data segments after inference is identical to the result of inference on the entire segment. Therefore, the linear layer in the model does not require any modification in the speech conversion model described in this application to achieve streaming inference.

[0090] Convolutional Layers: Convolutional layers have a significant impact on streaming due to their padding and receptive field settings. In common convolutional layers, to ensure the same input and output time dimensions, padding of the same length is often added at the beginning and end. If this strategy is retained during streaming inference, it's equivalent to inserting an extra blank space between two consecutively extracted first speech data points, causing data discontinuity. This discontinuity appears as bright lines on the spectrogram, and in audio testing, it manifests as stuttering at the junctions of different first speech data converted into second speech data.

[0091] Meanwhile, all data is visible throughout the entire synthesis process, whereas future data is obviously unavailable during streaming inference, so causal convolution must be used.

[0092] In this embodiment of the application, the specific strategy is to place all padding at the beginning of the data during training, while adding padding only at the beginning of the first segment during inference. Subsequent segments do not have padding added; instead, they add valid information from historical speech data preceding the first speech data. This achieves the same effect as streaming and whole-segment inference.

[0093] Transposed convolution: Transposed convolution has similar characteristics, also moving all the front and back padding to the head. However, due to the different convolution method, some modifications need to be made to the calculation of padding.

[0094] RNN: The autoregressive structure of RNN makes it naturally suitable for streaming inference. However, bidirectional RNNs such as BGRU and Bi-LSTM require a backward inference process from the tail to the head of the input data, which means they need future information for inference. Obviously, they cannot be streamed. They need to be changed to unidirectional, and inference can be performed only by the effective information of historical speech data.

[0095] With the introduction of attention mechanisms, Transformers have been widely applied to various models. To achieve streaming inference, segmented attention is needed, which essentially limits the scope of attention and requires attention masks. Unlike whole-segment attention, segmented attention masks future information, focusing only on the current segment, the current segment and the previous segment, or all historical segments.

[0096] Therefore, in the embodiments of this application, please refer to Figure 5 , Figure 5 This document illustrates a flowchart of a method for converting the first semantic information and valid information from historical speech data preceding the first speech data using the speech conversion model, as described in an embodiment of this application. The method includes the following steps: S501-S502.

[0097] S501. When the speech conversion model converts and processes the historical speech data preceding the first speech data, it obtains the second semantic information and the target speech feature information corresponding to the second semantic information and the speech factors of the target speaker, and outputs the first effective information of each convolutional layer; wherein, the second semantic information is extracted from the historical speech data preceding the first speech data;

[0098] S502. The first semantic information is sequentially input into each convolutional layer of the speech conversion model, and the first valid information corresponding to the convolutional layer is added to the header of the input data of each convolutional layer, so as to convert the first semantic information and the valid information of the historical speech data before the first speech data through the speech conversion model.

[0099] In other words, in this embodiment of the application, the historical voice data is the historical voice data preceding the first voice data, that is, the segment of historical voice data that is extracted before the first voice data and is closest to the first voice data.

[0100] Here, the second semantic information is extracted from the preceding historical speech data of the first speech data; the first semantic information is extracted from the first speech data. In other words, the second semantic information corresponds to the preceding historical speech data of the first speech data; the first semantic information corresponds to the first speech data.

[0101] Here, valid information includes at least one type of valid information, namely, first valid information.

[0102] It is important to understand that the first valid information output by each convolutional layer is the data required by that convolutional layer when processing the first semantic information of the next segment of the first speech data, and is not the processing result of the first convolutional layer processing the first speech information.

[0103] For example, the first valid information may be the tail data of a preset number of frames of input data for each convolutional layer.

[0104] It should be noted that during inference, only the first segment of the first speech data is processed. Since there is no previous historical speech data, the first semantic information of the first segment of the first speech data is sequentially input into each convolutional layer of the speech conversion model. Zeros are added to the beginning of the input data of each convolutional layer to ensure that the input and output dimensions of the convolutional layer are the same.

[0105] In this embodiment, the first semantic information is sequentially input into each convolutional layer of the speech conversion model, and the first valid information corresponding to the convolutional layer is added to the head of the input data of each convolutional layer. By adding the valid information of historical speech data, the input and output dimensions of the convolutional layer are ensured to be the same. At the same time, since there is no extra blank space inserted between the two segments of the first speech data, the problem of discontinuity of the two segments of the first speech data is not caused. There are no bright lines on the spectrogram, and in the listening test, this is specifically manifested as no stuttering at the audio splicing point.

[0106] In some embodiments, the speech conversion model can only convert the first speech data into target speech feature information of a specific speaker.

[0107] In some embodiments, the speech conversion model can convert the first speech data into target speech feature information of multiple different speaking objects; through a selection operation, the speech conversion model determines a target speaking object and converts the first semantic information into target speech feature information of the corresponding target speaking object's speech factors.

[0108] Specifically, input the first semantic information into a pre-trained speech conversion model, and the speech conversion model performs conversion processing on the effective information of the first semantic information and the historical speech data before the first speech data to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker, including:

[0109] Determine the target speech factor vector of the target speaker;

[0110] Input the first semantic information and the target speech factor vector of the target speaker into the pre-trained speech conversion model, so that the speech conversion model performs conversion processing on the first semantic information and the effective information of the historical speech data before the first speech data based on the target speech factor vector of the target speaker, and obtains target speech feature information corresponding to the first semantic information and the speech factors of the target speaker.

[0111] Specifically, the determination of the target speech factor vector of the target speaker includes:

[0112] Obtain the identification information of the target speaker;

[0113] According to the identification information of the target speaker, determine the target speech factor vector of the target speaker from the association relationship table of the pre-trained speech conversion model; wherein, the association relationship table represents the association relationship between the speaker and the speech factor vector.

[0114] By performing an input selection operation on the identification information of the target speaker, the speech conversion model determines a target speaker, which increases the flexibility of the speech conversion model and facilitates the speech conversion model to be applicable to various scenarios and develop various applications.

[0115] The identification information of the target speaker can be the number, ID, keyword, name, attribute, etc. of the target speaker. Exemplarily, for example, multiple speakers are: Jacky Cheung (ID is 001), Guo Degang (ID is 002), Song小宝 (ID is 003), Faye Wong (ID is 004).

[0116] Input 001, then determine Jacky Cheung as the target speaker; or, input Xueyou, and Jacky Cheung can also be determined as the target speaker; or, input Ge Shen, and Jacky Cheung can also be determined as the target speaker.

[0117] Please refer to Figure 6 , Figure 6The diagram shows the structure of the speech conversion model. The CBHG module, which serves as the encoder, consists of a one-dimensional convolutional filter bank, a highway network, and a bidirectional gated recurrent unit (GRU). The CBHG module has a strong ability to model sequence information and is suitable for speech conversion tasks.

[0118] The AR part acts as a decoder, comprising four modules: Prenet, GRU, linear layer, and Postnet. During each decoding process, the hidden state information from the previous output of GRU is concatenated with the result of CBHG after passing through Prenet. The concatenated result is then fed into GRU and passed through the linear layer to obtain a new result.

[0119] To achieve streaming inference, all ordinary convolutions should be replaced with causal convolutions, and padding should be added entirely to the left side of the input data, i.e., the header, without considering any future information. Simultaneously, in addition to generating inference results, the speech model should also return the cache of each convolutional layer in the model for use in the next inference. This cache extracts the last N frames of the current layer's input, where N is the receptive field size of the current layer.

[0120] Meanwhile, since CBHG uses a bidirectional GRU, it must be changed to a unidirectional one, meaning it relies solely on historical speech data for inference, without relying on future speech data. Here, to maximize its effectiveness, the number of RNN units is doubled, meaning the modified unidirectional GRU maintains the same number of parameters as the previous bidirectional GRU.

[0121] During the training phase, the data fed into the speech conversion model does not need to be modified, and the training method is the same as for ordinary models. During inference, padding is applied to the first chunk to ensure the input and output lengths are the same, and the buffer returned by the model is saved. When inferring subsequent chunks, the buffer generated in the previous step is fed into the model along with the input data to obtain new outputs. When using valid information from historical speech data, the results of segmented inference and whole-segment inference are almost identical, and the playback of the second speech data after different segment conversions is continuous without any stuttering.

[0122] To mitigate the performance degradation of streaming models compared to non-streaming models, a strategy of incorporating future information is employed. Since the final PostNet layer in the speech conversion model fine-tunes the generated Mel spectrum to improve its quality, incorporating future information into this layer is feasible. Causal convolution, completely independent of future information, essentially pads the left side of the input data with enough zeros to ensure that one input frame generates one output frame. If we need to infer n frames of future information, we move the n zeros from the left side of the input data to the right side, meaning an input length of n+1 is required to generate one output frame. In practical terms, this means incorporating n frames of future information. In this embodiment, for example, n is 1.

[0123] Based on the above analysis, please refer to Figure 7 The speech conversion model is used to convert the first semantic information and the effective information of the historical speech data before the first speech data to obtain the target speech feature information corresponding to the first semantic information and the speech factors of the target speaker; including the following steps S701-S704.

[0124] S701. Obtain the target speech factor vector of the target speaker;

[0125] S702. The encoder of the speech conversion model processes the target speech factor vector, the first semantic information, and the second effective information of the historical speech data before the first speech data to generate a target speech vector feature corresponding to the first semantic information and the target speech factor vector.

[0126] S703, Output the target speech vector features to the decoder of the speech conversion model;

[0127] S704. The decoder processes the target speech vector features and the third effective information of the historical speech data before the first speech data to obtain the target speech feature information corresponding to the first semantic information and the speech factors of the target speaking object.

[0128] Here, the target speech factor vector represents the speech factors of the target speaker, that is, it represents the speech characteristics of the target speaker, such as the timbre and tone characteristics of the target speaker when speaking.

[0129] The second valid information is the hidden state information generated by the encoder of the speech conversion model when processing historical speech data; the third valid information is the hidden state information generated by the decoder of the speech conversion model when processing historical speech data.

[0130] The hidden state information of the encoder and decoder is the output of their past processing; it is used to implement the reasoning of the inference speech conversion model. For example, in the sentence "I am thirsty and want to drink water", it is difficult to infer "drink water" based solely on "want"; however, if we combine "I am thirsty and want" with "I want to", the probability of inferring "drink water" can be greatly increased.

[0131] Based on this, the first semantic information is input into a pre-trained speech conversion model, and the speech conversion model converts the first semantic information and the effective information of the historical speech data preceding the first speech data to obtain the target speech feature information corresponding to the first semantic information and the speech factors of the target speaker. The effective information includes multiple types of effective information; in this embodiment, it specifically includes first effective information, second effective information, and third effective information. The first effective information is output by each convolutional layer when processing the previous historical speech data, and is used as padding added to the beginning of the input data of each current convolutional layer, thereby avoiding adding 0s to the beginning of the input data of each convolutional layer and solving the stuttering problem after segmented real-time speech conversion. The second effective information is the hidden state information of the encoder of the speech conversion model when processing historical speech data. The third effective information is the hidden state information of the decoder of the speech conversion model when processing historical speech data; here, historical speech data is not just the previous historical speech data. The speech conversion model is a streaming inference model; the second effective information is used to improve the accuracy of the encoder's inference results, and the third effective information is used to improve the accuracy of the decoder's inference results.

[0132] Based on this, the speech conversion model improves the real-time speech conversion effect in several ways.

[0133] In some embodiments, please refer to Figure 8 The encoder of the speech conversion model generates target speech vector features corresponding to the first semantic information and the target speech factor vector based on the target speech factor vector, the first semantic information, and the second effective information of the historical speech data before the first speech data, including the following steps S801-S802.

[0134] S801. Obtain the hidden state information generated by the encoder of the speech conversion model during the processing of historical speech data as the second valid information;

[0135] S802. The encoder of the speech conversion model processes the target speech factor vector, the second effective information and the first semantic information to generate target speech vector features corresponding to the first semantic information and the target speech factor vector.

[0136] In other words, historical speech data not only plays a role in the convolutional layer, but also in the reasoning process, enabling the encoder to refer to historical speech data for reasoning, thereby making the reasoning results more accurate and improving the quality of speech conversion.

[0137] In some embodiments, please refer to Figure 9 The decoder processes the target speech vector features and the third effective information of the historical speech data before the first speech data to obtain the target speech feature information corresponding to the first semantic information and the speech factors of the target speaker, including the following steps S901-S902.

[0138] S901. Obtain the hidden state information generated by the decoder of the speech conversion model during the processing of historical speech data as the third valid information;

[0139] S902. The target speech vector features and the third effective information are processed by the decoder of the speech conversion model to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker.

[0140] In other words, historical speech data plays a role in reasoning. It is not only through the encoder referencing historical speech data for reasoning, but also through the decoder referencing historical speech data for reasoning, thereby making the reasoning results more accurate and improving the quality of speech conversion.

[0141] Therefore, the conversion process of the first semantic information and the effective information of the historical speech data preceding the first speech data using the speech conversion model specifically includes:

[0142] The speech conversion model uses convolutional layers, and / or encoders, and / or decoders to convert the first semantic information and valid information from historical speech data preceding the first speech data.

[0143] In some embodiments, the target speech vector features and third effective information are processed by the decoder of the speech conversion model to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker, including:

[0144] The decoder of the speech conversion model processes the target speech vector features and the third effective information to obtain the first speech feature information corresponding to the first semantic information and the speech factor vector;

[0145] A preset data frame 0 is added to the end of the first speech feature information, and the first speech feature information after adding 0 is input into the output convolutional layer of the decoder, so that the first speech feature information after adding 0 is processed by the decoder to obtain the target speech feature information.

[0146] Here, by padding the left side of the actual input to the decoder with zeros when the decoder processes the target speech vector features, a strategy of looking for future information is adopted to reduce the performance degradation of the streaming model compared to the non-streaming model, and further improve the quality of real-time speech conversion.

[0147] In step S104, the target speech feature information is reconstructed to obtain second speech data after the conversion of the first speech data, including:

[0148] The target speech feature information is reconstructed using a vocoder to obtain the second speech data after the first speech data is converted.

[0149] In other words, the Mel spectrum corresponding to the first semantic information and the speech factors of the target speaker is synthesized by a vocoder into second speech data characterized by acoustic features, which is then used as the audio converted from the first speech data.

[0150] After reconstructing the target speech feature information to obtain the second speech data after converting the first speech data, the speech conversion method further includes:

[0151] The playback order of the second voice data is determined based on the capture time of the first voice data before the conversion of the second voice data.

[0152] The second voice data is played in the determined playback order.

[0153] In other words, multiple first audio data segments are sequentially captured during real-time recording, and after conversion, multiple second audio data segments are obtained. These second audio data segments are then played sequentially according to the capture time of the first audio data segments before conversion, thereby achieving the effect of converting and playing the entire segment as a whole. Furthermore, since the valid information from the previous segment of first audio data participates in the conversion process of the previous segment, the stuttering problem when playing different segments continuously is solved.

[0154] Here, multiple second voice data can be stored in the server according to the playback order, and the multiple arranged second voice data can be sent to the terminal device for playback according to the preset sending rules; or, after converting to second voice data, they can be sent to the terminal device in real time, and the terminal device can play them sequentially according to the determined playback order.

[0155] The second voice data is played in the determined playback order, that is, after the previous second voice data is played, the next second voice data is played immediately, so as to achieve the effect of converting the voice data of the source speaker recorded in real time into the second voice data of the target speaker in real time without delay.

[0156] Based on the real-time speech conversion method described in the embodiments of this application, streaming inference can be achieved at an RTF of less than 0.5, and the conversion effect does not decrease significantly.

[0157] For specific experimental results of this application, please refer to Table 1.

[0158] Table 1

[0159] RTF 10700 m1 VC 0.033 0.049

[0160] Here, RTF stands for Real Time Factor; VC guarantees the speech conversion method described in this application embodiment, or the real-time conversion model (including speech recognition model and speech conversion model) in this application embodiment; the table guarantees the real-time rate of the speech conversion method described in this application embodiment on two CPUs (m1 and i7-10700); analyzing Table 1, the real-time speech conversion method described in this application embodiment can achieve streaming inference with an RTF of less than 0.5, ensuring the real-time performance of speech conversion.

[0161] Based on the same inventive concept, this application also provides a real-time speech conversion device corresponding to the real-time speech conversion method in the above embodiments. Since the principle of the real-time speech conversion device in this application is similar to that of the real-time speech conversion method in the above embodiments of this application, the implementation of the real-time speech conversion device can refer to the implementation of the aforementioned real-time speech conversion method, and the repeated parts will not be described again.

[0162] Reference Figure 10 As shown, Figure 10 A schematic diagram of the structure of the real-time speech conversion device provided in an embodiment of this application is shown; the device includes:

[0163] The extraction module 1001 is used to extract the first voice data that meets the voice segmentation conditions from the voice data of the source speaker recorded in real time.

[0164] Extraction module 1002 is used to process the first speech data and extract the first semantic information of the first speech data;

[0165] The conversion module 1003 is used to input the first semantic information into a pre-trained speech conversion model, and to convert the first semantic information and the effective information of the historical speech data preceding the first speech data through the speech conversion model to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker; wherein, the effective information is information that affects the speech conversion of the first semantic information.

[0166] The reconstruction module 1004 is used to reconstruct the target speech feature information to obtain the second speech data after the first speech data is converted.

[0167] This application provides a real-time speech conversion device that inputs real-time recorded speech into a speech conversion model in segments, rather than as a whole segment, thereby reducing latency issues in real-time applications. Then, the device uses a pre-trained speech conversion model to process the first semantic information and valid information from historical speech data preceding the first speech data to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker. Finally, the target speech feature information is reconstructed to obtain second speech data converted from the first speech data. The use of valid information from historical speech data reduces the impact of speech segmentation on the continuity of the entire speech segment. The speech conversion model can perform inference in a continuous, streaming manner, and the resulting segmented second speech data can be played continuously, smoothly, and with high quality, meeting the high-performance requirements of real-time applications.

[0168] In some embodiments, the real-time speech conversion device further includes:

[0169] The determining module is used to determine the playback order of the second voice data based on the truncation time of the first voice data before the conversion of the second voice data after reconstructing the target voice feature information to obtain the second voice data after the first voice data is converted.

[0170] The playback module is used to play the second voice data in the determined playback order.

[0171] In some embodiments, in the real-time speech conversion device, when the conversion module performs conversion processing on the first semantic information and the effective information of historical speech data preceding the first speech data through the speech conversion model, it is specifically used for:

[0172] When the speech conversion model processes the second semantic information of the historical speech data preceding the first speech data to obtain the target speech feature information corresponding to the second semantic information and the speech factors of the target speaker, the first effective information output by each convolutional layer is obtained; wherein, the second semantic information is extracted from the historical speech data preceding the first speech data;

[0173] The first semantic information is sequentially input into each convolutional layer of the speech conversion model, and the first valid information corresponding to the convolutional layer is added to the header of the input data of each convolutional layer, so as to convert the first semantic information and the valid information of the historical speech data before the first speech data through the speech conversion model.

[0174] In some embodiments, in the real-time speech conversion device, the historical speech data is the historical speech data preceding the first speech data.

[0175] In some embodiments, in the real-time speech conversion device, the conversion module, when inputting the first semantic information into a pre-trained speech conversion model and converting the first semantic information and the effective information of the historical speech data preceding the first speech data through the speech conversion model to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker, is specifically used for:

[0176] Determine the target speech factor vector of the target speaker;

[0177] The first semantic information and the target speech factor vector of the target speaking object are input into a pre-trained speech conversion model, so that the speech conversion model converts the first semantic information and the effective information of the historical speech data before the first speech data based on the target speech factor vector of the target speaking object, and obtains the target speech feature information corresponding to the first semantic information and the speech factors of the target speaking object.

[0178] In some embodiments, in the real-time speech conversion device, the conversion module, when determining the target speech factor vector of the target speaker, is specifically used for:

[0179] Obtain the identification information of the target speaker;

[0180] Based on the identification information of the target speaker, the target speech factor vector of the target speaker is determined from the association table of the pre-trained speech conversion model; wherein the association table represents the association between the speaker and the speech factor vector.

[0181] In some embodiments, in the real-time speech conversion device, the conversion module, when converting the first semantic information and the effective information of the historical speech data preceding the first speech data through the speech conversion model to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker, is specifically used for:

[0182] Obtain the target speech factor vector of the target speaker;

[0183] The encoder of the speech conversion model processes the target speech factor vector and the first semantic information, as well as the second effective information of the historical speech data before the first speech data, to generate target speech vector features corresponding to the first semantic information and the target speech factor vector.

[0184] The target speech vector features are output to the decoder of the speech conversion model;

[0185] The decoder processes the target speech vector features and the third effective information of the historical speech data preceding the first speech data to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker.

[0186] In some embodiments, in the real-time speech conversion device, the conversion module, when generating target speech vector features corresponding to the first semantic information and the target speech factor vector through the encoder of the speech conversion model based on the target speech factor vector, the first semantic information, and the second valid information of historical speech data preceding the first speech data, is specifically used for:

[0187] The latent state information generated by the encoder of the speech conversion model during the processing of historical speech data is used as the second effective information;

[0188] The encoder of the speech conversion model processes the target speech factor vector, the second effective information, and the first semantic information to generate target speech vector features corresponding to the first semantic information and the target speech factor vector.

[0189] In some embodiments, in the real-time speech conversion device, when the conversion module processes the target speech vector features and the third valid information of historical speech data prior to the first speech data through the decoder to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker, it is specifically used for:

[0190] The hidden state information generated by the decoder of the speech conversion model during the processing of historical speech data is used as the third effective information;

[0191] The decoder of the speech conversion model processes the target speech vector features and the third effective information to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker.

[0192] In some embodiments, in the real-time speech conversion device, the conversion module, when processing the target speech vector features and the third effective information through the decoder of the speech conversion model to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker, is specifically used for:

[0193] The decoder of the speech conversion model processes the target speech vector features and the third effective information to obtain the first speech feature information corresponding to the first semantic information and the speech factor vector;

[0194] A preset data frame 0 is added to the end of the first speech feature information, and the first speech feature information after adding 0 is input into the output convolutional layer of the decoder, so that the first speech feature information after adding 0 is processed by the decoder to obtain the target speech feature information.

[0195] In some embodiments, the first speech data that satisfies the speech segmentation conditions in the real-time speech conversion device includes at least one of the following:

[0196] Record the first audio data in real time when the preset segment duration is reached;

[0197] Record the first audio data in real time when the frame rate reaches the preset frame threshold;

[0198] The first voice data captured upon receiving a preset segmentation command;

[0199] The first audio data captured upon completion of recording.

[0200] In some embodiments, in the real-time speech conversion device, the extraction module, when processing the first speech data and extracting the first semantic information of the first speech data, is specifically used for:

[0201] Obtain a pre-trained speech recognition model;

[0202] The first speech data is input into the speech recognition model, and the speech recognition model is used to decouple the timbre of the first speech data, remove noise from the first speech data, and extract the semantic information of the first speech data.

[0203] In some embodiments, the real-time speech conversion device further includes a training module for obtaining a pre-trained speech recognition model;

[0204] Acquire the third speech data of the target speaker and extract the target speech feature information from the third speech data;

[0205] The semantic information of the third speech data is extracted using the pre-trained speech recognition model.

[0206] The semantic information of the third speech data and the target speech feature information are input into the pre-established speech conversion model, and the speech conversion model is trained until the speech conversion model meets the training completion conditions.

[0207] In some embodiments, the reconstruction module in the real-time speech conversion device, when reconstructing the target speech feature information to obtain the second speech data after converting the first speech data, is specifically used for:

[0208] The target speech feature information is reconstructed using a vocoder to obtain the second speech data after the first speech data is converted.

[0209] Based on the same inventive concept, this application also provides an electronic device corresponding to the real-time speech conversion method in the above embodiments. Since the principle of solving the problem by the electronic device in this application is similar to that of the real-time speech conversion method in the above embodiments of this application, the implementation of the electronic device can refer to the implementation of the aforementioned real-time speech conversion method, and the repeated parts will not be described again.

[0210] Reference Figure 11 As shown, an electronic device 1100 includes: a processor 1102, a memory 1101, and a bus. The memory 1101 stores machine-readable instructions executable by the processor 1102. When the electronic device is running, the processor 1102 communicates with the memory 1101 via the bus. When the machine-readable instructions are executed by the processor 1102, the following steps of a real-time speech conversion method are performed:

[0211] Extract the first voice data that meets the voice segmentation conditions from the real-time recorded voice data of the source speaker;

[0212] Process the first speech data and extract the first semantic information from the first speech data;

[0213] The first semantic information is input into a pre-trained speech conversion model, and the effective information of the first semantic information and the historical speech data preceding the first speech data is converted by the speech conversion model to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker; wherein, the effective information is the information that affects the speech conversion of the first semantic information.

[0214] The target speech feature information is reconstructed to obtain the second speech data after the first speech data is converted.

[0215] In some embodiments, when the machine-readable instructions are executed by the processor, the following steps of the real-time speech conversion method are also performed:

[0216] After reconstructing the target speech feature information to obtain the second speech data after the first speech data is converted, the playback order of the second speech data is determined according to the extraction time of the first speech data before the second speech data is converted.

[0217] The second voice data is played in the determined playback order.

[0218] In some embodiments, in the real-time speech conversion device, when the conversion module performs conversion processing on the first semantic information and the effective information of historical speech data preceding the first speech data through the speech conversion model, the processor is specifically configured to perform the following steps:

[0219] When the speech conversion model processes the second semantic information of the historical speech data preceding the first speech data to obtain the target speech feature information corresponding to the second semantic information and the speech factors of the target speaker, the first effective information output by each convolutional layer is obtained; wherein, the second semantic information is extracted from the historical speech data preceding the first speech data;

[0220] The first semantic information is sequentially input into each convolutional layer of the speech conversion model, and the first valid information corresponding to the convolutional layer is added to the header of the input data of each convolutional layer, so as to convert the first semantic information and the valid information of the historical speech data before the first speech data through the speech conversion model.

[0221] In some embodiments, the historical voice data is the previous segment of historical voice data of the first voice data.

[0222] In some embodiments, when the first semantic information is input into a pre-trained speech conversion model, and the speech conversion model is used to convert the first semantic information and the effective information of the historical speech data preceding the first speech data to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker, the processor is specifically used to perform the following steps:

[0223] Determine the target speech factor vector of the target speaker;

[0224] The first semantic information and the target speech factor vector of the target speaking object are input into a pre-trained speech conversion model, so that the speech conversion model converts the first semantic information and the effective information of the historical speech data before the first speech data based on the target speech factor vector of the target speaking object, and obtains the target speech feature information corresponding to the first semantic information and the speech factors of the target speaking object.

[0225] In some embodiments, when determining the target speech factor vector of the target speaking object, the processor is specifically configured to perform the following steps:

[0226] Obtain the identification information of the target speaker;

[0227] Based on the identification information of the target speaker, the target speech factor vector of the target speaker is determined from the association table of the pre-trained speech conversion model; wherein the association table represents the association between the speaker and the speech factor vector.

[0228] In some embodiments, when converting the first semantic information and the effective information of historical speech data preceding the first speech data using the speech conversion model to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker, the processor is specifically used to perform the following steps:

[0229] Obtain the target speech factor vector of the target speaker;

[0230] The encoder of the speech conversion model processes the target speech factor vector and the first semantic information, as well as the second effective information of the historical speech data before the first speech data, to generate target speech vector features corresponding to the first semantic information and the target speech factor vector.

[0231] The target speech vector features are output to the decoder of the speech conversion model;

[0232] The decoder processes the target speech vector features and the third effective information of the historical speech data preceding the first speech data to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker.

[0233] In some embodiments, when the encoder of the speech conversion model generates target speech vector features corresponding to the first semantic information and the target speech factor vector based on the target speech factor vector, the first semantic information, and the second valid information of historical speech data preceding the first speech data, the processor is specifically configured to perform the following steps:

[0234] The latent state information generated by the encoder of the speech conversion model during the processing of historical speech data is used as the second effective information;

[0235] The encoder of the speech conversion model processes the target speech factor vector, the second effective information, and the first semantic information to generate target speech vector features corresponding to the first semantic information and the target speech factor vector.

[0236] In some embodiments, when the decoder processes the target speech vector features and the third valid information of historical speech data prior to the first speech data to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker, the processor is specifically used to perform the following steps:

[0237] The hidden state information generated by the decoder of the speech conversion model during the processing of historical speech data is used as the third effective information;

[0238] The decoder of the speech conversion model processes the target speech vector features and the third effective information to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker.

[0239] In some embodiments, when the target speech vector features and the third effective information are processed by the decoder of the speech conversion model to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker, the processor is specifically used to perform the following steps:

[0240] The decoder of the speech conversion model processes the target speech vector features and the third effective information to obtain the first speech feature information corresponding to the first semantic information and the speech factor vector;

[0241] A preset data frame 0 is added to the end of the first speech feature information, and the first speech feature information after adding 0 is input into the output convolutional layer of the decoder, so that the first speech feature information after adding 0 is processed by the decoder to obtain the target speech feature information.

[0242] In some embodiments, the first speech data that satisfies the speech segmentation conditions includes at least one of the following:

[0243] Record the first audio data in real time when the preset segment duration is reached;

[0244] Record the first audio data in real time when the frame rate reaches the preset frame threshold;

[0245] The first voice data captured upon receiving a preset segmentation command;

[0246] The first audio data captured upon completion of recording.

[0247] In some embodiments, when processing the first speech data and extracting the first semantic information of the first speech data, the processor is specifically configured to perform the following steps:

[0248] Obtain a pre-trained speech recognition model;

[0249] The first speech data is input into the speech recognition model, and the speech recognition model is used to decouple the timbre of the first speech data, remove noise from the first speech data, and extract the semantic information of the first speech data.

[0250] In some embodiments, when the machine-readable instructions are executed by the processor, the following steps of the real-time speech conversion method are also performed:

[0251] Obtain a pre-trained speech recognition model;

[0252] Acquire the third speech data of the target speaker and extract the target speech feature information from the third speech data;

[0253] The semantic information of the third speech data is extracted using the pre-trained speech recognition model.

[0254] The semantic information of the third speech data and the target speech feature information are input into the pre-established speech conversion model, and the speech conversion model is trained until the speech conversion model meets the training completion conditions.

[0255] In some embodiments, when the machine-readable instructions are executed by the processor, the following steps of the real-time speech conversion method are also performed:

[0256] When reconstructing the target speech feature information to obtain the second speech data after converting the first speech data, it is specifically used for:

[0257] The target speech feature information is reconstructed using a vocoder to obtain the second speech data after the first speech data is converted.

[0258] Based on the same inventive concept, this application also provides a storage medium corresponding to the real-time speech conversion method in the above embodiments. Since the principle of the storage medium in this application is similar to that of the real-time speech conversion method in the above embodiments, the implementation of the storage medium can refer to the implementation of the aforementioned real-time speech conversion method, and the repeated parts will not be described again.

[0259] A computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor, the processor performing the following steps:

[0260] Extract the first voice data that meets the voice segmentation conditions from the real-time recorded voice data of the source speaker;

[0261] Process the first speech data and extract the first semantic information from the first speech data;

[0262] The first semantic information is input into a pre-trained speech conversion model, and the effective information of the first semantic information and the historical speech data preceding the first speech data is converted by the speech conversion model to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker; wherein, the effective information is the information that affects the speech conversion of the first semantic information.

[0263] The target speech feature information is reconstructed to obtain the second speech data after the first speech data is converted.

[0264] In some embodiments, when the machine-readable instructions are executed by the processor, the following steps of the real-time speech conversion method are also performed:

[0265] After reconstructing the target speech feature information to obtain the second speech data after the first speech data is converted, the playback order of the second speech data is determined according to the extraction time of the first speech data before the second speech data is converted.

[0266] The second voice data is played in the determined playback order.

[0267] In some embodiments, in the real-time speech conversion device, when the conversion module performs conversion processing on the first semantic information and the effective information of historical speech data preceding the first speech data through the speech conversion model, the processor is specifically configured to perform the following steps:

[0268] When the speech conversion model processes the second semantic information of the historical speech data preceding the first speech data to obtain the target speech feature information corresponding to the second semantic information and the speech factors of the target speaker, the first effective information output by each convolutional layer is obtained; wherein, the second semantic information is extracted from the historical speech data preceding the first speech data;

[0269] The first semantic information is sequentially input into each convolutional layer of the speech conversion model, and the first valid information corresponding to the convolutional layer is added to the header of the input data of each convolutional layer, so as to convert the first semantic information and the valid information of the historical speech data before the first speech data through the speech conversion model.

[0270] In some embodiments, the historical voice data is the previous segment of historical voice data of the first voice data.

[0271] In some embodiments, when the first semantic information is input into a pre-trained speech conversion model, and the speech conversion model is used to convert the first semantic information and the effective information of the historical speech data preceding the first speech data to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker, the processor is specifically used to perform the following steps:

[0272] Determine the target speech factor vector of the target speaker;

[0273] The first semantic information and the target speech factor vector of the target speaking object are input into a pre-trained speech conversion model, so that the speech conversion model converts the first semantic information and the effective information of the historical speech data before the first speech data based on the target speech factor vector of the target speaking object, and obtains the target speech feature information corresponding to the first semantic information and the speech factors of the target speaking object.

[0274] In some embodiments, when determining the target speech factor vector of the target speaking object, the processor is specifically configured to perform the following steps:

[0275] Obtain the identification information of the target speaker;

[0276] Based on the identification information of the target speaker, the target speech factor vector of the target speaker is determined from the association table of the pre-trained speech conversion model; wherein the association table represents the association between the speaker and the speech factor vector.

[0277] In some embodiments, when converting the first semantic information and the effective information of historical speech data preceding the first speech data using the speech conversion model to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker, the processor is specifically used to perform the following steps:

[0278] Obtain the target speech factor vector of the target speaker;

[0279] The encoder of the speech conversion model processes the target speech factor vector and the first semantic information, as well as the second effective information of the historical speech data before the first speech data, to generate target speech vector features corresponding to the first semantic information and the target speech factor vector.

[0280] The target speech vector features are output to the decoder of the speech conversion model;

[0281] The decoder processes the target speech vector features and the third effective information of the historical speech data preceding the first speech data to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker.

[0282] In some embodiments, when the encoder of the speech conversion model generates target speech vector features corresponding to the first semantic information and the target speech factor vector based on the target speech factor vector, the first semantic information, and the second valid information of historical speech data preceding the first speech data, the processor is specifically configured to perform the following steps:

[0283] The latent state information generated by the encoder of the speech conversion model during the processing of historical speech data is used as the second effective information;

[0284] The encoder of the speech conversion model processes the target speech factor vector, the second effective information, and the first semantic information to generate target speech vector features corresponding to the first semantic information and the target speech factor vector.

[0285] In some embodiments, when the decoder processes the target speech vector features and the third valid information of historical speech data prior to the first speech data to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker, the processor is specifically used to perform the following steps:

[0286] The hidden state information generated by the decoder of the speech conversion model during the processing of historical speech data is used as the third effective information;

[0287] The decoder of the speech conversion model processes the target speech vector features and the third effective information to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker.

[0288] In some embodiments, when the target speech vector features and the third effective information are processed by the decoder of the speech conversion model to obtain target speech feature information corresponding to the first semantic information and the speech factors of the target speaker, the processor is specifically used to perform the following steps:

[0289] The decoder of the speech conversion model processes the target speech vector features and the third effective information to obtain the first speech feature information corresponding to the first semantic information and the speech factor vector;

[0290] A preset data frame 0 is added to the end of the first speech feature information, and the first speech feature information after adding 0 is input into the output convolutional layer of the decoder, so that the first speech feature information after adding 0 is processed by the decoder to obtain the target speech feature information.

[0291] In some embodiments, the first speech data that satisfies the speech segmentation conditions includes at least one of the following:

[0292] Record the first audio data in real time when the preset segment duration is reached;

[0293] Record the first audio data in real time when the frame rate reaches the preset frame threshold;

[0294] The first voice data captured upon receiving a preset segmentation command;

[0295] The first audio data captured upon completion of recording.

[0296] In some embodiments, when processing the first speech data and extracting the first semantic information of the first speech data, the processor is specifically configured to perform the following steps:

[0297] Obtain a pre-trained speech recognition model;

[0298] The first speech data is input into the speech recognition model, and the speech recognition model is used to decouple the timbre of the first speech data, remove noise from the first speech data, and extract the semantic information of the first speech data.

[0299] In some embodiments, when the machine-readable instructions are executed by the processor, the following steps of the real-time speech conversion method are also performed:

[0300] Obtain a pre-trained speech recognition model;

[0301] Acquire the third speech data of the target speaker and extract the target speech feature information from the third speech data;

[0302] The semantic information of the third speech data is extracted using the pre-trained speech recognition model.

[0303] The semantic information of the third speech data and the target speech feature information are input into the pre-established speech conversion model, and the speech conversion model is trained until the speech conversion model meets the training completion conditions.

[0304] In some embodiments, when the machine-readable instructions are executed by the processor, the following steps of the real-time speech conversion method are also performed:

[0305] When reconstructing the target speech feature information to obtain the second speech data after converting the first speech data, it is specifically used for:

[0306] The target speech feature information is reconstructed using a vocoder to obtain the second speech data after the first speech data is converted.

[0307] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0308] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0309] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0310] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a platform server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0311] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A real-time voice conversion method, characterized by, The method comprises the following steps: cutting out first voice data meeting a voice segmentation condition from real-time recorded voice data of a source speaker; processing the first voice data to extract first semantic information of the first voice data; inputting the first semantic information into a pre-trained voice conversion model, and performing conversion processing on effective information of historical voice data before the first voice data by the voice conversion model to obtain target voice feature information corresponding to the first semantic information and a voice factor of a target speaker; reconstructing the target voice feature information to obtain second voice data converted from the first voice data; the conversion processing on the effective information of the historical voice data before the first voice data by the voice conversion model comprises: obtaining first effective information output by each convolution layer when the voice conversion model converts second semantic information of the historical voice data before the first voice data to obtain target voice feature information corresponding to the second semantic information and a voice factor of a target speaker; wherein the second semantic information is extracted from the historical voice data before the first voice data; inputting the first semantic information into each convolution layer of the voice conversion model in sequence, and adding the first effective information corresponding to each convolution layer to the head of the input data of each convolution layer, so as to perform conversion processing on the effective information of the historical voice data before the first voice data by the voice conversion model.

2. The real-time voice conversion method of claim 1, wherein, After the target voice feature information is reconstructed to obtain the second voice data converted from the first voice data, the method further comprises: determining a playing order of the second voice data according to the cutting time of the first voice data before conversion of the second voice data; playing the second voice data according to the determined playing order.

3. The real-time voice conversion method of claim 1, wherein: the historical voice data is historical voice data before the first voice data.

4. The real-time voice conversion method of claim 1, wherein, The conversion processing on the effective information of the historical voice data before the first voice data by the voice conversion model based on the target voice factor vector of the target speaker comprises: determining a target voice factor vector of the target speaker; inputting the first semantic information and the target voice factor vector of the target speaker into the pre-trained voice conversion model, so that the voice conversion model performs conversion processing on the effective information of the historical voice data before the first voice data based on the target voice factor vector of the target speaker to obtain the target voice feature information corresponding to the first semantic information and the voice factor of the target speaker.

5. The method of claim 4, wherein, The determination of the target voice factor vector of the target speaker comprises: obtaining identification information of the target speaker; According to the identification information of the target speaker, a target voice factor vector of the target speaker is determined from an association table of a pre-trained voice conversion model, wherein the association table represents an association between a speaker and a voice factor vector.

6. The real-time voice conversion method of claim 1, wherein, The voice conversion model is used to convert the first semantic information and the effective information of the historical voice data before the first voice data to obtain target voice feature information corresponding to the first semantic information and the voice factor of the target speaker. A target voice factor vector of a target speaker is obtained. The target voice factor vector and the second effective information of the historical voice data before the first voice data are processed by the encoder of the voice conversion model to generate a target voice vector feature corresponding to the first semantic information and the target voice factor vector. The target voice vector feature is output to the decoder of the voice conversion model. The target voice vector feature and the third effective information of the historical voice data before the first voice data are processed by the decoder to obtain target voice feature information corresponding to the first semantic information and the voice factor of the target speaker.

7. The real-time voice conversion method of claim 6, wherein, The target voice factor vector and the second effective information of the historical voice data before the first voice data are processed by the encoder of the voice conversion model to generate a target voice vector feature corresponding to the first semantic information and the target voice factor vector, including: The hidden state information generated during the process of the encoder of the voice conversion model processing the historical voice data is obtained as the second effective information. The target voice factor vector, the second effective information, and the first semantic information are processed by the encoder of the voice conversion model to generate a target voice vector feature corresponding to the first semantic information and the target voice factor vector.

8. The real-time voice conversion method of claim 6, wherein, The target voice vector feature and the third effective information of the historical voice data before the first voice data are processed by the decoder to obtain target voice feature information corresponding to the first semantic information and the voice factor of the target speaker, including: The hidden state information generated during the process of the decoder of the voice conversion model processing the historical voice data is obtained as the third effective information. The target voice vector feature and the third effective information are processed by the decoder of the voice conversion model to obtain target voice feature information corresponding to the first semantic information and the voice factor of the target speaker.

9. The real-time voice conversion method of claim 8, wherein, The target voice vector feature and the third effective information are processed by the decoder of the voice conversion model to obtain target voice feature information corresponding to the first semantic information and the voice factor of the target speaker, including: The target voice vector feature and the third effective information are processed by the decoder of the voice conversion model to obtain target voice feature information corresponding to the first semantic information and the voice factor of the target speaker. The tail of the first voice feature information is added with 0 of a preset data frame, and the first voice feature information after adding 0 is input to the output convolution layer of the decoder to process the first voice feature information after adding 0 by the decoder to obtain target voice feature information.

10. The real-time voice conversion method of claim 1, wherein, The first voice data satisfying the voice segmentation condition at least includes one of the following: The first voice data recorded in real time reaches a preset segment length; The first voice data recorded in real time reaches a preset frame number threshold; The first voice data intercepted when a preset segment instruction is received; The first voice data intercepted when recording is completed.

11. The real-time voice conversion method of claim 1, wherein, processing the first voice data to extract first semantic information of the first voice data, comprising: obtaining a pre-trained voice recognition model; inputting the first voice data into the voice recognition model, and decoupling the tone of the first voice data through the voice recognition model, removing noise in the first voice data, and extracting semantic information of the first voice data.

12. The real-time voice conversion method of claim 1, wherein, The voice conversion model is trained in the following way: obtaining a pre-trained voice recognition model; obtaining third voice data of a target speaker and extracting target voice feature information of the third voice data; extracting third voice data semantic information through the pre-trained voice recognition model; inputting the semantic information and the target voice feature information of the third voice data into a pre-established voice conversion model, and training the voice conversion model until the voice conversion model meets the training completion condition.

13. The real-time voice conversion method of claim 1, wherein, reconstructing the target voice feature information to obtain second voice data converted from the first voice data, comprising: reconstructing the target voice feature information through a vocoder to obtain second voice data converted from the first voice data.

14. A real-time voice conversion apparatus, characterized by comprising: comprising: the intercepting module is configured to intercept first voice data meeting voice segment conditions from voice data of a source speaker recorded in real time; the extraction module is configured to process the first voice data to extract first semantic information of the first voice data; the conversion module is configured to input the first semantic information into a pre-trained voice conversion model, and convert effective information of historical voice data before the first semantic information and the first voice data through the voice conversion model to obtain target voice feature information corresponding to the first semantic information and voice factors of a target speaker; wherein the effective information is information affecting voice conversion of the first semantic information; the reconstruction module is configured to reconstruct the target voice feature information to obtain second voice data converted from the first voice data; When the conversion module converts the effective information of the historical voice data before the first semantic information and the first voice data through the voice conversion model, the processor is specifically configured to perform the following steps: obtain second semantic information of a historical voice data segment before the voice conversion model converts the first voice data, to obtain first effective information output by each volume layer corresponding to target voice feature information of the target speaker and the second semantic information; wherein the second semantic information is extracted from the historical voice data segment before the first voice data; The first semantic information is sequentially input to each convolution layer of the speech conversion model, and first effective information corresponding to each convolution layer is added to the head of the input data of each convolution layer, so as to convert and process the first semantic information and the effective information of the previous historical speech data of the first speech data by the speech conversion model.

15. An electronic device, comprising: Comprise: A processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, the machine readable instructions are executed by the processor to execute the steps of the real-time speech conversion method in any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to execute the steps of the real-time speech conversion method in any one of claims 1 to 13.

Images