Method for Changing Voice in Real-Time, Terminal Device, and Storage Medium

Abstract

The present invention is applicable to the field of audio processing, and discloses a method for voice conversion of real-time speech, a terminal device, and a storage medium. The method for voice conversion of real-time speech includes: generating original speech data according to real-time conversation audio, and determining a conditional feature, a diversity feature, and a padding data mask according to the original speech data; determining a first tensor information according to the conditional feature, the diversity feature, and the padding data mask, and determining a speaker embedding vector according to the original speech data; determining a second tensor information according to the first tensor information, the speaker embedding vector, and the padding data mask; generating a target timbre audio according to the second tensor information, the speaker embedding vector, and the pitch frequency of the original speech data. The present invention significantly improves the reconstruction accuracy of the original timbre features in the voice conversion process, enables the generated speech to reach the level of a real person in perception dimensions such as timbre similarity and intonation naturalness, and improves the authenticity of voice conversion of real-time speech.

Description

Technical Field

[0001] The present invention belongs to the field of audio processing, and particularly relates to a method for changing the voice of real-time speech, a terminal device, and a storage medium. Background Art

[0002] With the development of technology, the number of Internet users is increasing day by day. Through market research and user feedback in the past year, it can be known that people have a strong demand for voice change, and the market is also very promising. Therefore, the quality of voice change is extremely important to us.

[0003] Traditional real-time voice voice-changing technologies usually extract voice features according to a single model. This method has a relatively rough embedding process for the voice color features of the speaker, making it difficult to achieve flexible voice color conversion and diversity expansion, and lacking optimization processing of tensor information, thus resulting in poor authenticity of real-time voice voice change. A new technical means is needed to solve the above technical problems. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method for changing the voice of real-time speech, a terminal device, and a storage medium, which can solve the problem of poor authenticity of real-time voice voice change in related technologies.

[0005] The first aspect of the present invention provides a method for changing the voice of real-time speech, including:

[0006] Generating original voice data according to real-time conversation audio, and determining conditional features, diversity features, and padding data masks according to the original voice data;

[0007] Determining first tensor information according to the conditional features, diversity features, and padding data masks, and determining a speaker embedding vector according to the original voice data;

[0008] Determining second tensor information according to the first tensor information, the speaker embedding vector, and the padding data mask;

[0009] Generating a target voice color audio according to the second tensor information, the speaker embedding vector, and the pitch frequency of the original voice data.

[0010] Optionally, in the first implementation manner of the first aspect of the present invention, the step of determining first tensor information according to the conditional features, diversity features, and padding data masks, and determining a speaker embedding vector according to the original voice data includes:

[0011] Fusing the conditional features and the diversity features to generate an intermediate tensor;

[0012] Performing a masking operation on the intermediate tensor with the padding data mask to obtain the first tensor information;

[0013] Map the original speech data through a pre-trained speaker embedding layer to obtain the 128-dimensional speaker embedding vector.

[0014] Optionally, in the second implementation manner of the first aspect of the present invention, the step of determining the second tensor information according to the first tensor information, the speaker embedding vector, and the padding data mask includes:

[0015] Perform cross-attention calculation on the first tensor information and the speaker embedding vector to generate a fusion feature;

[0016] Apply a temporal alignment operation of the padding data mask to the fusion feature to obtain an aligned feature;

[0017] Upsample the aligned feature through a streaming decoder to obtain the second tensor information.

[0018] Optionally, in the third implementation manner of the first aspect of the present invention, the step of generating the target timbre audio according to the second tensor information, the speaker embedding vector, and the pitch frequency of the original speech data includes:

[0019] Perform phase alignment on the second tensor information and the pitch frequency of the original speech data to obtain an aligned spectral feature;

[0020] Input the spectral feature and the speaker embedding vector into a HiFi-GAN vocoder to obtain waveform data of the target timbre;

[0021] Generate the target timbre audio according to the waveform data.

[0022] Optionally, in the fourth implementation manner of the first aspect of the present invention, the step of generating the target timbre audio according to the waveform data includes:

[0023] Generate an intermediate audio according to the waveform data;

[0024] Perform audio time alignment and audio splicing on the intermediate audio according to the SOLA algorithm to obtain the target timbre audio.

[0025] Optionally, in the fifth implementation manner of the first aspect of the present invention, the step of performing audio time alignment and audio splicing on the intermediate audio according to the SOLA algorithm to obtain the target timbre audio includes:

[0026] Segment the intermediate audio to obtain a set of audio segments.

[0027] Use a one-dimensional convolutional neural network to perform convolution on the set of audio segments to obtain a set of target timbre audio segments;

[0028] Call the SOLA algorithm to find the best alignment points of the set of target timbre audio segments, and splice the set of target timbre audio segments according to the best alignment points to obtain the target timbre audio.

[0029] Optionally, in the sixth implementation manner of the first aspect of the present invention, the steps of determining the conditional feature, the diversity feature, and the padding data mask according to the original speech data include:

[0030] Determine the speech feature and the pitch information according to the original speech data;

[0031] Determine the conditional feature, the diversity feature, and the padding data mask according to the speech feature, the pitch information, and the TextEncoder model.

[0032] Optionally, in the seventh implementation manner of the first aspect of the present invention, the step of generating the original speech data according to the real-time conversation audio further includes:

[0033] Perform noise reduction processing on the real-time conversation audio to obtain the original speech data.

[0034] In a second aspect, an embodiment of the present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above real-time voice pitch-changing method are implemented.

[0035] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps of the above real-time voice pitch-changing method are implemented.

[0036] In a fourth aspect, an embodiment of the present invention provides a computer program product, which when running on a terminal device causes the terminal device to execute the above real-time voice pitch-changing method.

[0037] The beneficial effects of the embodiments of the present invention compared with the prior art are as follows: Through the collaborative extraction mechanism of conditional features and diversity features, decoupled representation of speech content and timbre attributes is achieved, enabling the speaker embedding vector to accurately represent the core features of the target timbre; Tensor information is dynamically masked according to the padding data mask, eliminating interference from invalid data while ensuring feature integrity; Through the deep fusion processing of pitch frequency and tensor information, a dynamic balance between pitch conversion and timbre preservation is achieved at the frequency domain feature level. The collaboration of the above steps significantly improves the reconstruction accuracy of the original timbre features in the voice conversion process, enabling the generated speech to reach a human-like level in perceptual dimensions such as timbre similarity and intonation naturalness, and improving the authenticity of real-time voice conversion. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the following-described drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0039] Figure 1 Schematic diagram of an embodiment of the real-time voice conversion method in the embodiments of the present invention;

[0040] Figure 2 Schematic diagram of a specific embodiment of step S102 of the real-time voice conversion method in the embodiments of the present invention;

[0041] Figure 3 Schematic diagram of a specific embodiment of step S103 of the real-time voice conversion method in the embodiments of the present invention;

[0042] Figure 4 Schematic diagram of a specific embodiment of step S104 of the real-time voice conversion method in the embodiments of the present invention;

[0043] Figure 5 Schematic diagram of an embodiment of the terminal device in the embodiments of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0044] In order to make the objectives, technical solutions, and advantages of the present invention clearer, the following further elaborates on the present invention in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the protection scope of the present invention.

[0045] It should be noted that the terms "comprising", "including" and "having" and any variations thereof in the specification, claims and drawings of the present invention are intended to cover non-exclusive inclusion. For example, a process, method, terminal, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally further include steps or units not listed, or may optionally further include other steps or units inherent to these processes, methods, products or devices. In the claims, specification and drawings of the present invention, relational terms such as "first" and "second" are only used to distinguish one entity / operation / object from another entity / operation / object, and do not necessarily require or imply any such actual relationship or order between these entities / operations / objects.

[0046] Reference to "embodiments" herein means that a particular feature, structure or characteristic described in connection with the embodiments can be included in at least one embodiment of the present invention. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0047] With the development of technology, the number of Internet users is increasing. Through market research and user feedback in the past year, it can be known that people have a strong demand for voice conversion, and the market is also very promising. Therefore, the quality of voice conversion is extremely important to us.

[0048] Traditional real-time voice conversion technology usually extracts voice features based on a single model. This method has a relatively rough embedding process for the speaker's timbre features, making it difficult to achieve flexible timbre conversion and diversity expansion, and lacking optimization processing of tensor information, which in turn leads to poor authenticity of voice conversion in real-time voice. A new technical means is needed to solve the above technical problems.

[0049] In view of this, the embodiments of the present invention provide a voice conversion method, terminal device and storage medium for real-time voice. Through the collaborative extraction mechanism of conditional features and diversity features, the decoupled representation of voice content and timbre attributes is realized, so that the speaker embedding vector can accurately represent the core features of the target timbre; the tensor information is dynamically masked according to the padding data mask, eliminating invalid data interference while ensuring feature integrity; through the deep fusion processing of pitch frequency and tensor information, a dynamic balance between pitch conversion and timbre preservation is achieved at the frequency domain feature level. The collaboration of the above steps significantly improves the reconstruction accuracy of the original timbre features in the voice conversion process, making the generated voice reach the level of a real person in terms of timbre similarity, intonation naturalness and other perceptual dimensions, and improving the authenticity of voice conversion in real-time voice.

[0050] The technical terms involved in the embodiments of the present invention in its field include but are not limited to:

[0051] Real-time voice pitch-shifting method: A technology that modifies the speaker's timbre and pitch characteristics in real time through algorithms during the audio stream transmission. Different from offline processing, it needs to meet the latency requirement of <50ms.

[0052] Condition feature: In the present invention, it is generated by fusing pitch information (PO) and Hubert feature (F), and is used to control the semantic consistency of voice conversion.

[0053] Diversity feature: A latent variable representing the change of speech style, obtained by separating the statistics (STATS) of the condition feature. Its function is to increase the randomness of the voice conversion result (such as emotional fluctuations, pronunciation habits) and avoid a mechanical feeling.

[0054] Padding data mask: A binary matrix that identifies the positions of valid data in the processing of variable-length sequences. It is used to mask the padded areas (such as silent segments) after audio framing to prevent invalid data from interfering with model calculations.

[0055] Speaker embedding vector: A 128-dimensional vector extracted through a pre-trained network, used to encode the uniqueness of the speaker's timbre (such as formant distribution, vocalization habits). It is generated by mapping the original speech data.

[0056] TextEncoder model: Based on the multi-head attention architecture of Transformer, it is used to fuse pitch information (PO) and speech features (F). It outputs condition features, diversity features, and masks, and is the core module for feature decoupling.

[0057] Streaming decoder: A decoder that supports frame-by-frame processing, which converts low-dimensional features into high-dimensional spectra through one-dimensional convolution and upsampling operations. Different from traditional batch processing, it can reduce the latency of real-time voice conversion.

[0058] HiFi-GAN vocoder: An audio synthesis model based on the generative adversarial network (GAN), which inputs spectral features and speaker embedding vectors and outputs high-quality waveform data. Its advantage is to retain timbre details and suppress synthetic noise.

[0059] SOLA algorithm: A time-domain audio splicing algorithm that eliminates the break sound at the splicing point by finding the optimal alignment point (minimum phase difference) of adjacent audio blocks.

[0060] Phase alignment: Adjust the phase information of the spectrum after voice conversion to match the harmonic structure of the pitch frequency (P1).

[0061] In order to illustrate the technical solution of the present invention, specific embodiments are used for illustration below.

[0062] Figure 1The figure shows a schematic implementation process of a voice pitch-changing method for real-time voice provided by an embodiment of the present invention. This method can be applied to terminal devices. The terminal device can be a mobile phone, a tablet computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, etc.

[0063] Specifically, the above voice pitch-changing method for real-time voice may include the following steps S101 to S104.

[0064] Step S101: Generate original voice data based on the real-time conversation audio, and determine a conditional feature, a diversity feature, and a padding data mask based on the original voice data;

[0065] In an embodiment of the present invention, the real-time collected conversation audio is subjected to noise reduction processing (such as based on spectrum filtering or a deep learning model) to remove background noise interference and obtain pure original voice data. Subsequently, the preprocessed audio data is input into a voice feature extraction module (R model) and a pitch extraction module (RVMPE model).

[0066] According to the encoder-decoder architecture, the original voice data is encoded through multiple layers of Transformer to extract voice features F including voice content and timbre attributes.

[0067] Based on an improved fundamental frequency detection algorithm, pitch information P0 is extracted from the original audio, and high-precision pitch tracking is achieved through training with labeled data to solve the problem of error accumulation of traditional methods in fast pitch change scenarios.

[0068] The voice features F and the pitch information P0 are input into a TextEncoder model (including an embedding layer, a multi-head attention mechanism, and a one-dimensional convolutional network) for feature fusion and encoding:

[0069] The voice features F and the pitch information P0 are combined into a tensor X1 through the embedding layer, and a tensor X2 is generated using the LeakyReLU activation function; specifically, the pitch information P0 and the voice features F are combined through the embedding layer to obtain the tensor X1. The tensor X1 is activated using the LeakyReLU activation function to obtain the tensor X2.

[0070] A transpose padding operation is performed on the tensor X2 to generate a padding data mask MASK1 for identifying valid voice segments;

[0071] Extract global context features according to the multi-head attention mechanism. After regularization and convolution operations, the conditional feature M (controlling voice conversion) and the diversity feature Log (enhancing generation diversity) are separated. Specifically, X2 and MASK1 are passed into the multi-head attention model to obtain the attention feature Y1. For the attention feature Y1, regularization is performed to alleviate overfitting and obtain the tensor X3. The tensor X3 is processed to obtain the feature STATS. STATS and MASK1 are operated on to remove padding data. STATS is separated according to the output dimension to obtain the conditional feature M and the diversity feature Log. Among them, MASK1 is a binary mask used to identify the valid data positions in the input sequence, enabling the model to ignore padding noise. STATS is an advanced feature extracted by one-dimensional convolution.

[0072] The cooperation between the two is the key mechanism for the encoder (TextEncoder) to efficiently process variable-length sequences and improve the generation quality.

[0073] Step S102: Determine the first tensor information according to the conditional feature, the diversity feature, and the padding data mask, and determine the speaker embedding vector according to the original speech data.

[0074] In the embodiment of the present invention, the conditional feature M and the diversity feature Log are tensor-concatenated or weighted-fused to generate an intermediate tensor, and the padding data mask MASK1 is applied to mask the invalid time periods to eliminate redundant data interference, obtaining the first tensor information Z1.

[0075] The original speech data is mapped through a pre-trained speaker embedding layer (such as based on the X-vector or ECAPA-TDNN model) to extract a 128-dimensional speaker embedding vector G, which represents the unique voice characteristics of the speaker and provides the target voice control parameters for subsequent voice conversion.

[0076] Step S103: Determine the second tensor information according to the first tensor information, the speaker embedding vector, and the padding data mask.

[0077] In the embodiment of the present invention, the first tensor information Z1 and the speaker embedding vector G are input into the Flow flow model (based on a reversible neural network structure), and the following operations are performed:

[0078] Z1 is split into sub-tensors F0 and F1 according to the input dimension, which are used for feature transformation and residual connection respectively.

[0079] Perform cross-attention calculation on F0 and the speaker embedding vector G to achieve dynamic alignment of voice characteristics and speech content.

[0080] Apply the padding data mask MASK1 to perform temporal alignment on the fused features to ensure the consistency of the generated features on the time axis.

[0081] The aligned features are subjected to non - linear transformation through a one - dimensional convolutional network, combined with residual connection (adding F1 to the transformation result) and weight normalization (WN model) to generate a high - dimensional tensor Z2 (the second tensor information), which contains the frequency - domain features that fuse the target timbre and speech content.

[0082] Step S104: Generate the target - timbre audio according to the second tensor information, the speaker embedding vector, and the pitch frequency of the original speech data.

[0083] In the embodiment of the present invention, the pitch frequency P1 extracted in step one (output by the RVMPE model) generates a harmonic sound source signal SINE_WAVE through the SineGen model:

[0084] The pitch frequency P1 is frequency - expanded to generate a frequency sequence containing the fundamental frequency and harmonic components;

[0085] Calculate the phase jump and superimpose random noise to simulate the vibration characteristics of the real vocal cords and improve the naturalness of the generated speech.

[0086] Input the second tensor information Z2 and the harmonic signal SINE_WAVE into the decoder module:

[0087] Upsample Z2 through a one - dimensional convolutional network and perform conditional weighting with the speaker embedding vector G to generate frequency - domain features Z5;

[0088] Fuse Z5 and the harmonic signal SINE_WAVE in the frequency domain and refine through a residual network to generate the target spectral features X S ;

[0089] Use the HiFi - GAN vocoder to convert X S into time - domain waveform data and output the target - timbre audio.

[0090] Frame - buffer the generated audio stream and output the target - timbre audio according to the real - time transmission requirements to meet the requirements of real - time conversation scenarios.

[0091] The beneficial effects of the embodiments of the present invention compared with the prior art are as follows: Through the collaborative extraction mechanism of conditional features and diversity features, the decoupled representation of speech content and timbre attributes is achieved, enabling the speaker embedding vector to accurately represent the core features of the target timbre; the tensor information is dynamically masked according to the padding data mask, eliminating the interference of invalid data while ensuring the integrity of the features; through the deep fusion processing of pitch frequency and tensor information, a dynamic balance between pitch conversion and timbre preservation is achieved at the frequency domain feature level. The collaboration of the above steps significantly improves the reconstruction accuracy of the original timbre features in the voice conversion process, making the generated speech reach the level of a real person in perceptual dimensions such as timbre similarity and intonation naturalness, and improving the authenticity of real-time voice conversion.

[0092] Traditional real-time voice conversion technologies usually rely on a single model (such as fundamental frequency shift or formant adjustment) for feature extraction and conversion, resulting in overly rough embedding processing of the speaker's timbre features and being unable to separate speech content and timbre attributes. Traditional methods also lack a masking mechanism for invalid padding data, are prone to introducing noise interference during frame processing, and it is difficult to accurately represent the target timbre through an embedding vector with a fixed dimension. Based on this, the present invention proposes an alternative embodiment.

[0093] Refer to Figure 2 , Figure 2 which is a schematic diagram of a specific embodiment of step S102 in the voice conversion method of real-time voice in the embodiments of the present invention. Step S102 also includes the following specific implementation manners.

[0094] Step S1021: Fuse the conditional features and diversity features to generate an intermediate tensor.

[0095] In the embodiment of the present invention, the conditional feature M (representing the timbre conversion control parameter) and the diversity feature Log (representing the generation diversity parameter) are fused by means of tensor concatenation (Concatenation) or element-wise weighted addition (such as attention weight assignment). For example, in the embedding layer, M and Log are concatenated along the feature dimension to form an intermediate tensor X with an extended dimension, and a non-linear transformation is performed through a fully connected layer to generate the intermediate tensor.

[0096] According to the context information of the speech segment (such as speech content, pitch change), the fusion weights of M and Log are dynamically adjusted according to the gating mechanism (GatingNetwork) to ensure that important features (such as the core timbre parameters) play a dominant role in the fusion process and improve the flexibility of feature expression.

[0097] Step S1022: Apply a masking operation of the padding data mask to the intermediate tensor to obtain the first tensor information.

[0098] In an embodiment of the present invention, the padding data mask MASK1 (a binary matrix, where valid speech segments are marked as 1 and padding segments are marked as 0) is multiplied element-wise with the intermediate tensor to mask the data in the invalid padding regions. For example, zero values are assigned to the positions in the intermediate tensor corresponding to MASK1 being 0, eliminating the redundant noise interference introduced by audio frame padding.

[0099] After the masking operation, temporal smoothing processing (such as one-dimensional convolution or mean pooling) is performed on the tensor of the valid speech segments, which can effectively avoid feature jumps caused by masking, so that the generated first tensor information Z1 remains continuous on the time axis.

[0100] Step S1023: Map the original speech data through a pre-trained speaker embedding layer to obtain a 128-dimensional speaker embedding vector.

[0101] In an embodiment of the present invention, a pre-trained speaker embedding layer (such as a model based on the X-vector or ECAPA-TDNN architecture) is used to map the original speech data:

[0102] Operations such as frame splitting, windowing, and Fourier transform are performed on the original speech data to extract Mel spectrogram features;

[0103] The Mel spectrogram is input into the pre-trained model, and global features are aggregated through a statistics pooling layer to generate a 128-dimensional speaker embedding vector G, where each dimension represents a specific attribute of the target timbre (such as timbre brightness, formant distribution).

[0104] According to the context information of the real-time speech (such as speech rate, emotion), G is dynamically fine-tuned (such as adjusting the embedding vector weights through a lightweight MLP network) to enhance the representation ability of the embedding vector for the target timbre.

[0105] In the embodiment of the present invention, through the dynamic fusion mechanism of conditional features and diversity features, the flexibility of timbre conversion and the diversity of generation results are significantly improved. Specifically, a padding data mask is used to accurately mask the intermediate tensor, effectively eliminating the interference of invalid padding segments on feature fusion, and enabling the temporal coherence of the generated speech; high-dimensional timbre features are extracted through a pre-trained speaker embedding layer, realizing fine-grained modeling of the target timbre. This alternative embodiment solves the problem of timbre distortion caused by rough feature extraction of traditional methods, enabling the voice conversion process to flexibly adapt to the timbre features of different speakers while ensuring the naturalness and authenticity of the generated speech.

[0106] Traditional real-time voice conversion technologies usually adopt static weights to fuse voice features and timbre parameters, and are unable to dynamically adjust the association strength according to the context, resulting in a rigid timbre conversion. At the same time, the traditional frame-by-frame processing lacks a timing alignment mechanism for padding segments, which is likely to cause audio breaks and noise interference. Moreover, the high computational complexity of non-streaming decoders makes it difficult to meet the real-time requirements. Based on this, an alternative embodiment of the present invention is proposed.

[0107] Referring to Figure 3 , Figure 3 FIG. is a schematic diagram of a specific embodiment of step S103 of the method for pitch-changing of real-time voice in an embodiment of the present invention, and step S103 further includes the following specific implementation manners.

[0108] Step S1031, perform cross-attention calculation on the first tensor information and the speaker embedding vector to generate a fused feature.

[0109] In an embodiment of the present invention, the first tensor information Z1 (including conditional features and masked diversity features) and the speaker embedding vector G (128-dimensional timbre representation) are input into a multi-head cross-attention mechanism: the first tensor information Z1 is linearly mapped to a query vector (Query), the speaker embedding vector G is mapped to a key vector (Key) and a value vector (Value) to construct a triple for attention calculation; calculate the similarity score between the query vector and the key vector, generate an attention weight matrix through Softmax normalization to represent the cross-modal association strength between the first tensor information Z1 and the speaker embedding vector G; perform weighted summation of the attention weight and the value vector to generate a fused feature Y1, which encodes the collaborative relationship between the speech content and the target timbre at the same time.

[0110] Perform a residual connection (add with Z1) and layer normalization (LayerNorm) on the fused feature Y1 to alleviate the problem of gradient disappearance and stabilize the training process.

[0111] Step S1032, perform a timing alignment operation on the fused feature by applying a padding data mask to obtain an aligned feature.

[0112] In an embodiment of the present invention, perform timing alignment on the padding data mask MASK1 (identifying valid speech segments) and the fused feature Y1 to obtain an aligned feature Y2:

[0113] Extract the start and end timestamps of continuous valid speech segments in MASK1, and segment and intercept the corresponding time periods in the aligned feature Y2;

[0114] Through the dynamic time warping (DTW) algorithm or one-dimensional interpolation method, smoothly connect each segment of features on the time axis to eliminate the timing break caused by frame padding.

[0115] The aligned feature Y2 is processed by a one-dimensional causal convolutional network to strengthen the contextual dependency between adjacent frames to ensure the coherence of the generated features on the time axis.

[0116] Step S1033: upsample the alignment features through a streaming decoder to obtain the second tensor information.

[0117] In an embodiment of the present invention, the time-aligned alignment feature Y2 is input into a streaming decoder (a reversible neural network based on Glow or RealNVP architecture):

[0118] The alignment feature Y2 is nonlinearly transformed through the Affine Coupling Layer to gradually expand the feature dimension;

[0119] Apply a learnable upsampling kernel (such as transposed convolution) to upsample the aligned feature Y2 by a factor of 4 to generate the second tensor information Z2.

[0120] In this embodiment of the present invention, a cross-attention mechanism is used to dynamically associate speech content with the target timbre, significantly improving the semantic consistency of timbre conversion. A padding mask-driven timing alignment operation effectively eliminates audio fragmentation caused by frame padding, enhancing the naturalness of the generated speech. A streaming decoder, combined with a reversible neural network architecture, achieves low-latency processing while ensuring high-fidelity upsampling. This optional embodiment solves the problem of abrupt voice changes caused by timing misalignment and rough feature association in traditional methods, enabling the generated speech to reach industry-leading levels in terms of timbre adaptability, timing fluency, and real-time performance.

[0121] Traditional real-time voice-changing technologies typically use non-parametric vocoders, such as Griffin-Lim, for waveform reconstruction, resulting in significant phase distortion and residual noise in the generated speech. Furthermore, traditional vocoders lack dynamic control over the target timbre, making high-fidelity timbre conversion difficult. Furthermore, they suffer from high computational complexity and significant latency, making them incapable of meeting the demands of real-time interaction. Based on this, the present invention proposes an alternative embodiment.

[0122] Reference Figure 4 , Figure 4 1 is a schematic diagram of a specific embodiment of step S104 of the method for changing the voice of real-time speech in an embodiment of the present invention. Step S104 also includes the following specific implementation methods.

[0123] Step S1041 , performing phase alignment on the second tensor information and the pitch frequency of the original speech data to obtain aligned spectral features.

[0124] In an embodiment of the present invention, the second tensor information Z2 (representing the frequency domain characteristics of the fused target timbre) is phase-aligned with the pitch frequency P1 of the original speech data (extracted by the RMVPE model):

[0125] Perform inverse short-time Fourier transform (ISTFT) on Z2 to obtain the time-domain waveform, extract its fundamental frequency component and perform dynamic time warping (DTW) with the pitch frequency P1 to eliminate the phase shift caused by pitch shifting;

[0126] Through a phase interpolation algorithm (such as the phase reconstruction module of World Vocoder), synchronously adjust the spectral amplitude of Z2 with the fundamental frequency information of the pitch frequency P1 to generate the aligned spectral feature S1, ensuring that the harmonic structure strictly matches the target pitch.

[0127] Perform anti-aliasing filtering (such as Butterworth low-pass filter) on the aligned spectral feature S1 to suppress high-frequency noise interference, and apply a masking threshold model (based on psychoacoustic model) to remove the spectrally insensitive components to improve the clarity of the generated speech.

[0128] Step S1042, input the spectral feature and the speaker embedding vector into the HiFi-GAN vocoder to obtain the waveform data of the target timbre.

[0129] In an embodiment of the present invention, the aligned spectral feature S1 and the speaker embedding vector G (128-dimensional timbre representation) are input into the HiFi-GAN vocoder:

[0130] Map G through a fully connected layer to a conditional weight matrix, and multiply it with S1 channel by channel to achieve dynamic injection of the timbre condition;

[0131] Perform multi-scale analysis on the spectral feature through a multi-scale discriminator to enhance the high-frequency details (such as consonant plosives, fricatives) of the generated waveform.

[0132] HiFi-GAN uses a transposed convolutional layer and a residual connection structure for multi-stage upsampling, for example:

[0133] Perform 4-fold upsampling on S1 to generate intermediate-frequency features, and combine a periodic activation function to simulate the vocal cord vibration characteristics;

[0134] Synthesize the time-domain waveform data W1 of the target timbre through a parallel waveform generation path (Parallel WaveNet architecture), and output a high-fidelity audio stream with an end-to-end delay of less than 20 ms.

[0135] Step S1043, generate the target timbre audio according to the waveform data.

[0136] In an embodiment of the present invention, the generated waveform data W1 is segmented into audio frames with a length of 50 ms, and streaming transmission is implemented through a circular buffer to ensure the synchronization of audio output and input in a conversation scenario.

[0137] Perform dynamic range compression (DRC) and loudness normalization (EBU R128 standard) on the output audio to eliminate volume jumps caused by timbre conversion and improve auditory comfort.

[0138] In an embodiment of the present invention, through the phase alignment mechanism of pitch frequency and spectral features, the harmonic consistency of the transformed audio is significantly improved, and the "mechanical voice" problem caused by phase distortion in traditional methods is solved; the multi-scale conditional generation architecture of the HiFi-GAN vocoder achieves real-time synthesis with ultra-low latency while ensuring high-fidelity sound quality; dynamic loudness equalization and anti-aliasing processing further optimize the auditory experience. This alternative embodiment enables the generated speech to reach the level of a real person in terms of pitch naturalness, timbre fidelity, and real-time performance, breaking through the sound quality bottleneck caused by insufficient vocoder performance in traditional voice transformation technologies.

[0139] Traditional real-time voice transformation technologies usually adopt a frame splicing method with a fixed offset, resulting in problems such as phase discontinuity and volume mutation at the junction of audio blocks, generating an obvious "mechanical splicing feeling". Based on this, the present invention proposes an alternative embodiment.

[0140] Step S101 further includes the following specific implementation manners.

[0141] Step S1011, generate intermediate audio according to the waveform data.

[0142] Step S1012, perform audio time alignment and audio splicing on the intermediate audio according to the SOLA algorithm to obtain the target timbre audio.

[0143] In an embodiment of the present invention, high-precision audio time alignment is achieved through the SOLA algorithm, effectively eliminating the phase jump and stuttering problems caused by traditional splicing methods, and significantly improving the fluency and naturalness of the generated speech.

[0144] Traditional real-time voice transformation technologies usually splice audio blocks with a fixed frame offset, resulting in fundamental frequency jumps, volume mutations, and phase discontinuities at the junction of adjacent frames, generating an obvious "mechanical splicing feeling"; moreover, traditional methods lack optimization processing of local features of audio segments and are difficult to suppress noise residue and detail distortion. Based on this, the present invention proposes an alternative embodiment.

[0145] Step S1012 further includes the following specific implementation manners.

[0146] Step S10121, segment the intermediate audio to obtain a set of audio segments.

[0147] In an embodiment of the present invention, the intermediate audio stream is segmented into a set of audio segments {S1, S2, ..., Sn} according to a dynamic frame length (adjustable from 20 to 50 ms), and Hamming Window processing is performed, with a 25% overlapping area (such as 12.5 ms overlap for a 50 ms frame length) retained between adjacent frames to mitigate the spectral leakage problem.

[0148] Silent segments are identified based on energy threshold detection (such as short-time energy below -40 dB), and invalid audio segments are dynamically removed using a padding data mask to reduce subsequent computational redundancy.

[0149] Step S10122: Use a one-dimensional convolutional neural network to perform convolution on the set of audio segments to obtain a set of audio segments with the target timbre.

[0150] In an embodiment of the present invention, a one-dimensional convolutional neural network (1D-CNN) is used to perform convolution processing on the set of audio segments:

[0151] Multiple groups of parallel convolutional kernels (such as 32 groups, kernel length 5, stride 1) are used to extract local temporal features of the speech signal (such as formant transitions, consonant plosion characteristics);

[0152] Feature selection is achieved through the activation function of the gated linear unit (GLU) to suppress noise interference.

[0153] Residual connection (Residual Connection) is performed on the convolution output to retain the details of the original waveform, and it is weighted and fused with the speaker embedding vector G frame by frame to generate a set of audio segments with the target timbre {T1, T2, ..., Tn}.

[0154] Step S10123: Call the SOLA algorithm to find the best alignment points for the set of audio segments with the target timbre, and splice the set of audio segments with the target timbre according to the best alignment points to obtain the audio with the target timbre.

[0155] In an embodiment of the present invention, for the overlapping area of adjacent audio segments with the target timbre (such as the end of Ti and the beginning of Tj), the following operations are performed:

[0156] Slide Tj within the overlapping window, calculate its normalized cross-correlation value (Normalized Cross-Correlation, NCC) with Ti, and locate the offset ΔT corresponding to the maximum NCC;

[0157] Combine the dynamic time warping (DTW) algorithm to align the pitch trajectories to ensure the continuity of the fundamental frequency.

[0158] Perform non-uniform time stretching on Tj according to ΔT (using a phase vocoder to adjust the waveform duration), eliminate the phase jump at the alignment point, and apply cosine crossfade to achieve a smooth transition.

[0159] A double ring buffer is used to store the processed audio stream and output the spliced target tone audio in real time.

[0160] In this embodiment of the present invention, a one-dimensional convolutional neural network is used to enhance the local features of audio clips, significantly improving the fidelity of detail in the voice-changed audio. The SOLA algorithm, combined with a dynamic time warping pitch alignment mechanism, achieves seamless splicing between audio blocks, eliminating the issues of discontinuity and phase distortion caused by traditional methods. The adaptive framing and buffer management strategy ensures sound quality while meeting stringent real-time requirements. This optional embodiment enables the voice-changed audio to achieve a seamless, lifelike effect in auditory perception, resolving the inherent drawbacks of traditional splicing techniques, which are stiff and unnatural.

[0161] Traditional real-time voice-changing technologies typically rely on a single model (such as fundamental frequency shifting or formant adjustment) to simultaneously extract both speech content and timbre features. This results in a high degree of feature representation coupling, making it difficult to achieve precise timbre control. Furthermore, traditional methods lack a dynamic masking mechanism for invalid filler segments, making them susceptible to noise interference. Based on this, the present invention proposes an alternative embodiment.

[0162] Step S102 also includes the following specific implementation methods.

[0163] Step S1024: Determine voice features and pitch information based on the original voice data.

[0164] In this embodiment of the present invention, raw speech data is input into the R model (Hubert model), and speech features F, representing speech content and timbre, are extracted using a multi-layer Transformer encoder. The Hubert model uses a self-supervised pre-training mechanism and masked speech modeling to capture the contextual dependencies of speech.

[0165] The RMVPE model (Robust Multi-Vocal Pitch Estimator) tracks the fundamental frequency of raw speech data and outputs pitch information P0. This model, based on an improved CREPE algorithm (Convolutional Pitch Estimation Network) and fine-tuned with manually corrected pitch-annotated data, significantly improves the accuracy of detecting rapid pitch changes.

[0166] Optionally, noise reduction processing is performed on the real-time conversation audio to obtain original voice data.

[0167] Step S1025: Determine the conditional feature, diversity feature, and padding data mask according to the voice feature pitch information and the TextEncoder model.

[0168] In an embodiment of the present invention, the voice feature F and the pitch information P0 are input into the TextEncoder model (an encoder based on the multi-head attention mechanism):

[0169] The F and the pitch information P0 are mapped to the same feature space through a linear projection layer, and are concatenated along the channel dimension to generate a tensor X1;

[0170] Perform multi-head self-attention calculation on X1 to extract cross-modal correlation features Y1, and apply layer normalization (LayerNorm) to stabilize the training process.

[0171] Perform one-dimensional causal convolution (Kernel Size = 5, stride = 1) on the attention feature Y1 to generate a feature STATS containing temporal context information; according to the effective voice duration (determined by voice activity detection VAD), generate a padding data mask MASK1, identifying the effective voice segment as 1 and the padding segment as 0.

[0172] Perform dimensional slicing on STATS to separate the conditional feature M (the core parameter controlling timbre conversion) and the diversity feature Log (the weight controlling generation diversity).

[0173] In the embodiment of the present invention, through the decoupling extraction mechanism of voice features and pitch information, the accurate separation of voice content and timbre attributes is realized, significantly improving the controllability of the voice conversion process; the design of the TextEncoder model combining multi-head attention and causal convolution ensures the temporal consistency of feature encoding while dynamically generating a padding mask; the independent output of the conditional feature and the diversity feature supports flexible timbre conversion and diversity expansion. This optional embodiment solves the problem of timbre distortion caused by the confusion of single-model features in traditional methods, making the generated voice reach the level of a real person in terms of timbre fidelity and naturalness.

[0174] As Figure 5 shown, it is a schematic diagram of a terminal device provided by an embodiment of the present invention. The terminal device 500 may include: a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501, such as a voice conversion program for real-time voice. When the processor 501 executes the computer program 503, the steps in the above-mentioned various real-time voice conversion embodiments are implemented.

[0175] A computer program can be divided into one or more modules / units. One or more modules / units are stored in the memory 502 and executed by the processor 501 to implement the present invention. One or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and these instruction segments are used to describe the execution process of the computer program in the terminal device.

[0176] The terminal device may include, but is not limited to, the processor 501 and the memory 502. Those skilled in the art can understand that Figure 5 merely examples of the terminal device, which do not constitute a limitation on the terminal device, may include more or fewer components than those shown in the figure, or combine certain components, or different components. For example, the terminal device may further include input / output devices, network access devices, buses, etc.

[0177] The so-called processor 501 may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0178] The memory 502 may be an internal storage unit of the terminal device, such as the hard disk or memory of the terminal device. The memory 502 may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device. Further, the memory 502 may also include both the internal storage unit and the external storage device of the terminal device. The memory 502 is used to store computer programs and other programs and data required by the terminal device. The memory 502 may also be used to temporarily store data that has been output or is to be output.

[0179] It should be noted that for the sake of convenience and brevity of description, the structure of the above terminal device may also refer to the specific description of the structure in the method embodiment, which will not be elaborated here.

[0180] An embodiment of the present invention also provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned real-time voice pitch-changing method can be implemented.

[0181] An embodiment of the present invention provides a computer program product, and when the computer program product runs on a mobile terminal, the mobile terminal can be made to execute the steps in the above-mentioned real-time voice pitch-changing method when executed.

[0182] In the above embodiments, the descriptions of the various embodiments each have their own focuses. For parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0183] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0184] In the embodiments provided by the present invention, it should be understood that the disclosed terminal devices and methods can be implemented in other ways. For example, the terminal device embodiments described above are merely illustrative. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of devices or units can be in electrical, mechanical or other forms.

[0185] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0186] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0187] When the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-described embodiment methods of the present invention, it can also be completed by a computer program instructing relevant hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-described various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0188] The above-described embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments or perform equivalent replacements for some of the technical features. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention and should all be included in the protection scope of the present invention.

Claims

1. A method for voice pitch transformation of real-time voice, characterized in that, Including: Generate original speech data based on real-time conversation audio, and determine conditional features, diversity features, and padding data masks based on the original speech data; Determine the first tensor information according to the conditional features, diversity features, and padding data masks, and determine the speaker embedding vector according to the original speech data; Determine the second tensor information according to the first tensor information, the speaker embedding vector, and the padding data mask; Generate the target timbre audio according to the second tensor information, the speaker embedding vector, and the pitch frequency of the original speech data; Among them, the step of determining the first tensor information according to the conditional features, diversity features, and padding data masks includes: Perform feature fusion on the conditional features and the diversity features to generate an intermediate tensor, where the feature fusion includes concatenating the conditional features and the diversity features along the feature dimension in the embedding layer and performing a non-linear transformation through a fully connected layer; Perform a masking operation of the padding data mask on the intermediate tensor to obtain the first tensor information, where the masking operation includes element-wise multiplication of the padding data mask and the intermediate tensor; Among them, the step of determining the second tensor information according to the first tensor information, the speaker embedding vector, and the padding data mask includes: Perform cross-attention calculation on the first tensor information and the speaker embedding vector to generate a fused feature; Perform a temporal alignment operation of the padding data mask on the fused feature to obtain an aligned feature; Upsample the aligned feature through a streaming decoder to obtain the second tensor information.

2. The pitch-shifting method for real-time voice according to claim 1, wherein The step of determining the speaker embedding vector according to the original speech data includes: Map the original speech data through a pre-trained speaker embedding layer to obtain the 128-dimensional speaker embedding vector.

3. The tone-changing method for real-time voice according to claim 1, wherein, The step of generating the target timbre audio according to the second tensor information, the speaker embedding vector, and the pitch frequency of the original speech data includes: Perform phase alignment on the second tensor information and the pitch frequency of the original speech data to obtain the aligned spectral features; Input the spectral features and the speaker embedding vector into the HiFi-GAN vocoder to obtain the waveform data of the target timbre; Generate the target timbre audio according to the waveform data.

4. The pitch-shifting method for real-time voice according to claim 3, characterized in that, The step of generating the target timbre audio according to the waveform data includes: Generate intermediate audio according to the waveform data; Perform audio time alignment and audio splicing on the intermediate audio according to the SOLA algorithm to obtain the target timbre audio.

5. The pitch-shifting method for real-time voice according to claim 4, wherein The step of performing audio time alignment and audio splicing on the intermediate audio according to the SOLA algorithm to obtain the target timbre audio includes: Segment the intermediate audio to obtain a set of audio segments; Use a one-dimensional convolutional neural network to perform convolution on the set of audio segments to obtain a set of target timbre audio segments; Call the SOLA algorithm to find the best alignment points of the set of target timbre audio segments, and splice the set of target timbre audio segments according to the best alignment points to obtain the target timbre audio.

6. The pitch-shifting method for real-time voice according to claim 1, characterized in that, The steps of determining conditional features, diversity features, and filling data masks according to the original speech data include: Determine speech features and pitch information according to the original speech data; Determine the conditional features, the diversity features, and the filling data masks according to the speech features, the pitch information, and the TextEncoder model.

7. The pitch-shifting method for real-time voice according to claim 6, characterized in that, The steps of generating original speech data according to the real-time conversation audio further include: Perform noise reduction processing on the real-time conversation audio to obtain the original speech data.

8. A terminal device, characterized in that It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the real-time voice pitch-changing method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the steps of the real-time voice pitch-changing method according to any one of claims 1 to 7 are implemented.

Images