Voice conversion method and device, electronic equipment and storage medium

Abstract

The invention provides a voice conversion method and device, electronic equipment and a storage medium, and the method comprises the steps: respectively extracting hidden layer representations of a source voice signal and a target reference voice signal, and obtaining a source voice content representation and a target reference voice content representation; performing first-level attention processing based on the target reference voice signal and the target reference voice content representation, and determining a corresponding frame-level speaker representation; performing second-level attention processing based on the source voice content representation and the frame-level speaker representation to obtain a target frame-level speaker representation; and processing based on the sentence-level speaker embedding, the target frame-level speaker representation and the source voice content representation to obtain a target voice, and outputting the target voice. According to the method, the speaker information of the target reference voice signal and the content context of the source voice signal content representation are deeply fused, and the frame-level speaker representation accurately aligned with the source voice signal content representation is generated, so that the conversion quality is improved.

Description

Technical Field

[0001] This invention relates to the field of voice data processing technology, and in particular to a voice conversion method, apparatus, electronic device, and storage medium. Background Technology

[0002] Speaker conversion (VC) technology is used to alter the speaker characteristics in a speech signal while preserving its linguistic content.

[0003] Cross-linguistic speaker translation further extends this technology to speakers of different languages, with broad application prospects, such as foreign language film dubbing and building low-resource language speech databases, making it of significant research importance and practical value. However, the performance of cross-linguistic speaker translation is generally inferior to that of intralingual scenarios. This is mainly due to the inherent differences in pronunciation and intonation between different language systems, which brings additional challenges to accurately modeling and translating speaker features.

[0004] Existing techniques typically perform vector quantization or K-means clustering operations on self-supervised learning representations, but this can easily introduce pronunciation errors. Furthermore, excessive decoupling may strip away important paralinguistic information such as prosody, resulting in unnatural converted speech. In cross-linguistic scenarios, the lack of shared phonemic cues across languages ​​makes it difficult to align the acoustic frames of the source and target speech, thus affecting conversion quality. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a speech conversion method, apparatus, electronic device, and storage medium to solve the problems affecting conversion quality in the prior art.

[0006] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0007] The first aspect illustrates a speech conversion method, the method comprising:

[0008] The hidden layer representations of the source speech signal and the target reference speech signal are extracted respectively to obtain the source speech content representation and the target reference speech content representation;

[0009] Based on the target reference speech signal and the target reference speech content representation, the first level of attention processing is performed to determine the corresponding frame-level speaker representation;

[0010] A second-level attention process is performed based on the source speech content representation and the frame-level speaker representation to obtain the target frame-level speaker representation.

[0011] The target speech is obtained and output by processing based on sentence-level speaker embedding, target frame-level speaker representation, and source speech content representation. The sentence-level speaker embedding is extracted from the target reference speech signal.

[0012] Optionally, the hidden layer representations of the source speech signal and the target reference speech signal are extracted separately to obtain the source speech content representation and the target reference speech content representation, including:

[0013] The source speech signal is randomly offset to obtain the offset speech signal;

[0014] The feature extraction model is invoked to extract the hidden layer representations from the source speech content representation and the target reference speech signal, respectively, to obtain the source speech content representation and the target reference speech content representation.

[0015] Optionally, a first-level attention processing is performed based on the target reference speech signal and the target reference speech content representation to determine the corresponding frame-level speaker representation, including:

[0016] Extract sentence-level speaker embeddings from the target reference speech signal;

[0017] Wavelet convolution is performed on the target reference speech content representation to obtain the frame-level speaker initial representation;

[0018] Attention processing is performed based on the sentence-level speaker embedding and the frame-level speaker initial representation to obtain the corresponding frame-level speaker representation.

[0019] Optionally, a second-level attention processing is performed based on the source speech content representation and the frame-level speaker representation to obtain the target frame-level speaker representation, including:

[0020] Align the source speech content representation with the frame-level speaker representation using pronunciation features to obtain the target frame-level speaker representation.

[0021] Optionally, the target speech is processed based on sentence-level speaker embedding, target frame-level speaker representation, and source speech content representation to obtain and output the target speech, including:

[0022] The first data is obtained by concatenating sentence-level speaker embeddings and target frame-level speaker representations.

[0023] The first data and the source speech content representation are used as input data and input into a preset speech generation model to process the input data based on the speech generation model and convert it into a Mel spectrogram.

[0024] The preset vocoder is invoked to convert the Mel spectrogram into a time-domain waveform to obtain the target speech, which is then output.

[0025] The second aspect discloses a speech conversion device, the device comprising:

[0026] The extraction unit is used to extract the hidden layer representations of the source speech signal and the target reference speech signal respectively, so as to obtain the source speech content representation and the target reference speech content representation.

[0027] The enhancement processing unit is used to perform a first-level attention processing based on the target reference speech signal and the target reference speech content representation to determine the corresponding frame-level speaker representation; and to perform a second-level attention processing based on the source speech content representation and the frame-level speaker representation to obtain the target frame-level speaker representation.

[0028] The generation unit is used to process the target speech based on sentence-level speaker embedding, target frame-level speaker representation, and source speech content representation to obtain the target speech and output it.

[0029] Optionally, the extraction unit is specifically used for:

[0030] The source speech signal is randomly offset to obtain the offset speech signal;

[0031] The feature extraction model is invoked to extract the hidden layer representations from the source speech content representation and the target reference speech signal, respectively, to obtain the source speech content representation and the target reference speech content representation.

[0032] Optionally, based on the target reference speech signal and the target reference speech content representation, a first-level attention processing is performed to determine the corresponding frame-level speaker representation enhancement processing unit, specifically used for:

[0033] Extract sentence-level speaker embeddings from the target reference speech signal;

[0034] Wavelet convolution is performed on the target reference speech content representation to obtain the frame-level speaker initial representation;

[0035] Attention processing is performed based on the sentence-level speaker embedding and the frame-level speaker initial representation to obtain the corresponding frame-level speaker representation.

[0036] The third aspect discloses an electronic device including a processor and a memory, the memory being used to store program code and data for data generation, and the processor being used to invoke program instructions in the memory to execute a speech conversion method as described in any of the first aspects.

[0037] The fourth aspect discloses a storage medium including a stored program, wherein, when the program is executed, it controls the device on which the storage medium resides to perform a speech conversion method as described in any of the first aspects.

[0038] Based on the above embodiments of the present invention, a speech conversion method, apparatus, electronic device, and storage medium are provided. The method includes: extracting hidden layer representations of a source speech signal and a target reference speech signal respectively to obtain a source speech content representation and a target reference speech content representation; performing a first-level attention processing based on the target reference speech signal and the target reference speech content representation to determine the corresponding frame-level speaker representation; performing a second-level attention processing based on the source speech content representation and the frame-level speaker representation to obtain a target frame-level speaker representation; and processing based on sentence-level speaker embedding, the target frame-level speaker representation, and the source speech content representation to obtain the target speech and output it. In the embodiments of the present invention, the source speech content representation and the target reference speech content representation of the source speech signal and the target reference speech signal are extracted; then, the speaker information of the target reference speech signal is deeply fused with the content context of the source speech signal content representation to generate a frame-level speaker representation that is precisely aligned with the source speech signal content representation, i.e., the target frame-level speaker representation; finally, the final time-domain waveform is synthesized through sentence-level speaker embedding, the target frame-level speaker representation, and the source speech content representation, thereby improving the conversion quality. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0040] Figure 1 This is a schematic flowchart illustrating a speech conversion method according to an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of the speech conversion process shown in an embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of the structure of a speech conversion device according to an embodiment of the present invention. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0045] It should be noted that the descriptions involving "first," "second," etc., in this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0046] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0047] This application first constructs a shared multilingual continuous representation space, and performs unified modeling of interlocutor and content representations. Then, it obtains fine-grained speaker representations that are precisely aligned with the content through a multi-stage attention-based query enhancement mechanism. Finally, it uses a conditional flow matching model for efficient and high-quality speech generation.

[0048] See Figure 1 The following is a flowchart illustrating a speech conversion method according to an embodiment of the present invention. The method includes:

[0049] Step S101: Extract the source speech signals respectively and target reference speech signal The hidden layer representation is used to obtain the source speech content representation. and target reference speech content representation ;

[0050] Optionally, before performing step S101, the source speech signal corresponding to the user's input speech and the selected cross-language target reference speech signal are obtained.

[0051] It should be noted that the specific implementation of step S101 includes the following steps.

[0052] Step S11: Process the source speech signal Perform random offset to obtain the offset speech signal. .

[0053] In the specific implementation step S11, the source speech signal is processed at the speech signal level. By randomly shifting the fundamental frequency and formants, the speaker information of the source speech is altered, resulting in the shifted speech signal. .

[0054] Step S12: Call the feature extraction model to extract the offset speech signals respectively. and the target reference speech signal The hidden layer representation is used to obtain the source speech content representation. and target reference speech content representation ;

[0055] In the specific implementation step S12, the offset speech signal is extracted using a pre-trained feature extraction model. The hidden layer representation is used to obtain the source speech content representation. This refers to a deep feature vector sequence. Unlike traditional Mel-spectrograms, it contains rich contextual and semantic information. A pre-trained feature extraction model is then used to extract the target reference speech signal. The hidden layer representation is used to obtain the target reference speech content representation. ;

[0056] It should be noted that since the feature extraction model is trained on large-scale multilingual data, its representation space contains shared information across languages. Therefore, these representations together constitute a shared multilingual continuous space. This space acts as a "cross-lingual bridge," effectively alleviating the alignment difficulties caused by different phoneme systems and laying the foundation for subsequent processing.

[0057] It should be noted that the feature extraction model is an SSL model;

[0058] SSL comprises a Record Layer and a Transport Layer, with the Record Layer protocol determining the encapsulation format of the Transport Layer data.

[0059] Source speech content representation and target reference speech content representation It can be an SSL identifier.

[0060] Step S102: Based on the target reference speech signal and target reference speech content representation Attention processing is performed to determine the corresponding frame-level speaker representation. .

[0061] It should be noted that the specific implementation of step S102 includes the following steps.

[0062] Step S21: Extract the target reference speech signal Sentence-level speaker embedding .

[0063] It should be noted that the speaker verification model CAM++ is trained in advance based on historical multilingual speech data and pre-labeled sentence-level speaker embeddings;

[0064] In the specific implementation step S21, the speaker verification model CAM++ obtains the target reference speech signal. Extracting sentence-level speaker embeddings .

[0065] Step S22: Represent the target reference speech content Wavelet convolution is performed to obtain the initial speaker representation at the frame level.

[0066] In the specific implementation step S22, the target reference speech content is first represented. The cascaded wavelet decomposition technique is used to decompose the data into frequency components of different scales. Then, convolution operations are applied to the feature components of the multiple frequency bands obtained from the decomposition to obtain the frame-level speaker initial representation WTConv. ).

[0067] Specifically, the target reference speech content representation A stepwise one-dimensional wavelet transform is performed, with each level decomposing into low-frequency L and high-frequency H. Convolution is applied to L and H at each level to achieve separate convolution of multiple frequency bands. Then, a stepwise one-dimensional wavelet inverse transform is performed, fusing the convolution results of the current layer with the reconstruction results of the next layer to obtain the frame-level speaker initial representation WTConv( ).

[0068] Step S23: Based on the sentence-level speaker embedding And frame-level speaker initial representation WTConv ( Attention processing is performed to obtain the corresponding frame-level speaker representation. .

[0069] In the specific implementation step S23, the sentence-level speaker is embedded. The query vector as the first level of attention Frame-level speaker initial representation WTConv ( ) as the key vector K1 and the value vector V1; and use them as the key vector K1 and the value vector V1. Substituting K1 and V1 into formula (1) for attention processing, the corresponding frame-level speaker representation is obtained. .

[0070] Formula (1):

[0071]

[0072] in, For frame-level speaker representation, Embedding for sentence-level speakers K1 and V1 are both frame-level speaker initial representations WTConv ( ), The length of the target reference speech signal in the time domain is represented by C, where C represents the dimension of the hidden layer.

[0073] In this embodiment of the invention, sentence-level speaker embeddings of the target reference speech signal are extracted from a pre-trained speaker verification model and used as query vectors. To enrich the speaker information contained in the SSL representation, this invention performs wavelet convolution on the frame-level SSL representation of the target speaker reference speech obtained from the target reference speech content representation. This decomposes the representation into multiple different frequency sub-bands using a set of wavelet convolution kernels and fuses speaker features with varying spectral scales (such as fundamental frequency correlation features and spectral envelope correlation features) to obtain a more fine-grained and information-rich frame-level initial speaker representation. This enhanced frame-level initial speaker representation is then used as a key and value for the first round of attention calculation. This step aims to inject the global identity features of the target speaker into the local representation of each frame.

[0074] Step S103: Based on the source speech content representation and frame-level speaker representation Pronunciation feature alignment is performed to obtain the target frame-level speaker representation. ;

[0075] In the specific implementation step S103, the source speech content representation and the frame-level speaker representation are aligned by pronunciation features to obtain the target frame-level speaker representation.

[0076] Specifically, representing the source speech content That is, the content representation after perturbation The query serves as the second-level attention mechanism; the content representation obtained through the first-level attention is not aligned in length with the content representation from the source. This is to obtain the target frame-level speaker representation. A content query module based on multi-head cross-attention Transformer is introduced to guide frame-level speaker representation. Towards the representation of source speech content Content-related pronunciation features are aligned to obtain a target frame-level speaker representation. ;

[0077] In other words, the source speech content is represented The query vector as the first level of attention Frame-level speaker representation As the key vector K2 and the value vector V2; and as well as... Substituting K2 and V2 into formula (2) for attention processing, we obtain the target frame-level speaker representation. .

[0078] Formula (2):

[0079]

[0080] in, For the target frame-level speaker representation, Representation of source speech content Both K2 and V2 are frame-level speaker representations. , Represents the source speech signal In the time domain, C represents the hidden layer dimension;

[0081] In this embodiment of the invention, the perturbed source speech content representation, which serves as the query vector for the next round, is used as the key and value of the enhanced frame-level speaker representation for a second round of attention calculation. Guided by the content of the source speech content representation, the most relevant acoustic features are dynamically retrieved and aggregated from all frame-level features of the target speaker, thereby generating a content-aligned frame-level speaker representation. This frame-level speaker representation is then fused with the global speaker representation to obtain the final frame-level speaker representation, which is the target frame-level speaker representation.

[0082] Step S104: Sentence-level speaker embedding Target frame-level speaker representation and representation of source speech content The target speech is processed and output.

[0083] It should be noted that the specific implementation of step S104 includes the following steps.

[0084] Step S31: Embed sentence-level speaker and target frame-level speaker representation The first data is obtained by splicing them together;

[0085] In the specific implementation step S31, the sentence-level speaker is first embedded. and target frame-level speaker representation The first data is obtained by splicing them together;

[0086] Step S32: Represent the first data and the source speech content As input data, a preset speech generation model is input to process the input data based on the speech generation model and convert it into a Mel spectrogram.

[0087] It should be noted that the training process of the speech generation model is as follows:

[0088] First, learn from the source distribution arrive continuous flow , is determined by the ordinary differential equation ODE of formula (3);

[0089] Formula (3):

[0090]

[0091] Among them, continuous flow The value is , To predict the vector field, we need to learn the vector field related to time step t, where t satisfies... The model directly estimates this vector field along the path of the conditional probability distribution. Specifically, the optimal transport (OT) conditional flow matching method utilizes boundary conditions to introduce a simplified continuous flow. The form is as shown in formula (4).

[0092] Formula (4):

[0093]

[0094] in, This is the preset noise factor.

[0095] It follows a standard normal distribution. Let x0 be the normal distribution of the target speech signal, i.e., the target (converted Mel spectrum) distribution; x0 is... Sampling, for Sampling;

[0096] Next, during training, the vector field is predicted based on the downsampling module, intermediate block, and upsampling module. Then, using the current input data c as a condition, the prediction vector field is calculated. The mean square error between the optimal transport vector field and the loss function As shown in formula (5);

[0097] Formula (5):

[0098]

[0099] Where c represents the first data and the source speech content representation. , The learnable parameters of the preset neural network; in formula (5) and These represent standard Gaussian noise samples and real Mel spectrum samples, respectively.

[0100] Secondly, during training, conditions are replaced with learnable vectors with a certain probability to enhance the control of conditions over the generative model, thus obtaining vector fields under both conditional and unconditional conditions. During inference, the weighted sum of these two trained vector fields is used to gradually obtain target samples from noisy samples; finally, when the loss function is determined to be less than a preset threshold, the target sample is obtained based on the current loss function. The corresponding model parameters are used to construct the speech generation model.

[0101] In the specific implementation step S32, the deterministic mapping of the speech generation model from the standard normal distribution to the complex conditional distribution of the target Mel-spectrogram is invoked to directly generate the Mel-spectrograms after the input data transformation in a non-autoregressive manner.

[0102] The speech generation model shown in this application utilizes the powerful generation capability of the diffusion model to ensure high naturalness and prosodic fidelity of the generated speech under the same sampling efficiency.

[0103] Step S33: Call the preset vocoder to convert the Mel spectrogram into a time-domain waveform to obtain the target speech and output it.

[0104] In the specific implementation step S33, the generated Mel spectrogram is input into a pre-trained vocoder to synthesize the Mel spectrogram into the final time-domain waveform, obtain the target speech, and output it to complete the cross-language speaker conversion.

[0105] It should be noted that the specific implementation of steps S101 to S104 can also be as follows: Figure 2 Please provide an explanation.

[0106] In this embodiment of the invention, addressing the problems of low naturalness of converted speech, pronunciation errors, insufficient speaker modeling, inadequate similarity, and difficulty in cross-language alignment in existing zero-shot cross-language speaker conversion technologies, a zero-shot cross-language speaker conversion method based on query enhancement and conditional flow matching is proposed. This method obtains fine-grained, content-relevant speaker representations through query enhancement to overcome the deficiency of insufficient sentence-level speaker embedding information. It achieves efficient and stable non-autoregressive speech generation through a conditional flow matching model to improve prosodic naturalness and avoid the error accumulation problem of autoregressive models. By organically combining these two methods, the method ultimately generates high-quality, highly natural cross-language converted speech with high target speaker similarity under zero-shot conditions.

[0107] In this invention, a pre-defined speech generation model performs a deterministic mapping from a standard normal distribution to a complex conditional distribution of the target Mel-spectrogram, directly generating transformed Mel-spectrograms from the input data in a non-autoregressive manner. This model leverages the powerful generative capabilities of the diffusion model, ensuring high naturalness and prosodic fidelity of the generated speech while maintaining the same sampling efficiency. The generated Mel-spectrograms are then input into a pre-trained vocoder to synthesize the final time-domain waveform, yielding the target speech for cross-language speaker conversion.

[0108] In this invention, a query enhancement mechanism is used to obtain fine-grained, content-relevant frame-level speaker representations to overcome the deficiency of insufficient sentence-level speaker embedding information. Processing is performed based on sentence-level speaker embeddings, target frame-level speaker representations, and source speech content representations to achieve efficient and stable non-autoregressive speech generation, thereby improving prosodic naturalness and avoiding the error accumulation problem of autoregressive models. By organically combining the two, high-quality, highly natural cross-language converted speech with high target speaker similarity is finally generated under zero-sample conditions.

[0109] Based on the speech conversion method shown in the above embodiments of the present invention, correspondingly, the present invention also shows a speech conversion device, such as... Figure 3 As shown, the device includes:

[0110] Extraction unit 301 is used to extract the hidden layer representations of the source speech signal and the target reference speech signal respectively, so as to obtain the source speech content representation and the target reference speech content representation.

[0111] Enhancement processing unit 302 is used to perform first-level attention processing based on the target reference speech signal and the target reference speech content representation to determine the corresponding frame-level speaker representation; and to perform second-level attention processing based on the source speech content representation and the frame-level speaker representation to obtain the target frame-level speaker representation.

[0112] The generation unit 303 is used to process the target speech based on sentence-level speaker embedding, target frame-level speaker representation, and source speech content representation to obtain the target speech and output it.

[0113] Optionally, based on the speech conversion device shown in the above embodiments of the present invention, the extraction unit 301 is specifically used for:

[0114] The source speech signal is randomly offset to obtain the offset speech signal;

[0115] The feature extraction model is invoked to extract the hidden layer representations from the source speech content representation and the target reference speech signal, respectively, to obtain the source speech content representation and the target reference speech content representation.

[0116] Optionally, based on the speech conversion device shown in the above embodiments of the present invention, the first-level attention processing unit 302, which determines the corresponding frame-level speaker representation enhancement processing unit based on the target reference speech signal and the target reference speech content representation, is specifically used for:

[0117] Extract sentence-level speaker embeddings from the target reference speech signal;

[0118] Wavelet convolution is performed on the target reference speech content representation to obtain the frame-level speaker initial representation;

[0119] Attention processing is performed based on the sentence-level speaker embedding and the frame-level speaker initial representation to obtain the corresponding frame-level speaker representation.

[0120] Optionally, based on the speech conversion device shown in the above embodiments of the present invention, the enhancement processing unit 302, which performs second-level attention processing based on the source speech content representation and frame-level speaker representation to obtain the target frame-level speaker representation, is specifically used for:

[0121] Align the source speech content representation with the frame-level speaker representation using pronunciation features to obtain the target frame-level speaker representation.

[0122] Optionally, based on the speech conversion device shown in the above embodiments of the present invention, the generation unit 303 is specifically used for:

[0123] The first data is obtained by concatenating sentence-level speaker embeddings and target frame-level speaker representations.

[0124] The first data and the source speech content representation are used as input data and input into a preset speech generation model to process the input data based on the speech generation model and convert it into a Mel spectrogram.

[0125] The preset vocoder is invoked to convert the Mel spectrogram into a time-domain waveform to obtain the target speech, which is then output.

[0126] The specific principles and execution processes of each unit in the speech conversion device disclosed in the above embodiments of the present invention are the same as the corresponding contents in the speech conversion method provided in the above embodiments of the present invention. Please refer to the corresponding parts in the speech conversion method disclosed in the above embodiments of the present invention, and they will not be repeated here.

[0127] This application provides an electronic device, which includes a processor and a memory. The memory is used to store speech conversion program code and data, and the processor is used to call the program instructions in the memory to execute the steps shown in the speech conversion method in the above embodiments.

[0128] This invention provides a storage medium, namely a computer-readable storage medium, which includes the electronic device provided in the above-described embodiments of this application. The electronic device is used to execute the speech conversion method disclosed in the embodiments of this application.

[0129] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0130] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0131] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech conversion method, characterized in that, The method includes: The hidden layer representations of the source speech signal and the target reference speech signal are extracted respectively to obtain the source speech content representation and the target reference speech content representation; Based on the target reference speech signal and the target reference speech content representation, the first level of attention processing is performed to determine the corresponding frame-level speaker representation; A second-level attention process is performed based on the source speech content representation and the frame-level speaker representation to obtain the target frame-level speaker representation. The target speech is obtained and output by processing based on sentence-level speaker embedding, target frame-level speaker representation, and source speech content representation. The sentence-level speaker embedding is extracted from the target reference speech signal.

2. The method according to claim 1, characterized in that, Hidden layer representations are extracted from the source speech signal and the target reference speech signal respectively to obtain the source speech content representation and the target reference speech content representation, including: The source speech signal is randomly offset to obtain the offset speech signal; The feature extraction model is invoked to extract the hidden layer representations from the source speech content representation and the target reference speech signal, respectively, to obtain the source speech content representation and the target reference speech content representation.

3. The method according to claim 1, characterized in that, Based on the target reference speech signal and the target reference speech content representation, a first-level attention processing is performed to determine the corresponding frame-level speaker representation, including: Extract sentence-level speaker embeddings from the target reference speech signal; Wavelet convolution is performed on the target reference speech content representation to obtain the frame-level speaker initial representation; Attention processing is performed based on the sentence-level speaker embedding and the frame-level speaker initial representation to obtain the corresponding frame-level speaker representation.

4. The method according to claim 1, characterized in that, A second-level attention process is performed based on the source speech content representation and the frame-level speaker representation to obtain the target frame-level speaker representation, including: Align the source speech content representation with the frame-level speaker representation using pronunciation features to obtain the target frame-level speaker representation.

5. The method according to claim 1, characterized in that, The target speech is obtained and output based on sentence-level speaker embedding, target frame-level speaker representation, and source speech content representation, including: The first data is obtained by concatenating sentence-level speaker embeddings and target frame-level speaker representations. The first data and the source speech content representation are used as input data and input into a preset speech generation model to process the input data based on the speech generation model and convert it into a Mel spectrogram. The preset vocoder is invoked to convert the Mel spectrogram into a time-domain waveform to obtain the target speech, which is then output.

6. A voice conversion device, characterized in that, The device includes: The extraction unit is used to extract the hidden layer representations of the source speech signal and the target reference speech signal respectively, so as to obtain the source speech content representation and the target reference speech content representation. The enhancement processing unit is used to perform a first-level attention processing based on the target reference speech signal and the target reference speech content representation to determine the corresponding frame-level speaker representation; and to perform a second-level attention processing based on the source speech content representation and the frame-level speaker representation to obtain the target frame-level speaker representation. The generation unit is used to process the target speech based on sentence-level speaker embedding, target frame-level speaker representation, and source speech content representation to obtain the target speech and output it.

7. The apparatus according to claim 6, characterized in that, The extraction unit is specifically used for: The source speech signal is randomly offset to obtain the offset speech signal; The feature extraction model is invoked to extract the hidden layer representations from the source speech content representation and the target reference speech signal, respectively, to obtain the source speech content representation and the target reference speech content representation.

8. The apparatus according to claim 6, characterized in that, Based on the target reference speech signal and the target reference speech content representation, a first-level attention processing is performed to determine the corresponding frame-level speaker representation enhancement processing unit, specifically used for: Extract sentence-level speaker embeddings from the target reference speech signal; Wavelet convolution is performed on the target reference speech content representation to obtain the frame-level speaker initial representation; Attention processing is performed based on the sentence-level speaker embedding and the frame-level speaker initial representation to obtain the corresponding frame-level speaker representation.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory being used to store program code and data for data generation, and the processor being used to call program instructions in the memory to execute the speech conversion method as described in any one of claims 1-5.

10. A storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, it controls the device where the storage medium is located to perform the speech conversion method as described in any one of claims 1-5.

Images