A speech conversion method, device, equipment and readable storage medium

Abstract

The present invention provides a speech conversion method, device, equipment and readable storage medium. The method compresses and quantizes the audio to be converted and the reference audio through an audio quantizer to generate a semantic code of the audio to be converted and a reference code of the reference audio. Then, the semantic code and the reference code are aligned according to the length of their Mel spectrum to splice into a set of feature vectors. The audio large language model is called to perform autoregressive decoding on the feature vector to generate an output code. The Mel spectrum of the reference audio is input into a timbre style network to generate a perception vector. The output code and the perception vector are fused to generate a vector code. Then, a flow prediction network is called to process the vector code in combination with time series embedding and prior distribution to gradually generate a Mel spectrum. Finally, the Mel spectrum is restored to a target speech signal through a vocoder, thereby solving the problem that the existing small-scale model has an unsatisfactory restoration effect on subtle sound features.

Description

Technical Field

[0001] The present invention relates to the field of speech processing, and in particular to a speech conversion method, device, equipment and readable storage medium. Background Art

[0002] In the development of speech conversion technology, early methods mainly relied on a large amount of parallel corpus for training. These parallel corpora require different people to say the same words, and such training data is time-consuming and expensive to produce. In addition, due to the limitations of actual data collection, the types of sounds that can be converted are also very limited. In most cases, only a single fixed timbre can be converted, which cannot meet people's actual needs of converting to various sounds at will.

[0003] With the advancement of deep learning technology, SO-VITS based on the VITS framework and its improved version RVC have gradually become the mainstream voice conversion solutions. The advantages of these methods are that they have lower requirements on the quality of training data and can support the conversion of multiple timbres, and they perform well in practical applications. However, these methods also have some limitations. Due to the small size of the model, they cannot achieve the "zero-sample training" capability of imitating new sounds immediately after seeing them. At the same time, small-scale models cannot make full use of the existing massive open source voice data, which leads to less than ideal restoration effects when processing subtle sound features such as laughter and breathing during speaking.

[0004] In view of this, this application is filed. Summary of the invention

[0005] The present invention discloses a speech conversion method, device, equipment and readable storage medium, aiming to solve the problem that the existing small-scale model has an unsatisfactory effect in restoring subtle sound features.

[0006] The first aspect of the present invention provides a speech conversion method, comprising:

[0007] Compressing and quantizing the audio to be converted and the reference audio through an audio quantizer to generate a semantic code of the audio to be converted and a reference code of the reference audio, wherein the audio quantizer includes a Mel filter bank, an encoder based on a convolutional network, and a residual vector quantizer;

[0008] The semantic code and the reference code are aligned based on the Mel spectrum length of the semantic code and the Mel spectrum length of the reference code to splice them into a set of feature vectors, the audio large language model is called to perform autoregressive decoding on the feature vector to generate an output code, the Mel spectrum of the reference audio is input into a timbre style network to generate a perception vector, and the output code and the perception vector are fused to generate a vector code, wherein the audio large language model is a deep learning model based on a multi-layer Transformer, and the timbre style network includes a causal convolutional network and a cross-attention mechanism;

[0009] Input the vector encoding into a flow prediction network, combine it with time series embedding, and gradually generate a target Mel spectrum, wherein the flow prediction network includes a downsampling network, an intermediate network, and an upsampling network;

[0010] The target Mel spectrum is restored to a target speech signal through a vocoder.

[0011] Preferably, the audio to be converted and the reference audio are compressed and quantized by an audio quantizer to generate a semantic code of the audio to be converted and a reference code of the reference audio, specifically:

[0012] Performing Fourier transform on the audio to be converted and the reference audio, and extracting the Mel spectrum of the audio to be converted and the Mel spectrum of the reference audio;

[0013] Calling an encoder based on a convolutional network to simultaneously downsample the Mel spectrum of the audio to be converted and the Mel spectrum of the reference audio in a time dimension to generate continuous features of the audio to be converted and continuous features of the reference audio;

[0014] The continuous features of the audio to be converted and the continuous features of the reference audio are processed layer by layer through a residual vector quantizer to generate a semantic code of the audio to be converted and a reference code of the reference audio.

[0015] Preferably, the semantic code and the reference code are aligned according to the length of their Mel spectrum to splice into a set of feature vectors, the audio large language model is called to perform autoregressive decoding on the feature vector to generate an output code, the Mel spectrum of the reference audio is input into the timbre style network to generate a perception vector, and the output code and the perception vector are fused to generate a vector code, specifically:

[0016] Upsampling and interpolating the semantic code and the reference code respectively, and restoring their lengths to the lengths of the corresponding original Mel spectrum;

[0017] The processed reference code is used as a constraint condition and concatenated with the semantic code to generate a set of feature vectors;

[0018] Inputting the feature vector into a large audio language model including a multi-layer transformer, and generating an output code by autoregression;

[0019] The Mel spectrum of the reference audio is input into the causal convolutional network in the timbre style perception network for feature extraction, and the extracted features are processed using a cross-attention mechanism to obtain a latent vector, and the latent vector is normalized to obtain a perception vector;

[0020] The perception vector is fused with the output code to generate the vector code.

[0021] Preferably, the vector encoding is input into the stream prediction network, combined with time series embedding, to gradually generate the target Mel spectrum, specifically:

[0022] A random sample from the prior distribution, the vector encoding, and the time series embedding are used as inputs of the flow prediction network, and the vector encoding is used as a conditional average to control the generation of a target Mel spectrum.

[0023] Preferably, the target Mel spectrum is restored to a target speech signal through a vocoder, specifically:

[0024] The Mel spectrum is subjected to transposed convolution, and the output of the transposed convolution is processed using a plurality of convolution modules with different kernel sizes and dilation rates to capture audio features of different scales and granularities;

[0025] The output features of the convolution module are accumulated and fused, and the fused features are processed by a residual module to output a target speech signal.

[0026] A second aspect of the present invention provides a speech conversion device, comprising:

[0027] A compression and quantization unit, configured to compress and quantize the audio to be converted and the reference audio through an audio quantizer to generate a semantic code of the audio to be converted and a reference code of the reference audio, wherein the audio quantizer includes a Mel filter bank, an encoder based on a convolutional network, and a residual vector quantizer;

[0028] A vector encoding unit, configured to align the semantic code and the reference code based on the Mel spectrum length of the semantic code and the Mel spectrum length of the reference code to splice them into a set of feature vectors, call the audio large language model to perform autoregressive decoding on the feature vector to generate an output code, input the Mel spectrum of the reference audio into a timbre style network to generate a perception vector, and fuse the output code and the perception vector to generate a vector code, wherein the audio large language model is a deep learning model based on a multi-layer Transformer, and the timbre style network includes a causal convolutional network and a cross-attention mechanism;

[0029] A Mel spectrum generating unit, used for inputting the vector encoding into a flow prediction network, combining with time series embedding, and gradually generating a target Mel spectrum, wherein the flow prediction network includes a downsampling network, an intermediate network and an upsampling network;

[0030] The target speech signal generating unit is used to restore the target Mel spectrum into a target speech signal through a vocoder.

[0031] A third aspect of the present invention provides a speech conversion device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement a speech conversion method as described in any one of the above.

[0032] A fourth aspect of the present invention provides a computer-readable storage medium, characterized in that a computer program is stored therein, and the computer program can be executed by a processor of a device where the computer-readable storage medium is located to implement a speech conversion method as described in any one of the above items.

[0033] Based on a speech conversion method, device, equipment and readable storage medium provided by the present invention, the audio to be converted and the reference audio are first compressed and quantized by an audio quantizer to generate a semantic code of the audio to be converted and a reference code of the reference audio. Then, the semantic code and the reference code are aligned according to the length of their Mel spectrum to splice into a set of feature vectors, and an audio large language model is called to perform autoregressive decoding on the feature vector to generate an output code. The Mel spectrum of the reference audio is input into a timbre style network to generate a perception vector, and the output code and the perception vector are fused to generate a vector code. Then, the vector code is input into a stream prediction network, combined with time series embedding, to gradually generate a target Mel spectrum. Finally, the target Mel spectrum is restored to a target speech signal through a vocoder, thereby solving the problem that the existing small-scale model has an unsatisfactory restoration effect on subtle sound features. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1is a flow chart of a speech conversion method provided by the first embodiment of the present invention;

[0035] Figure 2 It is a schematic diagram of the quantization encoding process provided by the present invention;

[0036] Figure 3 It is a schematic diagram of the generation process of the vector code provided by the present invention;

[0037] Figure 4 It is a schematic diagram of the generation process of the Mel spectrum provided by the present invention;

[0038] Figure 5 It is a schematic diagram of the target speech signal generation process provided by the present invention;

[0039] Figure 6 It is a module schematic diagram of a speech conversion device provided by the second embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] In order to better understand the technical solution of the present invention, the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0042] It should be clear that the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0044] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0045] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0046] The "first\second" mentioned in the embodiments is only to distinguish similar objects, and does not represent a specific order for the objects. It is understandable that the "first\second" can be interchanged with the specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0047] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0048] The present invention discloses a speech conversion method, device, equipment and readable storage medium, aiming to solve the problem that the existing small-scale model has an unsatisfactory effect in restoring subtle sound features.

[0049] See also Figure 1 The first embodiment of the present invention provides a speech conversion method, which can be executed by a speech conversion device (hereinafter referred to as the conversion device), and in particular, by one or more processors in the conversion device to implement at least the following steps:

[0050] S101, compressing and quantizing the audio to be converted and the reference audio through an audio quantizer to generate a semantic code of the audio to be converted and a reference code of the reference audio;

[0051] In this embodiment, the conversion device may be a terminal with data processing capabilities such as a server, workstation, or desktop computer. The conversion device may be installed with a corresponding operating system and application software, and the functions required by this embodiment may be implemented through the combination of the operating system and application software.

[0052] Specifically, in this embodiment, please combine Figure 2 , the audio to be converted and the reference audio are respectively transformed from time domain signals to frequency domain signals through Fourier transform. By decomposing the audio signal into different frequency components, the audio signal after Fourier transform is further processed by Mel filter bank to generate Mel spectrum. Mel spectrum is an audio representation method that simulates the perception characteristics of human ears. It can effectively retain the semantic and timbre characteristics of audio while greatly reducing redundant information.

[0053] Next, the encoder based on the convolutional network is called to process the Mel spectrum of the audio to be converted and the reference audio. The encoder adopts a multi-layer convolutional neural network (CNN) structure, which can not only deeply extract the local features of the audio, but also capture the dynamic changes of the audio features over time through the sliding operation of the convolution kernel. In addition, the encoder can downsample in the time dimension, that is, reduce the data dimension by reducing the input time step. The downsampling operation can reduce the computational complexity of the model while retaining the main information in the Mel spectrum. For the audio to be converted and the reference audio, the above processing generates continuous feature representations, where the continuous features of the audio to be converted mainly describe the semantic content, while the continuous features of the reference audio retain the timbre style information.

[0054] After the continuous features are generated, they are processed layer by layer through residual vector quantizer alignment, so that the continuous features are encoded into discrete features, and semantic codes and reference codes are generated respectively. The residual vector quantizer mainly performs hierarchical quantization. Specifically: each layer of sub-quantizer processes the residual information of the previous layer and gradually approaches the output features of the encoder. The layer-by-layer quantization method can not only efficiently represent complex audio features, but also significantly reduce information loss, so that the final generated code is highly compressible and accurate. Specifically, the quantization process maps continuous features to a finite set of discrete vectors.

[0055] It is worth noting that through the above processing, the semantic coding of the audio to be converted and the reference coding of the reference audio focus on features of different dimensions respectively. The semantic coding emphasizes the semantic information of the audio content, while the reference coding highlights the timbre characteristics of the audio. Feature separation can significantly improve the flexibility and accuracy of speech conversion. For example, in the subsequent speech generation process, the model can generate speech outputs with different timbres by adjusting the reference coding, thereby meeting diverse application requirements. In addition, the residual vector quantizer can avoid the problem of audio information loss that may be caused by traditional quantization methods.

[0056] S102, aligning the semantic code and the reference code according to the length of their Mel spectrum to splice into a set of feature vectors, calling the audio large language model to perform autoregressive decoding on the feature vector to generate an output code, inputting the Mel spectrum of the reference audio into a timbre style network to generate a perception vector, and fusing the output code and the perception vector to generate a vector code;

[0057] In this embodiment, please combine Figure 3In order to ensure the consistency of the semantic code and the reference code in time series, both need to be upsampled and interpolated. The main purpose is to restore the compressed code length to a state that matches its original Mel spectrum length, thereby ensuring that the two codes are fully aligned in the time dimension. Upsampling increases the time step by inserting intermediate data points, and interpolation further optimizes the values ​​of these newly added points so that they smoothly transition to the values ​​of adjacent points.

[0058] Next, the processed reference code is used as a constraint to concatenate with the semantic code to generate a set of feature vectors. During the concatenation process, the reference code is used to guide the timbre characteristics of the generated output, while the semantic code retains the semantic content of the audio to be converted. The conditional control can accurately combine the target timbre with the semantic information, and the generated feature vector is represented by its multi-dimensional features.

[0059] Next, the concatenated feature vectors are input into the Audio Large Language Model (ALLM) for processing. ALLM is a large-scale deep learning model based on a multi-layer Transformer structure with powerful feature understanding and generation capabilities. Its core lies in generating output codes through autoregressive decoding. During the processing, the ALLM model gradually predicts the output features of the current time step. Each prediction step is based on the previously generated features and input conditions, thereby ensuring that the generated output codes are consistent in time series and can accurately reflect the semantic and timbre characteristics of the input data.

[0060] In order to further improve the timbre matching of the output, this embodiment also introduces a timbre style perception network, which is specifically used to extract potential timbre features from reference audio. Specifically, the Mel spectrum of the reference audio is input into the causal convolution network in the timbre style perception network, and the causal convolution can effectively capture the time series dependency of the audio features. Subsequently, the extracted features are further processed by the cross-attention mechanism to generate a latent vector. The cross-attention mechanism can give higher weights to key features when extracting features, while suppressing noise or irrelevant information, so that the latent vector can accurately represent the timbre characteristics of the reference audio. Finally, the generated latent vector is normalized to obtain a stable perception vector.

[0061] In the final fusion stage, the perceptual vector is combined with the output code to generate the final vector code. The addition of the perceptual vector injects precise timbre characteristics into the output code, so that the generated vector code not only semantically maintains the content of the audio to be converted, but also presents the timbre characteristics of the reference audio with high fidelity.

[0062] S103, inputting the vector code into a stream prediction network, combining it with time series embedding, and gradually generating a target Mel spectrum;

[0063] In this embodiment, please combine Figure 4 , random sampling in the prior distribution provides the initial state for Mel spectrum generation. Random initialization can simulate the natural generation process of Mel spectrum and provide a random starting point for the network's denoising process. Vector encoding, as a conditional average, plays a role in constraining the generation of Mel spectrum throughout the process. Specifically, vector encoding retains the comprehensive feature information extracted from semantic encoding and reference encoding. As a generation condition, it can effectively control the generation result so that it not only conforms to the semantic features of the input, but also accurately matches the style of the reference timbre. Time series embedding injects information of the time dimension into the generation process. Time information is crucial for capturing the temporal changes of syllables and pitch in speech, and can help the network generate a coherent and natural spectral structure.

[0064] The flow prediction network is based on a U-shaped structure and consists of three main parts: a downsampling network, an intermediate network, and an upsampling network. The downsampling network gradually reduces the time dimension of the input data through a series of convolution operations, while extracting the global representation of the features. It can effectively capture the potential patterns of the Mel spectrum in the low-dimensional space while compressing the input feature quantity. The intermediate network consists of a residual network and a Transformer module. The residual structure enhances the efficiency of feature transmission, allowing the network to retain the detailed information of the input data in deep processing. At the same time, the Transformer module uses the self-attention mechanism to further optimize the processing of time series features, ensuring that the generated Mel spectrum has a high degree of semantic consistency.

[0065] In the upsampling network, the flow prediction network gradually restores the time dimension of the data, and restores the details of the Mel spectrum layer by layer through the inverse convolution operation. Among them, the vector encoding is continuously used as a conditional average to adjust the output of the network, so that the spectral features generated at each step can maintain consistency with the input features. The generation process adopts a step-by-step denoising method, gradually optimizing from the initial randomly sampled rough spectrum to a high-resolution clear spectrum. Through this progressive generation method, the network can correct the deviation of spectral features at multiple scales, thereby ensuring that the generated Mel spectrum can not only truly reflect the content information of the speech, but also accurately reproduce the style characteristics of the reference timbre.

[0066] It is worth mentioning that the flow prediction network in this embodiment further improves the dynamics and coherence of the generated spectrum by introducing time series embedding into the generation process. Time series embedding adds time information to each input data point in units of time steps, allowing the network to perceive the temporal structure of speech, thereby better capturing the temporal features of speech such as syllables, stress, and intonation. It is suitable for speech application scenarios that require precise time synchronization, such as multi-language translation speech synthesis and video dubbing, and can significantly improve the match between the generated speech and the original content.

[0067] S104, restoring the target Mel spectrum to a target speech signal through a vocoder.

[0068] In this embodiment, please combine Figure 5 , and input the generated Mel spectrum into the transposed convolution module of the vocoder. The main purpose is to gradually restore the spectral information from the feature space to the waveform space. The transposed convolution amplifies the time step and frequency range of the input through the reverse operation, which is equivalent to an upsampling process. On the basis of the transposed convolution, the vocoder further adopts multiple convolution modules with different kernel sizes and dilation rates. These convolution modules process the input data through different receptive fields and can capture features of different scales in the audio signal. For example, small kernel convolution is good at processing short-term high-frequency information, while large kernel convolution and dilated convolution can capture long-term low-frequency characteristics and global changes. The feature information in the audio signal can be extracted to the maximum extent, thereby generating more expressive waveform data.

[0069] Based on the multi-scale features generated by the convolution module, the vocoder integrates these features through cumulative fusion operations. The cumulative fusion process is equivalent to the weighted summation of the features output by different convolution modules, so that the multi-level information of the audio signal can be organically combined. It can effectively eliminate the redundant information that may exist between different convolution results, while enhancing the complementarity between features, thereby generating a more accurate audio representation. In addition, the vocoder also introduces a residual module to further optimize the fused features. The residual module enables the network to more efficiently transmit and utilize feature information by adding directly connected jump paths, avoiding the problem of feature degradation or loss in deep networks.

[0070] Throughout the audio restoration process, the structure and operation of the vocoder ensure that the generated target speech signal can restore the characteristics of the input Mel spectrum as realistically as possible. For example, the vocoder can accurately reproduce the short-term impact characteristics (such as consonants) and long-term smooth characteristics (such as vowels) in the audio signal, making the output speech more natural and clear. At the same time, through multi-scale processing and residual optimization, the vocoder's adaptability to complex audio scenes (such as multi-timbre and multi-emotional expression) is significantly improved.

[0071] Compared with the existing technology:

[0072] After the speech conversion is completed, the current mainstream neural network speech conversion method So-VITS is used to perform speech conversion to obtain the converted speech.

[0073] Manual blind testing was used to conduct subjective evaluation of speech similarity (SIM), and automatic speech recognition was used to conduct objective evaluation of word error rate (WER) to compare the effect differences between the two methods.

[0074] In terms of test data, internal business data was used for comparative testing. The conversion method of this embodiment has significantly improved the error rate and similarity compared with the mainstream neural network speech conversion method So-VITS. SIM is the subjective similarity of real-person evaluation. The higher the parameter, the more similar it is. WER is the error rate. The lower the parameter, the lower the speech conversion error rate.

[0075]

[0076] In actual use, thanks to the modeling method of decoupling semantics and timbre separately in this embodiment, compared with the traditional black box method, even without obtaining a large amount of data of the timbre to be converted for pre-training, voice conversion with a high degree of similarity can be achieved based on the timbre features obtained from the reference audio. It is also possible to achieve multi-timbre voice conversion by replacing reference audio with different timbres.

[0077] See also Figure 6 The second aspect of the present invention provides a speech conversion device, comprising:

[0078] A compression and quantization unit 201 is used to compress and quantize the audio to be converted and the reference audio through an audio quantizer to generate a semantic code of the audio to be converted and a reference code of the reference audio, wherein the audio quantizer includes a Mel filter bank, an encoder based on a convolutional network, and a residual vector quantizer;

[0079] A vector encoding unit 202 is used to align the semantic code and the reference code based on the Mel spectrum length of the semantic code and the Mel spectrum length of the reference code to splice them into a set of feature vectors, call the audio large language model to perform autoregressive decoding on the feature vector to generate an output code, input the Mel spectrum of the reference audio into a timbre style network to generate a perception vector, and fuse the output code and the perception vector to generate a vector code, wherein the audio large language model is a deep learning model based on a multi-layer Transformer, and the timbre style network includes a causal convolutional network and a cross-attention mechanism;

[0080] A Mel spectrum generating unit 203 is used to input the vector encoding into a flow prediction network, and combine the time series embedding to gradually generate a target Mel spectrum, wherein the flow prediction network includes a downsampling network, an intermediate network and an upsampling network;

[0081] The target speech signal generating unit 204 is configured to restore the target Mel spectrum into a target speech signal through a vocoder.

[0082] A third aspect of the present invention provides a speech conversion device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement a speech conversion method as described in any one of the above.

[0083] A fourth aspect of the present invention provides a computer-readable storage medium, characterized in that a computer program is stored therein, and the computer program can be executed by a processor of a device where the computer-readable storage medium is located to implement a speech conversion method as described in any one of the above items.

[0084] Based on a speech conversion method, device, equipment and readable storage medium provided by the present invention, the audio to be converted and the reference audio are first compressed and quantized by an audio quantizer to generate a semantic code of the audio to be converted and a reference code of the reference audio. Then, the semantic code and the reference code are aligned according to the length of their Mel spectrum to splice into a set of feature vectors, and a large audio language model is called to perform autoregressive decoding on the feature vector to generate a vector code; then, the vector code is input into a stream prediction network, combined with time series embedding, to gradually generate a target Mel spectrum; finally, the target Mel spectrum is restored to a target speech signal through a vocoder, thereby solving the problem that the existing small-scale model has an unsatisfactory restoration effect on subtle sound features.

[0085] Exemplarily, the computer program described in the third embodiment and the fourth embodiment of the present invention can be divided into one or more modules, and the one or more modules are stored in the memory and executed by the processor to complete the present invention. The one or more modules can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the device for implementing a speech conversion. For example, the device described in the second embodiment of the present invention.

[0086] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the speech conversion method, and uses various interfaces and lines to connect the various parts of the speech conversion method.

[0087] The memory can be used to store the computer program and / or module, and the processor realizes various functions of a speech conversion method by running or executing the computer program and / or module stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, a text conversion function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, text message data, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0088] Wherein, if the implemented module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0089] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.

[0090] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A speech conversion method, characterized in that: include The audio to be converted and the reference audio are compressed and quantized by an audio quantizer to generate a semantic code of the audio to be converted and a reference code of the reference audio, specifically: Fourier transform is performed on the audio to be converted and the reference audio, and the Mel spectrum of the audio to be converted and the Mel spectrum of the reference audio are extracted; an encoder based on a convolutional network is called to simultaneously perform downsampling on the Mel spectrum of the audio to be converted and the Mel spectrum of the reference audio in the time dimension to generate continuous features of the audio to be converted and the continuous features of the reference audio; the continuous features of the audio to be converted and the continuous features of the reference audio are processed layer by layer by a residual vector quantizer to generate a semantic code of the audio to be converted and a reference code of the reference audio; wherein the audio quantizer includes a Mel filter bank, an encoder based on a convolutional network, and a residual vector quantizer; The semantic code and the reference code are aligned based on the Mel spectrum length of the semantic code and the Mel spectrum length of the reference code to splice them into a set of feature vectors, the audio large language model is called to perform autoregressive decoding on the feature vector to generate an output code, the Mel spectrum of the reference audio is input into the timbre style network to generate a perception vector, and the output code and the perception vector are fused to generate a vector code, specifically: the semantic code and the reference code are upsampled and interpolated respectively to restore their lengths to the corresponding original Mel spectrum lengths; the processed reference code is used as a restriction condition and spliced ​​with the semantic code to generate a set of feature vectors. vector; input the feature vector into an audio language model including a multi-layer transformer, and generate an output code by autoregression; input the Mel spectrum of the reference audio into a causal convolutional network in a timbre style perception network for feature extraction, and use a cross-attention mechanism to process the extracted features and obtain a potential vector, and normalize the potential vector to obtain a perception vector; fuse the perception vector with the output code to generate the vector code; wherein the audio language model is a deep learning model based on a multi-layer transformer, and the timbre style network includes a causal convolutional network and a cross-attention mechanism; The vector code is input into the flow prediction network, combined with the time series embedding, and the target Mel spectrum is gradually generated, specifically: a random sample in the prior distribution, the vector code, and the time series embedding are used as inputs of the flow prediction network, and the vector code is used as a conditional average to control the generation of the target Mel spectrum, wherein the flow prediction network includes a downsampling network, an intermediate network, and an upsampling network; The target Mel spectrum is restored to a target speech signal through a vocoder.

2. A speech conversion method according to claim 1, characterized in that: The target Mel spectrum is restored to the target speech signal through a vocoder, specifically: The Mel spectrum is subjected to transposed convolution, and the output of the transposed convolution is processed using a plurality of convolution modules with different kernel sizes and dilation rates to capture audio features of different scales and granularities; The output features of the convolution module are accumulated and fused, and the fused features are processed by a residual module to output a target speech signal.

3. A speech conversion device, characterized in that: include: A compression and quantization unit, used for compressing and quantizing the audio to be converted and the reference audio through an audio quantizer to generate a semantic code of the audio to be converted and a reference code of the reference audio, specifically used for: performing Fourier transform on the audio to be converted and the reference audio, and extracting the Mel spectrum of the audio to be converted and the Mel spectrum of the reference audio; calling an encoder based on a convolutional network to simultaneously perform downsampling on the Mel spectrum of the audio to be converted and the Mel spectrum of the reference audio in the time dimension to generate continuous features of the audio to be converted and the continuous features of the reference audio; performing layer-by-layer processing on the continuous features of the audio to be converted and the continuous features of the reference audio through a residual vector quantizer to generate a semantic code of the audio to be converted and a reference code of the reference audio; wherein the audio quantizer includes a Mel filter bank, an encoder based on a convolutional network, and a residual vector quantizer; A vector encoding unit is used to align the semantic code and the reference code based on the Mel spectrum length of the semantic code and the Mel spectrum length of the reference code to splice them into a set of feature vectors, call the audio large language model to perform autoregressive decoding on the feature vector to generate an output code, input the Mel spectrum of the reference audio into a timbre style network to generate a perception vector, and fuse the output code and the perception vector to generate a vector code, specifically used to: upsample and interpolate the semantic code and the reference code respectively, and restore their lengths to the corresponding original Mel spectrum lengths; use the processed reference code as a restriction condition and splice it with the semantic code to Generate a set of feature vectors; input the feature vectors into an audio language model including a multi-layer transformer, and generate output codes by autoregression; input the Mel spectrum of the reference audio into the causal convolutional network in the timbre style perception network for feature extraction, and use the cross-attention mechanism to process the extracted features and obtain the latent vector, and normalize the latent vector to obtain the perception vector; fuse the perception vector with the output code to generate the vector code; wherein the audio language model is a deep learning model based on a multi-layer transformer, and the timbre style network includes a causal convolutional network and a cross-attention mechanism; A Mel spectrum generating unit is used to input the vector code into a flow prediction network, and combine the time series embedding to gradually generate a target Mel spectrum, specifically used to: use a random sample in a prior distribution, the vector code, and the time series embedding as inputs of the flow prediction network, and use the vector code as a conditional average value to control the generation of the target Mel spectrum, wherein the flow prediction network includes a downsampling network, an intermediate network, and an upsampling network; The target speech signal generating unit is used to restore the target Mel spectrum into a target speech signal through a vocoder.

4. A speech conversion device, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement a speech conversion method as described in any one of claims 1 to 2.

5. A computer-readable storage medium, characterized in that: A computer program is stored, and the computer program can be executed by a processor of the device where the computer-readable storage medium is located to implement a speech conversion method as described in any one of claims 1 to 2.

Images