Tone conversion method, apparatus and device, storage medium and program product

Abstract

A tone conversion method, apparatus and device, a storage medium and a program product. The method comprises: acquiring an original audio, and, by means of an arranged first encoding module and a tone vector, encoding the original audio to obtain first audio data (S101); performing vector quantization processing on the first audio data, so as to obtain corresponding embedding vector information (S102); and sending the embedding vector information to a decoding end for the decoding end to generate second audio data on the basis of the tone vector and the embedding vector information, and decoding the second audio data by means of an arranged decoding module, so as to obtain a voice-changed audio (S103). The method employs tone conversion services at clients while ensuring the tone conversion quality, thus reducing server costs.

Description

Tone conversion method, device, equipment, storage medium and program product

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 21, 2023, with application number 202311054309.4, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of audio processing technology, and in particular to a timbre conversion method, apparatus, device, storage medium, and program product. Background Art

[0003] Voice conversion is a crucial technology in the audio field, widely used in audio content generation, entertainment audio production, and secure communication. Voice conversion transforms the timbre of the original audio to a target timbre while preserving the original audio's content. Codec technology plays a vital role in many fields, widely used in audio, video, and compression. For example, audio transmission involves encoding audio to generate an intermediate representation. After network transmission, the intermediate representation is decoded to restore the audio.

[0004] Traditional AI voice-changing services are generally deployed on servers to ensure the quality of the voice conversion. However, if the model is reduced in size and placed on the client to save costs, the quality of the voice conversion will be significantly reduced. Currently, there is no relevant technology that can solve the problem of ensuring good voice conversion quality while extremely saving costs.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a timbre conversion method, apparatus, device, storage medium and program product, which deploy the timbre conversion encoding process on the client while ensuring the quality of the timbre conversion, thereby reducing server costs.

[0007] In a first aspect, an embodiment of the present application provides a method for converting timbre, the method comprising:

[0008] Obtaining original audio, and encoding the original audio using a set first encoding module and a timbre vector to obtain first audio data;

[0009] Performing vector quantization processing on the first audio data to obtain corresponding embedded vector information;

[0010] The embedded vector information is sent to a decoding end, so that the decoding end generates second audio data based on the timbre vector and the embedded vector information, and decodes the second audio data to obtain voice-changed audio through a set decoding module.

[0011] In a second aspect, an embodiment of the present application further provides a timbre conversion device, comprising:

[0012] A first encoding module is configured to obtain original audio, and encode the original audio based on the set timbre vector to obtain first audio data;

[0013] A vector quantization processing module configured to perform vector quantization processing on the first audio data to obtain corresponding embedded vector information;

[0014] The sending module is configured to send the embedded vector information to the decoding end, so that the decoding end generates second audio data based on the timbre vector and the embedded vector information, and decodes the second audio data through the set decoding module to obtain voice-changed audio.

[0015] In a third aspect, an embodiment of the present application further provides a timbre conversion method, which is applied to a decoding end and includes:

[0016] Receiving embedded vector information sent by the client, encoding the embedded vector information using a set second encoding module and a timbre vector to obtain second audio data;

[0017] The second audio data is decoded by a set decoding module to obtain voice-changed audio.

[0018] In a fourth aspect, an embodiment of the present application further provides a timbre conversion device, comprising:

[0019] A receiving module configured to receive embedding vector information sent by a client;

[0020] A second encoding module is configured to encode the embedded vector information using the set timbre vector to obtain second audio data;

[0021] The decoding module is configured to decode the second audio data to obtain voice-changed audio.

[0022] In a fifth aspect, an embodiment of the present application further provides a timbre conversion device, the device comprising:

[0023] one or more processors;

[0024] a storage device for storing one or more programs,

[0025] When the one or more programs are executed by the one or more processors, the one or more processors implement the timbre conversion method described in the embodiment of the present application.

[0026] In a sixth aspect, an embodiment of the present application further provides a non-volatile storage medium storing computer-executable instructions, which, when executed by a computer processor, are used to execute the timbre conversion method described in the embodiment of the present application.

[0027] In the seventh aspect, an embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor of the device reads and executes the computer program from the computer-readable storage medium, so that the device performs the timbre conversion method described in the embodiment of the present application.

[0028] In an embodiment of the present application, in the client, the original audio is obtained, and the original audio is encoded by the set first encoding module and timbre vector to obtain first audio data, the first audio data is vector quantized to obtain corresponding embedded vector information, and the embedded vector information is sent to the decoding end, for the decoding end to generate second audio data based on the timbre vector and the embedded vector information, and the second audio data is decoded by the set decoding module to obtain voice-changed audio. This timbre conversion method combines timbre conversion with audio encoding, and deploys the timbre conversion function on the client, which reduces the server cost and alleviates the data processing pressure of the server. In the above method, timbre conversion is combined with the audio encoding stage, and the processing of timbre vectors is combined in the audio encoding process, which reduces the deployment cost of timbre conversion while ensuring the quality of timbre conversion. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] FIG1 is a flow chart of a timbre conversion method provided by an embodiment of the present application;

[0030] FIG2 is a flow chart of another timbre conversion method provided in an embodiment of the present application;

[0031] FIG3 is a schematic diagram of an audio encoding process provided in an embodiment of the present application;

[0032] FIG4 is a flow chart of another timbre conversion method provided in an embodiment of the present application;

[0033] FIG5 is a flow chart of another timbre conversion method provided in an embodiment of the present application;

[0034] FIG6 is a structural block diagram of a timbre conversion device provided in an embodiment of the present application;

[0035] FIG7 is a block diagram of another timbre conversion device provided in an embodiment of the present application;

[0036] FIG8 is a schematic structural diagram of a timbre conversion device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The following is a further detailed description of the embodiments of the present application in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the embodiments of the present application, and are not intended to limit the embodiments of the present application. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions of the embodiments of the present application, rather than all structures.

[0038] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0039] The timbre conversion method provided in the embodiments of the present application can be executed by a client, which can be a smartphone, tablet computer, laptop computer, etc. The timbre conversion method can be applied to scenarios where the original audio needs to be voice-changed, such as audio content generation, entertainment audio production, and confidential calls, to convert the original audio into the target audio after timbre conversion.

[0040] FIG1 is a flow chart of a timbre conversion method provided in an embodiment of the present application, as shown in FIG1 , which specifically includes the following steps:

[0041] Step S101: Obtain original audio, and encode the original audio using a set first encoding module and timbre vector to obtain first audio data.

[0042] Among them, the original audio is the audio that needs to be converted into timbre, which can be the user's audio collected by the client microphone, or the audio stored by the client, etc. Optionally, the timbre conversion method in this solution is AI voice changing. Among them, the timbre vector is the audio vector set according to different voice changing effects when the original audio is voice-changed. For example, the timbre vector that is pre-set for different voice changing requirements such as confidential calls and children's voices can be used to convert the timbre of the original audio.

[0043] Among them, the first encoding module can adopt a flow-based generative model, such as the wn network module in WaveGlow. Optionally, the first encoding module takes the original audio as input and compresses the input spectrum as much as possible. It uses convolution to build the network in a residual manner. Optionally, the first encoding module includes multiple convolutional layers, each of which includes a causal convolutional neural network to achieve the purpose of zero delay. At the same time, groups=4 parameters are added to each layer of convolution, so that the number of network parameters is reduced and the number of layers is deeper to achieve the purpose of being able to represent more information. Its input is a linear spectrum, and exemplary parameters are: filter_length (filter length) = 1024, hop_length (hop length) = 320, win_length (window length) = 640, sample_reate (sampling rate) = 16000.

[0044] Optionally, the first encoding module input dimension is: [batch_size, in_dim, T], and the first encoding module output dimension is: [batch_size, out_dim, T / scala], where scala is the compression factor of the time dimension, and exemplarily scala = 1. Among them, batch_size refers to the number of data input to the first encoding module at one time. The number of data input at the same time is batch_size*in_dim, and the number of data output is batch_size*out_dim. The parameter T represents different time.

[0045] In one embodiment, encoding the original audio using the first encoding module and the timbre vector to obtain the first audio data may include, during the encoding process by the first encoding module, superimposing the output of a convolutional layer with the timbre vector to obtain the first audio data. Alternatively, the timbre vector may be superimposed on the output of each convolutional layer or selected convolutional layers.

[0046] Step S102: Perform vector quantization processing on the first audio data to obtain corresponding embedded vector information.

[0047] In one embodiment, after obtaining the first audio data with the superimposed timbre vector, the first audio data is vector quantized to obtain the corresponding embedded vector information. This step can be performed by a set vq module. Optionally, a residual vq module is used to input the first audio data into the residual vq module for residual vector quantization to obtain index information output by each layer, wherein the index information of each layer corresponds to the embedded vector of the corresponding index value in the set code table. In the residual vq module, each layer maintains a corresponding code table (codebook). Optionally, the codebook dimension is set to 128, the codebook size of each layer is 256=2^8, codebook-bit=8, and the number of layers n_q=8. Based on the above-set parameters, it can be seen that the code rate of the model is: sample_rate / hope_length*codebook-bit*n_q=16000 / 320*8*8=3.2kbps. The input of the residual vq module is the output of the first encoding module. For each frame of the first audio data output by the first encoding module, each layer of the residual vq module is represented by an embedded vector, that is, each frame of the first audio data is represented in a residual manner. At this time, the information output by the residual vq module is embedded vector information, which is used to vectorize the first audio data. Optionally, the embedded vector information can be the index information of each layer of the residual vq module. The index information can be determined by querying the code table to determine the corresponding embedded vector. The index information is transmitted during the transmission process to reduce the amount of data transmission.

[0048] Step S103: Send the embedded vector information to a decoding end, so that the decoding end generates second audio data based on the timbre vector and the embedded vector information, and decodes the second audio data to obtain voice-changed audio through a set decoding module.

[0049] In one embodiment, after the embedded vector information is obtained, it is sent to a decoding end, which can be another client that needs to play the voice-changed audio. The decoding end is configured with a decoding module to decode the voice-changed audio. Optionally, the client can integrate both a first encoding module and a decoding module to perform timbre conversion processing on the output audio and decode the received timbre-changed audio to obtain the voice-changed audio.

[0050] In one embodiment, when decoding, the decoding end first generates second audio data based on the timbre vector and the embedded vector information, and then performs decoding by the decoding module to obtain the voice-changed audio.

[0051] As can be seen from the above, in the client, the original audio is obtained, and the original audio is encoded by the set first encoding module and timbre vector to obtain the first audio data, the first audio data is vector quantized to obtain the corresponding embedded vector information, and the embedded vector information is sent to the decoding end, for the decoding end to generate the second audio data based on the timbre vector and the embedded vector information, and the second audio data is decoded by the set decoding module to obtain the voice-changed audio. This timbre conversion method combines timbre conversion with audio encoding, and deploys the timbre conversion function on the client, which reduces the server cost and alleviates the data processing pressure of the server. In the above method, the timbre conversion is combined with the audio encoding stage, and the processing of the timbre vector is combined in the audio encoding process, which reduces the deployment cost of the timbre conversion while ensuring the quality of the timbre conversion.

[0052] FIG2 is a flow chart of another timbre conversion method provided in an embodiment of the present application, as shown in FIG2 , including:

[0053] Step S201: Obtain original audio, and encode the original audio through a set first encoding module. During the encoding process, the output of each convolution layer is superimposed with the set timbre vector as the input of the next convolution layer to obtain first audio data.

[0054] In one embodiment, during the encoding process of the original audio, the output of each convolutional layer is superimposed with the set timbre vector as the input of the next convolutional layer to obtain the first audio data. For example, as shown in Figure 3, Figure 3 is a schematic diagram of an audio encoding process provided by an embodiment of the present application. The encoding module is composed of multiple convolutional layers, and each convolutional layer includes a causal convolutional neural network (causal cnn). In the network model structure, the timbre vector (speaker-embedding) set on the output of each convolutional layer is superimposed as the input of the next layer. Optionally, the dimension of the output data of the convolutional layer is the same as the dimension of the timbre vector, and the data of each dimension is added accordingly to complete the superposition of the timbre vector.

[0055] Step S202: Perform vector quantization processing on the first audio data to obtain corresponding embedded vector information.

[0056] Step S203: Send the embedded vector information to a decoding end, so that the decoding end generates second audio data based on the timbre vector and the embedded vector information, and decodes the second audio data to obtain voice-changed audio through a set decoding module.

[0057] As can be seen from the above, in the client, the original audio is obtained, and the original audio is encoded by the set first encoding module and timbre vector to obtain the first audio data, the first audio data is vector quantized to obtain the corresponding embedded vector information, and the embedded vector information is sent to the decoding end, for the decoding end to generate the second audio data based on the timbre vector and the embedded vector information, and the second audio data is decoded by the set decoding module to obtain the voice-changed audio. This timbre conversion method combines timbre conversion with audio encoding, and deploys the timbre conversion function on the client, which reduces the server cost and alleviates the data processing pressure of the server. In the above method, the timbre conversion is combined with the audio encoding stage, and the processing of the timbre vector is combined in the audio encoding process, which reduces the deployment cost of the timbre conversion while ensuring the quality of the timbre conversion.

[0058] FIG4 is a flow chart of another timbre conversion method provided in an embodiment of the present application, which is applied to a decoding end and specifically includes:

[0059] Step S301: Receive embedded vector information sent by the client, and encode the embedded vector information through the set second encoding module and timbre vector to obtain second audio data.

[0060] During decoding at the decoding end, the embedded vector information is first encoded using a second encoding module and a timbre vector to obtain second audio data. Optionally, the module architecture of the second encoding module is the same as that of the first encoding module, and specifically can be a scaled-down version of the first encoding module, i.e., the number of convolutional layers for each module, such as the first encoding module, is smaller than that of the first encoding module. The second encoding module can also include multiple convolutional layers, each of which includes a causal convolutional neural network.

[0061] Optionally, the encoding process of the embedded vector information with the timbre vector by the second encoding module may include superimposing the timbre vector with the embedding vector corresponding to the embedded vector information by the second encoding module to obtain the second audio data. Optionally, when the embedded vector information is index information, the embedding vector corresponding to the index information is queried based on a set code table, and the timbre vector and the embedding vector are superimposed by the second encoding module to obtain the second audio data. The processing by the second encoding module obtains audio data further superimposed with the timbre vector, i.e., the second audio data.

[0062] Step S302: Decode the second audio data by a set decoding module to obtain voice-changed audio.

[0063] In one embodiment, the decoding module is used to decode the output of the second encoding module, such as decoding the spectrum into a WAV file. Optionally, the decoding module can be modified based on the HifiGAN model. Optionally, the convolution in the HifiGAN model is replaced with a causal CNN, and the upsampling factor of the generator is changed to CausalConvTranspose1d. The network structure is reduced, and the optimal parameters are found through multiple experiments:

[0064] resblock_kernel_sizes:[5,3,3]

[0065] resblock_dilation_sizes:[[1,3,3],[1,3,3],[1,3,3]]

[0066] upsample_rates:[4,8,5,2]

[0067] Experimental data testing shows that the modified HifiGan framework has only 5,000 parameters (neural network size), while the original model has 141M parameters. Comparing the performance of the spectrum-to-WAV file conversion trained separately, the modified framework achieves comparable results, with no loss in timbre conversion quality. The timbre conversion model deployed in this way has a total size of 5M, which can be deployed on 80% of mobile devices without server involvement, enabling offline AI voice changing.

[0068] As can be seen from the above, at the decoding end, by receiving the embedded vector information sent by the client, the embedded vector information is encoded and processed by the set second encoding module and the timbre vector to obtain the second audio data, and then the second audio data is decoded by the set decoding module to obtain the voice-changed audio. This timbre conversion method combines timbre conversion with audio decoding, and the timbre conversion function can be deployed on the client, which reduces the server cost and alleviates the data processing pressure of the server. In the above method, the timbre conversion is combined with the audio decoding stage, and the processing of the timbre vector is combined in the audio decoding process, which reduces the deployment cost of the timbre conversion while ensuring the quality of the timbre conversion.

[0069] FIG5 is a flow chart of another timbre conversion method provided in an embodiment of the present application, as shown in FIG5 , including:

[0070] Step S401: Receive embedded vector information sent by the client, and encode the embedded vector information through the set second encoding module and timbre vector to obtain second audio data.

[0071] Step S402: Decode the second audio data using the set decoding module and the timbre vector to obtain voice-changed audio.

[0072] In one embodiment, timbre vectors are superimposed during decoding. Alternatively, the second audio data may be decoded by a predetermined decoding module. During the decoding process, the timbre vectors superimposed on the output of each convolutional layer are used as input to the next convolutional layer to obtain the voice-changed audio. This further enhances the timbre conversion effect.

[0073] As can be seen from the above, at the decoding end, by receiving the embedded vector information sent by the client, the embedded vector information is encoded and processed by the set second encoding module and the timbre vector to obtain the second audio data, and then the second audio data is decoded by the set decoding module to obtain the voice-changed audio. This timbre conversion method combines timbre conversion with audio decoding, and the timbre conversion function can be deployed on the client, which reduces the server cost and alleviates the data processing pressure of the server. In the above method, the timbre conversion is combined with the audio decoding stage, and the processing of the timbre vector is combined in the audio decoding process, which reduces the deployment cost of the timbre conversion while ensuring the quality of the timbre conversion.

[0074] FIG6 is a block diagram of a timbre conversion device provided in an embodiment of the present application. As shown in FIG6 , the device is used to execute the timbre conversion method provided in the above embodiment, and has the corresponding functional modules and beneficial effects of the execution method. As shown in FIG6 , the device specifically includes: a first encoding module 101, a vector quantization processing module 102, and a sending module 103, wherein:

[0075] A first encoding module 101 is configured to obtain original audio, and encode the original audio based on a set timbre vector to obtain first audio data;

[0076] A vector quantization processing module 102 is configured to perform vector quantization processing on the first audio data to obtain corresponding embedded vector information;

[0077] The sending module 103 is configured to send the embedded vector information to the decoding end, so that the decoding end generates second audio data based on the timbre vector and the embedded vector information, and decodes the second audio data through the set decoding module to obtain voice-changed audio.

[0078] As can be seen from the above, in the client, the original audio is obtained, and the original audio is encoded by the set first encoding module and timbre vector to obtain the first audio data, the first audio data is vector quantized to obtain the corresponding embedded vector information, and the embedded vector information is sent to the decoding end, for the decoding end to generate the second audio data based on the timbre vector and the embedded vector information, and the second audio data is decoded by the set decoding module to obtain the voice-changed audio. This timbre conversion method combines timbre conversion with audio encoding, and deploys the timbre conversion function on the client, which reduces the server cost and alleviates the data processing pressure of the server. In the above method, the timbre conversion is combined with the audio encoding stage, and the processing of the timbre vector is combined in the audio encoding process, which reduces the deployment cost of the timbre conversion while ensuring the quality of the timbre conversion.

[0079] In a possible embodiment, the first encoding module includes a plurality of convolutional layers, each of the convolutional layers includes a causal convolutional neural network, and the first encoding module 101 is configured as follows:

[0080] The original audio is encoded by a set first encoding module. During the encoding process, the output of each convolution layer is superimposed on the set timbre vector as the input of the next convolution layer to obtain the first audio data.

[0081] In a possible embodiment, the vector quantization processing module 102 is configured as follows:

[0082] Perform residual vector quantization processing on the first audio data to obtain index information outputted at each layer, wherein the index information at each layer corresponds to an embedded vector of a corresponding index value in a set code table.

[0083] FIG7 is a block diagram of another timbre conversion device provided in an embodiment of the present application. As shown in FIG7 , the device is used to execute the timbre conversion method provided in the above embodiment, and has the corresponding functional modules and beneficial effects of the execution method. As shown in FIG7 , the device specifically includes: a receiving module 104, a second encoding module 105 and a decoding module 106, wherein,

[0084] Receiving module 104, configured to receive embedding vector information sent by the client;

[0085] A second encoding module 105 is configured to encode the embedded vector information using the set timbre vector to obtain second audio data;

[0086] The decoding module 106 is configured to decode the second audio data to obtain voice-changed audio.

[0087] As can be seen from the above, at the decoding end, by receiving the embedded vector information sent by the client, the embedded vector information is encoded and processed by the set second encoding module and the timbre vector to obtain the second audio data, and then the second audio data is decoded by the set decoding module to obtain the voice-changed audio. This timbre conversion method combines timbre conversion with audio decoding, and the timbre conversion function can be deployed on the client, which reduces the server cost and alleviates the data processing pressure of the server. In the above method, the timbre conversion is combined with the audio decoding stage, and the processing of the timbre vector is combined in the audio decoding process, which reduces the deployment cost of the timbre conversion while ensuring the quality of the timbre conversion.

[0088] In a possible embodiment, the second encoding module includes a plurality of convolutional layers, each of the convolutional layers includes a causal convolutional neural network, and the second encoding module 105 is configured as follows:

[0089] The timbre vector and the embedding vector corresponding to the embedding vector information are superimposed by a second encoding module to obtain second audio data.

[0090] In a possible embodiment, the embedded vector information includes index information, and the second encoding module 105 is configured to:

[0091] In a case where the embedded vector information is index information, querying the embedded vector corresponding to the index information based on a set code table;

[0092] The timbre vector and the embedding vector are superimposed on each other through a second encoding module to obtain second audio data.

[0093] In a possible embodiment, the decoding module 106 is configured to:

[0094] The second audio data is decoded by the provided decoding module and the timbre vector to obtain voice-changed audio.

[0095] In one possible embodiment, the decoding module includes multiple convolutional layers, each of which includes a causal convolutional neural network, and the decoding module 106 is configured as follows:

[0096] The second audio data is decoded by a set decoding module. During the decoding process, the output of each convolution layer is superimposed on the timbre vector as the input of the next convolution layer to obtain voice-changed audio.

[0097] Figure 8 is a schematic diagram of the structure of a voice conversion device provided in an embodiment of the present application. As shown in Figure 8, the device includes a processor 201, a memory 202, an input device 203, and an output device 204. The device may include one or more processors 201, with one processor 201 being used as an example in Figure 8. The processor 201, memory 202, input device 203, and output device 204 in the device may be connected via a bus or other means, with bus connection being used as an example in Figure 8. Memory 202, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the voice conversion method in the embodiment of the present application. Processor 201 executes the software programs, instructions, and modules stored in memory 202 to execute various functional applications and data processing of the device, thereby implementing the voice conversion method described above. Input device 203 can be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. Output device 204 may include a display device such as a display screen.

[0098] The present application also provides a non-volatile storage medium containing computer-executable instructions. When executed by a computer processor, the computer-executable instructions are used to perform a timbre conversion method described in the above embodiment, which includes:

[0099] Obtaining original audio, and encoding the original audio using a set first encoding module and a timbre vector to obtain first audio data;

[0100] Performing vector quantization processing on the first audio data to obtain corresponding embedded vector information;

[0101] The embedded vector information is sent to a decoding end, so that the decoding end generates second audio data based on the timbre vector and the embedded vector information, and decodes the second audio data to obtain voice-changed audio through a set decoding module. And / or,

[0102] Receiving embedded vector information sent by the client, encoding the embedded vector information using a set second encoding module and a timbre vector to obtain second audio data;

[0103] The second audio data is decoded by a set decoding module to obtain voice-changed audio.

[0104] It is worth noting that in the embodiment of the above-mentioned tone conversion device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of this application.

[0105] In some possible implementations, various aspects of the methods provided herein may also be implemented in the form of a program product, which includes program code. When the program product is executed on a computer device, the program code is used to cause the computer device to perform the steps of the methods according to the various exemplary embodiments of the present application described above. For example, the computer device may perform the timbre conversion method described in the embodiments of the present application. The program product may be implemented using any combination of one or more readable media.

Claims

1. The timbre conversion method is applied to the client, wherein: include: Acquire original audio, and encode the original audio by using a set first encoding module and a timbre vector to obtain first audio data; Performing vector quantization processing on the first audio data to obtain corresponding embedded vector information; The embedded vector information is sent to a decoding end, so that the decoding end generates second audio data based on the timbre vector and the embedded vector information, and decodes the second audio data through a set decoding module to obtain voice-changed audio.

2. The timbre conversion method according to claim 1, wherein: The first encoding module includes a plurality of convolutional layers, each of which includes a causal convolutional neural network, and the first encoding module and the timbre vector are used to encode the original audio to obtain the first audio data, including: The original audio is encoded by a first encoding module. During the encoding process, the output of each convolution layer is superimposed with the set timbre vector as the input of the next convolution layer to obtain the first audio data.

3. The timbre conversion method according to claim 1, wherein: The performing vector quantization processing on the first audio data to obtain corresponding embedded vector information includes: The first audio data is subjected to residual vector quantization processing to obtain index information outputted at each layer, wherein the index information at each layer corresponds to an embedded vector of a corresponding index value in a set code table.

4. The timbre conversion method is applied to the decoding end, wherein: include: Receiving the embedded vector information sent by the client, and encoding the embedded vector information by using the set second encoding module and the timbre vector to obtain the second audio data; The second audio data is decoded by a set decoding module to obtain voice-changed audio.

5. The timbre conversion method according to claim 4, wherein: The second encoding module includes a plurality of convolutional layers, each of which includes a causal convolutional neural network, and the second encoding module and the timbre vector are set to encode the embedded vector information to obtain the second audio data, including: The timbre vector and the embedded vector corresponding to the embedded vector information are superimposed by a second encoding module to obtain second audio data.

6. The timbre conversion method according to claim 5, wherein: The embedded vector information includes index information, and the second encoding module is set to superimpose the timbre vector with the embedded vector corresponding to the embedded vector information to obtain the second audio data, including: In the case where the embedded vector information is index information, querying the embedded vector corresponding to the index information based on a set code table; The timbre vector and the embedding vector are superimposed on each other through a second encoding module to obtain second audio data.

7. The timbre conversion method according to claim 4, wherein: The step of decoding the second audio data by a set decoding module to obtain the voice-changed audio includes: The second audio data is decoded by the set decoding module and the timbre vector to obtain the voice-changed audio.

8. The timbre conversion method according to claim 7, wherein: The decoding module includes a plurality of convolutional layers, each of which includes a causal convolutional neural network, and the second audio data is decoded by the set decoding module and the timbre vector to obtain the voice-changed audio, including: The second audio data is decoded by a set decoding module. During the decoding process, the output of each convolution layer is superimposed on the timbre vector as the input of the next convolution layer to obtain the voice-changed audio.

9. A device for converting tone, wherein: include: A first encoding module is configured to obtain original audio, and encode the original audio based on the set timbre vector to obtain first audio data; A vector quantization processing module, configured to perform vector quantization processing on the first audio data to obtain corresponding embedded vector information; The sending module is configured to send the embedded vector information to the decoding end, so that the decoding end generates second audio data based on the timbre vector and the embedded vector information, and decodes the second audio data through the set decoding module to obtain the voice-changed audio.

10. A timbre conversion device, wherein: include: A receiving module, configured to receive the embedding vector information sent by the client; A second encoding module is configured to encode the embedded vector information through the set timbre vector to obtain second audio data; The decoding module is configured to decode the second audio data to obtain the voice-changed audio.

11. A timbre conversion device, comprising: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the timbre conversion method described in any one of claims 1-8.

12. A non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute the timbre conversion method according to any one of claims 1 to 8 when executed by a computer processor.

13. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the timbre conversion method according to any one of claims 1 to 8 is implemented.

Images