Sound cloning system and method, electronic equipment and storage medium

Abstract

The invention relates to the technical field of electric digital data processing, in particular to a sound cloning system and method, electronic equipment and a storage medium, and the system comprises an audio input module which is used for receiving a reference voice segment of any sound source; a semantic extraction module for identifying a linguistic discrete semantic representation of the reference speech segment based on the reference speech segment; the voiceprint feature extraction module is used for extracting voiceprint features of any sound source based on the reference voice segment; the acoustic synthesis module is used for reconstructing the linguistic discrete semantic representation and the voiceprint feature into a corresponding acoustic spectrum; and the vocoder is used for taking the acoustic spectrum as input so as to output voice with the tone of any sound source by using the reference voice segment. Therefore, the technical problems that in the related technology, a sound cloning model needs target sound source voice of a certain duration, instant cloning cannot be achieved, and the phenomenon of tone leakage exists, so that the synthesized voice still contains part of original sound source features and the like are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic digital data processing, and in particular to a sound cloning system, method, electronic equipment and storage medium. Background Art

[0002] In the related art, most of the sound cloning algorithms based on DNN (Deep Neural Network) are pre-trained models, that is, each trained model is obtained through pre-training based on an existing data set. In the absence of target sound data or without model training, the cloning of the target sound source sound cannot be achieved. For example, VITS, Soft-VC, etc., the sound conversion that can be achieved after training is a many-to-one conversion, that is, each time a new target sound source sound is cloned, it is necessary to train on the target sound source sound training set to obtain appropriate model parameters. This means that the existing sound cloning model requires a certain length of target sound source speech and cannot achieve instant cloning.

[0003] In addition, there is a timbre leakage phenomenon in Soft-VC, which uses soft speech representation. The sound source features are not completely removed from the extracted speech vector, resulting in the synthesized speech still containing some of the original sound source features.

[0004] In summary, in the related technologies, the sound cloning model requires a certain length of target sound source speech, cannot achieve instant cloning, and has the phenomenon of timbre leakage, resulting in the synthesized speech still containing some original sound source characteristics, which needs to be improved. Summary of the invention

[0005] The present invention provides a sound cloning system, method, electronic device and storage medium to solve the technical problems in the related art that the sound cloning model requires a target sound source voice of a certain duration, cannot achieve instant cloning, and has a timbre leakage phenomenon, resulting in the synthesized voice still containing some original sound source characteristics.

[0006] A first aspect of the present invention provides a sound cloning system, comprising: an audio input module for receiving a reference speech segment of any sound source; a semantic extraction module for identifying a linguistic discrete semantic representation of the reference speech segment based on the reference speech segment; a voiceprint feature extraction module for extracting a voiceprint feature of any sound source based on the reference speech segment; an acoustic synthesis module for reconstructing the linguistic discrete semantic representation and the voiceprint feature into a corresponding acoustic spectrum; and a vocoder for taking the acoustic spectrum as input to output a speech having the timbre of any sound source using the reference speech segment.

[0007] Optionally, in one embodiment of the present invention, the semantic extraction module is further used to discretize the reference speech segment representation into an integer sequence within a preset interval using a pre-trained Hubert algorithm to obtain the linguistic discrete semantic representation.

[0008] Optionally, in one embodiment of the present invention, the voiceprint feature extraction module includes: a Mel filter, used to convert the speech waveform in the reference speech segment from the time domain to the frequency domain to obtain the Mel spectrum of the reference speech segment; a conversion unit, used to combine the Mel spectrum, a pre-built convolutional neural network and a pre-built long short-term memory neural network to obtain the final voiceprint attributes of any sound source, and convert the final voiceprint attributes into a voiceprint feature vector of a fixed length.

[0009] Optionally, in one embodiment of the present invention, the acoustic synthesis module includes: a synthesis unit, used to use a non-linear layer to match the Mel spectrum of the timbre of any sound source generated in the previous time step, the integer sequence of the linguistic discrete semantic representation, and the vector length of the voiceprint feature to obtain matching aggregated data; an output unit, used to input the aggregated data into a preset multi-layer residual long short-term memory neural network to obtain the acoustic spectrum having the characteristics of any sound source.

[0010] Optionally, in one embodiment of the present invention, the vocoder includes: a conversion unit, used to convert the acoustic spectrum into an audio waveform of speech with the timbre of any sound source using a HiFi-GAN algorithm, so as to output speech with the timbre of any sound source.

[0011] A second aspect of the present invention provides a method for cloning a sound, comprising the following steps: receiving a reference speech segment of any sound source; identifying a linguistic discrete semantic representation of the reference speech segment based on the reference speech segment; extracting a voiceprint feature of the any sound source based on the reference speech segment; reconstructing the linguistic discrete semantic representation and the voiceprint feature into a corresponding acoustic spectrum; and using the acoustic spectrum as input to output a speech having the timbre of the any sound source using the reference speech segment.

[0012] Optionally, in one embodiment of the present invention, identifying the linguistic discrete semantic representation of the reference speech segment based on the reference speech segment includes: using a pre-trained Hubert algorithm to discretize the reference speech segment representation into an integer sequence within a preset interval to obtain the linguistic discrete semantic representation.

[0013] Optionally, in one embodiment of the present invention, extracting the voiceprint features of any sound source based on the reference speech segment includes: converting the speech waveform in the reference speech segment from the time domain to the frequency domain to obtain the Mel spectrum of the reference speech segment; combining the Mel spectrum, a pre-built convolutional neural network and a pre-built long short-term memory neural network to obtain the final voiceprint attributes of any sound source, and converting the final voiceprint attributes into a voiceprint feature vector of fixed length.

[0014] Optionally, in one embodiment of the present invention, the reconstructing the linguistic discrete semantic representation and the voiceprint features into a corresponding acoustic spectrum includes: using a nonlinear layer to match the Mel spectrum of the timbre of any sound source generated in the previous time step, the integer sequence of the linguistic discrete semantic representation, and the vector length of the voiceprint features to obtain matching aggregated data; inputting the aggregated data into a preset multi-layer residual long short-term memory neural network to obtain the acoustic spectrum having the characteristics of any sound source.

[0015] Optionally, in one embodiment of the present invention, taking the acoustic spectrum as input to output speech with the timbre of any sound source using the reference speech segment includes: using a HiFi-GAN algorithm to convert the acoustic spectrum into an audio waveform of speech with the timbre of any sound source to output speech with the timbre of any sound source.

[0016] A third aspect of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the sound cloning method as described in the above embodiment.

[0017] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the sound cloning method as described in the above embodiment.

[0018] A fifth aspect of the present invention provides a computer program product, including a computer program, which, when executed, is used to implement the above sound cloning method.

[0019] The embodiment of the present invention can use an audio input module to receive a reference speech segment of any sound source, identify the linguistic discrete semantic representation of the reference speech segment through a semantic extraction module, extract the voiceprint features of any sound source through a voiceprint feature extraction module, reconstruct the linguistic discrete semantic representation and voiceprint features into a corresponding acoustic spectrum through an acoustic synthesis module, and finally use the acoustic spectrum as input through a vocoder to output a speech with the timbre of any sound source using the reference speech segment, extract the linguistic discrete semantic representation of the original speech as a linguistic feature, extract the voiceprint features embedded in the speech of any length of the target sound source, synthesize the original speech content speech with the timbre of the target sound source through an acoustic model, and clone the timbre of the target sound source without additional training steps, thereby achieving instant sound cloning. Therefore, the technical problems in the related art that the sound cloning model requires a certain length of target sound source speech, cannot achieve instant cloning, and has a timbre leakage phenomenon, resulting in the synthesized speech still containing some original sound source features, etc. are solved.

[0020] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0022] Figure 1 A schematic diagram of the structure of a sound cloning system provided according to an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of the principle of a sound cloning system according to an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of a semantic extraction module according to an embodiment of the present invention;

[0025] Figure 4 A schematic diagram of a voiceprint feature extraction module according to an embodiment of the present invention;

[0026] Figure 5 A schematic diagram of the principle of an acoustic synthesis module according to an embodiment of the present invention;

[0027] Figure 6 A schematic diagram of a decoder according to an embodiment of the present invention;

[0028] Figure 7 A flowchart of a sound cloning method provided according to an embodiment of the present invention;

[0029] Figure 8It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present invention.

[0030] Among them, 10-sound cloning system, 100-audio input module, 200-semantic extraction module, 300-voiceprint feature extraction module, 400-acoustic synthesis module, 500-vocoder. DETAILED DESCRIPTION

[0031] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0032] The following describes the sound cloning system, method, electronic device and storage medium of the embodiment of the present invention with reference to the accompanying drawings. In view of the technical problems that the sound cloning model requires a target sound source speech of a certain length in the related art mentioned in the above background technology, it is not possible to clone instantly, and there is a phenomenon of timbre leakage, resulting in the synthesized speech still containing some original sound source features, the present invention provides a sound cloning system, in which an audio input module can be used to receive a reference speech segment of any sound source, a semantic extraction module can be used to identify the linguistic discrete semantic representation of the reference speech segment, a voiceprint feature of any sound source can be extracted by a voiceprint feature extraction module, and the linguistic discrete semantic representation and voiceprint feature can be reconstructed into a corresponding acoustic spectrum by an acoustic synthesis module, and finally the acoustic spectrum is used as input by a vocoder to output a speech with the timbre of any sound source by using the reference speech segment, extract the linguistic discrete semantic representation of the original speech as a linguistic feature, extract the voiceprint feature embedded in the speech of any length of the target sound source, and synthesize the original speech content speech with the timbre of the target sound source by an acoustic model, so that the timbre of the target sound source can be cloned without additional training steps, thereby achieving instant sound cloning. This solves the technical problems in related technologies, such as the sound cloning model requires a target sound source voice of a certain length, cannot achieve instant cloning, and there is a phenomenon of timbre leakage, resulting in the synthesized speech still containing some original sound source characteristics.

[0033] Specifically, Figure 1 A schematic diagram of the structure of a sound cloning system provided by an embodiment of the present invention.

[0034] like Figure 1 As shown, the voice cloning system 10 includes: an audio input module 100 , a semantic extraction module 200 , a voiceprint feature extraction module 300 , an acoustic synthesis module 400 and a vocoder 500 .

[0035] Specifically, the audio input module 100 is used to receive a reference speech segment from any sound source.

[0036] In actual implementation, the embodiment of the present invention can use the audio input module 100 to obtain a reference speech segment of any sound source, wherein the sound source can be a human voice.

[0037] The semantic extraction module 200 is used to identify the linguistic discrete semantic representation of the reference speech segment based on the reference speech segment.

[0038] Among them, linguistic discrete semantic representation (DSL) is a formal method for representing the semantic information of natural language sentences or phrases. It aims to convert the semantic information in natural language into a discrete and structured form to facilitate semantic analysis and understanding by computers. The goal of linguistic discrete semantic representation is to capture the semantic meaning in natural language so that computers can better understand and process text data. The representation form of DSL can be a logical form, such as first-order logic, predicate logic, etc., or a graph structure, tree structure, etc. DSL usually includes a combination of vocabulary, syntax, and semantic information to accurately describe the semantic meaning of a sentence.

[0039] As a possible implementation method, the embodiment of the present invention can use the semantic extraction module 200 to extract the linguistic discrete semantic representation of the audio corresponding to the reference speech segment, so as to perform subsequent data processing and realize real-time cloning.

[0040] Optionally, in one embodiment of the present invention, the semantic extraction module 200 is further configured to discretize the reference speech segment representation into an integer sequence within a preset interval using a pre-trained Hubert algorithm to obtain a linguistic discrete semantic representation.

[0041] For example, the semantic extraction module 200 may use the pre-trained Hubert algorithm to discretize the speech representation into an integer sequence of 1 to 100 through vector quantization.

[0042] The voiceprint feature extraction module 300 is used to extract the voiceprint features of any sound source based on a reference speech segment.

[0043] Among them, the acoustic features of voiceprint features include the frequency of the sound, the intensity of the sound, the duration of the sound, the harmonic structure of the sound, etc. These acoustic features can be extracted through methods such as spectrum analysis, frequency spectrum analysis, and waveform analysis. Information such as the frequency distribution, peak frequency, and spectrum envelope of the sound can all be used to extract and analyze voiceprint features.

[0044] The linguistic features of voiceprint features include the speaker's voice features, speaking speed, intonation, voice rhythm, etc., which can reflect the speaker's personality characteristics, cultural background and other information.

[0045] Furthermore, the embodiment of the present invention can use the voiceprint feature extraction module 300 to extract the voiceprint features of any sound source from the reference speech segment to ensure the similarity of the cloning result. For example, the embodiment of the present invention can input a reference speech segment of any length from any sound source into the voiceprint feature extraction module 300, and then output the corresponding voiceprint embedding.

[0046] Optionally, in one embodiment of the present invention, the voiceprint feature extraction module 300 includes: a Mel filter and a conversion unit.

[0047] The Mel filter is used to convert the speech waveform in the reference speech segment from the time domain to the frequency domain to obtain the Mel spectrum of the reference speech segment.

[0048] The conversion unit is used to combine the Mel spectrum, the pre-built convolutional neural network and the pre-built long short-term memory neural network to obtain the final voiceprint attributes of any sound source, and convert the final voiceprint attributes into a voiceprint feature vector of fixed length.

[0049] In some embodiments, the voiceprint feature extraction module 300 can use a Mel filter to convert the speech waveform from the time domain to the frequency domain, and then input the obtained Mel spectrum into three layers of convolutional neural networks for extracting spectral features and their changing trends and a layer of long short-term memory neural network for extracting the time scale attributes of hidden features. The output of the last step is taken as the final voiceprint attribute, which is converted into a voiceprint feature vector of fixed length through a linear layer for result mapping.

[0050] The acoustic synthesis module 400 is used to reconstruct the linguistic discrete semantic representation and voiceprint features into the corresponding acoustic spectrum.

[0051] Among them, the acoustic spectrum refers to the representation of the sound signal in the frequency domain, which is used to describe the intensity and distribution of different frequency components in the sound signal. Acoustic spectrum analysis can be used to analyze the frequency components and characteristics of the sound, thereby realizing the recognition, classification and analysis of the sound.

[0052] Furthermore, the linguistic discrete semantic representation in the original reference speech segment and the voiceprint features of any sound source are reconstructed into a corresponding acoustic spectrum, namely, a Mel spectrum, through the acoustic synthesis module 400 .

[0053] Optionally, in one embodiment of the present invention, the acoustic synthesis module 400 includes: a synthesis unit and an output unit.

[0054] Among them, the synthesis unit is used to use a nonlinear layer to match the Mel spectrum of the timbre of any sound source generated in the previous time step, the integer sequence of linguistic discrete semantic representation, and the vector length of the voiceprint feature to obtain matching aggregated data.

[0055] The output unit is used to input the aggregated data into a preset multi-layer residual long short-term memory neural network to obtain an acoustic spectrum with the characteristics of any sound source.

[0056] In other embodiments, the acoustic synthesis module 400 may be an autoregressive recurrent neural network, which uses a nonlinear layer for input mapping to match the Mel spectrum of the timbre of any sound source generated in the previous time step, the input linguistic discrete semantic representation sequence, and the length of the input voiceprint feature vector, adds and aggregates the data, and inputs a multi-layer residual long short-term memory neural network for autoregressive reconstruction of acoustic information to obtain a Mel spectrum with the characteristics of any sound source.

[0057] The vocoder 500 is used to take the acoustic spectrum as input to output a speech with the timbre of any sound source by using a reference speech segment. The vocoder 500 includes: a conversion unit.

[0058] The conversion unit is used to convert the acoustic spectrum into an audio waveform of a speech with the timbre of any sound source by using the HiFi-GAN algorithm, so as to output the speech with the timbre of any sound source.

[0059] That is, the vocoder 500 can use the HiFi-GAN algorithm to convert the input Mel spectrum with any sound source characteristics into an audio waveform of a speech with any sound source characteristics.

[0060] Combination Figures 2 to 6 As shown, the working principle of the sound cloning system of the embodiment of the present invention is explained by taking an embodiment as an example.

[0061] Specifically, the technical solution adopted by the embodiment of the present invention may include: a modular multi-step deep neural network model, which is composed of an encoder and a decoder. The steps of the encoder and the decoder for instant sound cloning may be as follows: Figure 2 As shown:

[0062] Step S1: Obtain an original audio waveform from a reference speech segment of any sound source.

[0063] Step S2: The encoder uses the semantic extraction module 200 to extract the linguistic discrete semantic representation of the original audio, performs Hubert discretization encoding on the original audio waveform, and obtains a discretization encoding representation.

[0064] like Figure 3As shown, the semantic extraction module 200 inputs the original speech waveform that needs to retain the speech content and remove the voiceprint features. The convolution layer is used to convert the PCM code of the audio into a high-dimensional vector, and the random mask and position information encoding layer are used for additive embedding. The token is input into the BERT model. The output of BERT passes through the linear mapping layer and the clustering quantization layer to become a linguistic discrete semantic representation, which is input into the acoustic synthesis module 400.

[0065] The model can use Hubert with pre-trained parameters as the semantic extraction module 200, or can autonomously train the parameters of the semantic extraction module 200 based on a given data set. The model performs joint training for the voiceprint feature extraction module 300 and the acoustic synthesis module 400, using the trained semantic extraction module 200 on a given data set to obtain input, and using the L2 norm as the target loss function for the output.

[0066] The embodiment of the present invention can also adjust the semantic clarity of the final synthesized speech by adjusting the scale of the cluster quantization layer of the semantic extraction module 200. The higher the number of clusters in the cluster quantization, the higher the semantic clarity.

[0067] The embodiment of the present invention can also bypass the cluster quantization layer and use the result of the linear mapping layer of the semantic extraction module 200 directly as the input of the acoustic synthesis module 400, and can match the input vector size by modifying the shape of the embedding layer. In this case, the model will retain more sound details of the original audio.

[0068] Step S3: The encoder uses the voiceprint feature extraction module 300 to extract the voiceprint features of any sound source, such as using an RNN voiceprint extractor to extract the voiceprint features and obtain a vectorized voiceprint representation.

[0069] like Figure 4 As shown, the voiceprint feature extraction module 300 inputs the voice sample of any sound source whose voiceprint features need to be retained and the voice content is removed. The frequency domain representation of the audio is obtained by using the Mel filter bank, and the implicit vector is extracted by the first spectrum convolution and the second spectrum convolution, which is input into the long short-term memory layer to extract the voiceprint features, and then input into the acoustic synthesis module 400 using a linear mapping layer.

[0070] Step S4: The decoder uses the acoustic synthesis module 400 to reconstruct the linguistic discrete semantic representation of the original audio and the voiceprint features of any sound source into a Mel spectrum.

[0071] like Figure 5As shown, the acoustic synthesis module 400 can accept the features from the semantic extraction module 200 through the embedding layer, the nonlinear layer and the convolution upsampling layer as input, the voiceprint features of the voiceprint feature extraction module 300 as input, and at the same time accept the earlier output of the acoustic synthesis module 400 (such as the output of the previous step), pass through the nonlinear transformation layer as the input of the current time step, concatenate and embed the spectral features with the voiceprint semantics after addition and embedding, and then pass through the residual long short-term memory layer, and output the Mel spectrum of the current time step through the output mapping layer.

[0072] Step S5: The decoder uses the vocoder 500 to reconstruct the waveform.

[0073] like Figure 6 As shown, the vocoder 500 receives the Mel spectrum from the acoustic synthesis module 400, passes through a convolution layer, a series of convolution upsampling layers and a multi-receptive field fusion module, and finally passes through an output mapping layer to output a target audio waveform.

[0074] The selection of the vocoder 500 depends on the actual application scenario. HiFi-GAN can be used as the vocoder 500, or WaveNet2 or other models can be used as the vocoder 500.

[0075] According to the sound cloning system proposed in the embodiment of the present invention, the reference speech segment of any sound source can be received by the audio input module, the linguistic discrete semantic representation of the reference speech segment can be identified by the semantic extraction module, the voiceprint feature of any sound source can be extracted by the voiceprint feature extraction module, the linguistic discrete semantic representation and the voiceprint feature can be reconstructed into a corresponding acoustic spectrum by the acoustic synthesis module, and finally the acoustic spectrum can be used as input by the vocoder to output a speech with the timbre of any sound source by using the reference speech segment, the voiceprint feature embedded in the speech of any length of the target sound source can be extracted by extracting the linguistic discrete semantic representation of the original speech as the linguistic feature, and the original speech content speech with the timbre of the target sound source can be synthesized by the acoustic model, and the timbre of the target sound source can be cloned without additional training steps, thereby realizing instant sound cloning. Therefore, the technical problems in the related art that the sound cloning model requires a certain length of the target sound source speech, cannot achieve instant cloning, and has a timbre leakage phenomenon, resulting in the synthesized speech still containing some original sound source features, etc. are solved.

[0076] Next, the sound cloning method according to the embodiment of the present invention is described with reference to the accompanying drawings.

[0077] Figure 7 The figure is a flow chart of a sound cloning method according to an embodiment of the present invention.

[0078] like Figure 7 As shown, the sound cloning method includes the following steps:

[0079] In step S701, a reference speech segment of any sound source is received.

[0080] In step S702, based on the reference speech segment, a linguistic discrete semantic representation of the reference speech segment is identified.

[0081] In step S703, the voiceprint features of any sound source are extracted based on the reference speech segment.

[0082] In step S704, the linguistic discrete semantic representation and the voiceprint feature are reconstructed into a corresponding acoustic spectrum.

[0083] In step S705 , the acoustic spectrum is used as an input to output a speech having the timbre of any sound source using a reference speech segment.

[0084] Optionally, in one embodiment of the present invention, based on a reference speech segment, identifying a linguistic discrete semantic representation of the reference speech segment includes: using a pre-trained Hubert algorithm to discretize the reference speech segment representation into an integer sequence within a preset interval to obtain a linguistic discrete semantic representation.

[0085] Optionally, in one embodiment of the present invention, based on a reference speech segment, voiceprint features of any sound source are extracted, including: converting the speech waveform in the reference speech segment from the time domain to the frequency domain to obtain a Mel spectrum of the reference speech segment; combining the Mel spectrum, a pre-built convolutional neural network, and a pre-built long short-term memory neural network to obtain the final voiceprint attributes of any sound source, and converting the final voiceprint attributes into a voiceprint feature vector of a fixed length.

[0086] Optionally, in one embodiment of the present invention, the linguistic discrete semantic representation and the voiceprint features are reconstructed into corresponding acoustic spectra, including: using a nonlinear layer to match the Mel spectrum of the timbre of any sound source generated in the previous time step, the integer sequence of the linguistic discrete semantic representation, and the vector length of the voiceprint features to obtain matching aggregated data; inputting the aggregated data into a preset multi-layer residual long short-term memory neural network to obtain an acoustic spectrum with the characteristics of any sound source.

[0087] Optionally, in one embodiment of the present invention, an acoustic spectrum is used as input to output speech with the timbre of any sound source using a reference speech segment, including: using a HiFi-GAN algorithm to convert the acoustic spectrum into an audio waveform of speech with the timbre of any sound source to output speech with the timbre of any sound source.

[0088] It should be noted that the above explanation of the sound cloning system embodiment is also applicable to the sound cloning method of this embodiment, which will not be repeated here.

[0089] According to the sound cloning method proposed in the embodiment of the present invention, the reference speech segment of any sound source can be received by the audio input module, the linguistic discrete semantic representation of the reference speech segment can be identified by the semantic extraction module, the voiceprint feature of any sound source can be extracted by the voiceprint feature extraction module, the linguistic discrete semantic representation and the voiceprint feature can be reconstructed into a corresponding acoustic spectrum by the acoustic synthesis module, and finally the acoustic spectrum can be used as input by the vocoder to output a speech with the timbre of any sound source by using the reference speech segment, extract the linguistic discrete semantic representation of the original speech as the linguistic feature, extract the voiceprint feature embedded in the speech of any length of the target sound source, synthesize the original speech content speech with the timbre of the target sound source by the acoustic model, and clone the timbre of the target sound source without additional training steps, thereby realizing instant sound cloning. Therefore, the technical problems in the related art that the sound cloning model requires a certain length of the target sound source speech, cannot achieve instant cloning, and has a timbre leakage phenomenon, resulting in the synthesized speech still containing some original sound source features, etc. are solved.

[0090] Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device may include:

[0091] A memory 801 , a processor 802 , and a computer program stored in the memory 801 and executable on the processor 802 .

[0092] When the processor 802 executes the program, the sound cloning method provided in the above embodiment is implemented.

[0093] Furthermore, the electronic device further comprises:

[0094] The communication interface 803 is used for communication between the memory 801 and the processor 802 .

[0095] The memory 801 is used to store computer programs that can be executed on the processor 802 .

[0096] The memory 801 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0097] If the memory 801, the processor 802 and the communication interface 803 are implemented independently, the communication interface 803, the memory 801 and the processor 802 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0098] Optionally, in a specific implementation, if the memory 801, the processor 802 and the communication interface 803 are integrated on a chip, the memory 801, the processor 802 and the communication interface 803 can communicate with each other through an internal interface.

[0099] The processor 802 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0100] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned sound cloning method is implemented.

[0101] The embodiment of the present invention further provides a computer program product, including a computer program, which implements the sound cloning method provided by the embodiment of the present invention when executed by a processor.

[0102] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0103] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0104] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

[0105] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0106] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0107] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0108] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0109] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A sound cloning system, characterized in that: include: An audio input module, used for receiving a reference speech segment from any sound source; A semantic extraction module, configured to identify a linguistic discrete semantic representation of the reference speech segment based on the reference speech segment; A voiceprint feature extraction module, used to extract the voiceprint features of any sound source based on the reference voice segment; An acoustic synthesis module, used for reconstructing the linguistic discrete semantic representation and the voiceprint feature into a corresponding acoustic spectrum; A vocoder is used to take the acoustic spectrum as input to output speech with the timbre of any sound source using the reference speech segment.

2. The sound cloning system according to claim 1, characterized in that: The semantic extraction module is further used to discretize the reference speech segment representation into an integer sequence within a preset interval using a pre-trained Hubert algorithm to obtain the linguistic discrete semantic representation.

3. The sound cloning system according to claim 1, characterized in that: The voiceprint feature extraction module includes: A Mel filter is used to convert the speech waveform in the reference speech segment from the time domain to the frequency domain to obtain a Mel spectrum of the reference speech segment; A conversion unit is used to combine the Mel spectrum, the pre-constructed convolutional neural network and the pre-constructed long short-term memory neural network to obtain the final voiceprint attribute of any sound source, and convert the final voiceprint attribute into a voiceprint feature vector of fixed length.

4. The sound cloning system according to claim 1, characterized in that: The acoustic synthesis module comprises: A synthesis unit, used for matching the Mel spectrum of the timbre of any sound source generated in the previous time step, the integer sequence of the linguistic discrete semantic representation, and the vector length of the voiceprint feature using a nonlinear layer to obtain matching aggregated data; The output unit is used to input the aggregated data into a preset multi-layer residual long short-term memory neural network to obtain the acoustic spectrum having the characteristics of any sound source.

5. The sound cloning system according to claim 1, characterized in that: The vocoder comprises: The conversion unit is used to convert the acoustic spectrum into an audio waveform of a speech having the timbre of any sound source by using a HiFi-GAN algorithm, so as to output the speech having the timbre of any sound source.

6. A method for cloning a sound, characterized in that: The following steps are involved: receiving a reference speech segment from any sound source; Based on the reference speech segment, identifying a linguistically discrete semantic representation of the reference speech segment; Extracting the voiceprint features of any sound source based on the reference voice segment; Reconstructing the linguistic discrete semantic representation and the voiceprint feature into a corresponding acoustic spectrum; The acoustic spectrum is used as an input to output a speech having the timbre of any sound source using the reference speech segment.

7. The sound cloning method according to claim 6, characterized in that: The step of identifying the linguistic discrete semantic representation of the reference speech segment based on the reference speech segment comprises: The reference speech segment representation is discretized into an integer sequence within a preset interval using a pre-trained Hubert algorithm to obtain the linguistic discrete semantic representation.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the sound cloning method according to claim 6 or 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the sound cloning method as claimed in claim 6 or 7.

10. A computer program product, characterized in that It comprises a computer program, which, when executed, is used to implement the sound cloning method as claimed in claim 6 or 7.

Images