Voice conversion method and device based on diffusion model, equipment and storage medium
Through the diffusion model-based speech conversion method, the current cumbersome problem of the speech conversion process is solved, and more efficient speech conversion is achieved, which reduces the degree of distortion and conversion time, and improves reliability.
Patent Information
- Application Number
- CN202411535936.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-30
- Publication Date
- 2025-03-11
AI Technical Summary
The current speech conversion process is cumbersome, which is not conducive to improving conversion efficiency. It is mainly due to the lack of pitch and spectrum processing of existing methods, which leads to high distortion of converted speech, which requires repeated conversion, which increases the conversion time.
The speech conversion method based on the diffusion model is adopted, by obtaining the Mel spectral map of the original speech and the target speech, extracting the pitch characteristics and Mel spectral characteristics, calculating the loss value, training the diffusion model, and generating the converted speech.
It reduces the degree of distortion of converted voice, avoids repeated conversions, reduces conversion time, helps improve speech conversion efficiency, and improves the reliability of converted voice.
Smart Images
Figure CN119601033A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology and can be applied to the field of financial technology. In particular, the present invention relates to a speech conversion method, device, equipment and storage medium based on a diffusion model. Background Art
[0002] Voice conversion technology has a wide range of applications in entertainment, information security, smart life, etc., and is currently a hot research topic in the field of voice. The current voice conversion can overcome language and cultural barriers, allowing users with different backgrounds and language habits to communicate more smoothly, greatly improving the efficiency and breadth of communication.
[0003] However, the current voice conversion process is cumbersome and is not conducive to improving the conversion efficiency of the current voice. The reason is that the existing voice conversion method, when converting the current voice, lacks pitch and spectrum processing, and therefore cannot reduce the distortion of the converted voice of the current voice. Therefore, the current voice needs to be converted repeatedly, thereby increasing the conversion time of the current voice, and therefore, is not conducive to improving the conversion efficiency of the current voice. Summary of the invention
[0004] The present invention provides a speech conversion method, device, computer equipment and storage medium based on a diffusion model to solve the technical problem that the current speech conversion process is cumbersome and is not conducive to improving the current speech conversion efficiency.
[0005] In a first aspect, a method for voice conversion based on a diffusion model is provided, comprising:
[0006] Acquire an original voice and a target voice corresponding to the original voice;
[0007] Obtain a first Mel-spectrogram output by a preset diffusion model based on the original speech, and obtain a second Mel-spectrogram corresponding to the target speech;
[0008] Based on the first Mel-spectrogram, a first pitch feature and a first Mel-spectrogram feature are obtained; based on the second Mel-spectrogram, a second pitch feature and a second Mel-spectrogram feature are obtained;
[0009] Obtaining a first loss value between the first pitch feature and the second pitch feature, and obtaining a second loss value between the first Mel spectrum feature and the second Mel spectrum feature;
[0010] Based on the first loss value, the second loss value and a preset acquisition method, acquiring the trained diffusion model;
[0011] A third Mel-spectrogram outputted by the trained diffusion model based on the current speech is obtained, and a converted speech corresponding to the current speech is generated based on a third pitch feature and a third Mel-spectrogram feature in the third Mel-spectrogram.
[0012] Furthermore, the step of obtaining a preset diffusion model based on a first mel-spectrogram outputted by the original speech to obtain a second mel-spectrogram corresponding to the target speech includes:
[0013] Convert the original speech into a first text content, perform text feature extraction on the first text content to obtain a first text vector, input the first text vector into a preset diffusion model, and obtain a first Mel-spectrogram output by the diffusion model based on the first text vector;
[0014] The target speech is converted into a second text content, text features are extracted for the second text content to obtain a second text vector, the second text vector is input into a decoder, and a second Mel-Mel spectrogram output by the decoder is obtained.
[0015] Further, the acquiring a first pitch feature and a first Mel spectrum feature based on the first Mel spectrum graph, and acquiring a second pitch feature and a second Mel spectrum feature based on the second Mel spectrum graph, includes:
[0016] Selecting the pitch feature of the first Mel-spectrogram as the first pitch feature, and selecting the Mel-spectrogram feature of the first Mel-spectrogram as the first Mel-spectrogram feature;
[0017] The pitch feature of the second Mel-spectrogram is selected as the second pitch feature, and the Mel-spectrogram feature of the second Mel-spectrogram is selected as the second Mel-spectrogram feature.
[0018] Further, the obtaining of a first loss value between the first pitch feature and the second pitch feature, and the obtaining of a second loss value between the first Mel spectrum feature and the second Mel spectrum feature include:
[0019] Obtaining a first loss value between the first pitch feature and the second pitch feature through a preset absolute loss function;
[0020] A second loss value between the first Mel frequency spectrum feature and the second Mel frequency spectrum feature is obtained through the absolute loss function.
[0021] Furthermore, the acquiring the trained diffusion model based on the first loss value, the second loss value and a preset acquisition method includes:
[0022] Training the diffusion model by using the first loss value and the second loss value;
[0023] When both the first loss value and the second loss value are smaller than preset values, the training of the diffusion model is stopped, and the trained diffusion model is obtained.
[0024] Further, the acquiring of the trained diffusion model based on a third Mel-spectrogram outputted from the current speech, and based on a third pitch feature and a third Mel-spectrogram feature in the third Mel-spectrogram, generating a converted speech corresponding to the current speech, comprises:
[0025] Acquire a current speech, input the current speech into the trained diffusion model, acquire a third Mel-spectrogram output by the trained diffusion model based on the current speech, select a pitch feature of the third Mel-spectrogram as a third pitch feature, and select a Mel-spectrogram feature of the third Mel-spectrogram as a third Mel-spectrogram feature;
[0026] The third pitch feature and the third Mel spectrum feature are input into a preset speech converter, a fourth Mel spectrum graph output by the speech converter is obtained, the fourth Mel spectrum graph is input into a preset vocoder, the fourth Mel spectrum graph is converted by the vocoder, and a converted speech corresponding to the current speech is generated.
[0027] Further, after acquiring the third Mel-spectrogram output by the diffusion model after training based on the current speech, and generating the converted speech corresponding to the current speech based on the third pitch feature and the third Mel-spectrogram feature in the third Mel-spectrogram, the speech conversion method includes:
[0028] A push instruction of the converted voice is obtained, the push instruction is executed, and the converted voice is pushed to a preset system.
[0029] In a second aspect, a speech conversion device based on a diffusion model is provided, comprising:
[0030] A first acquisition module, used to acquire an original voice and a target voice corresponding to the original voice;
[0031] A second acquisition module is used to acquire a first Mel-spectrogram output by a preset diffusion model based on the original speech, and acquire a second Mel-spectrogram corresponding to the target speech;
[0032] A third acquisition module, configured to acquire a first pitch feature and a first Mel spectrum feature based on the first Mel spectrum graph, and acquire a second pitch feature and a second Mel spectrum feature based on the second Mel spectrum graph;
[0033] a fourth acquisition module, configured to acquire a first loss value between the first pitch feature and the second pitch feature, and acquire a second loss value between the first Mel spectrum feature and the second Mel spectrum feature;
[0034] a fifth acquisition module, configured to acquire the trained diffusion model based on the first loss value, the second loss value and a preset acquisition method;
[0035] The conversion module is used to obtain the third Mel-spectrogram output by the trained diffusion model based on the current speech, and generate a converted speech corresponding to the current speech based on the third pitch feature and the third Mel-spectrogram feature in the third Mel-spectrogram.
[0036] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned speech conversion method when executing the computer program.
[0037] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned speech conversion method are implemented.
[0038] The present application provides a speech conversion method, apparatus, computer equipment and storage medium based on a diffusion model, which obtain an original speech and a target speech corresponding to the original speech; obtain a first Mel-spectrogram output based on the original speech by a preset diffusion model, and obtain a second Mel-spectrogram corresponding to the target speech; obtain a first pitch feature and a first Mel-spectrogram feature based on the first Mel-spectrogram, and obtain a second pitch feature and a second Mel-spectrogram feature based on the second Mel-spectrogram; obtain a first loss value between the first pitch feature and the second pitch feature, and obtain a second loss value between the first Mel-spectrogram feature and the second Mel-spectrogram feature; obtain the trained diffusion model based on the first loss value, the second loss value and a preset acquisition method; obtain the trained diffusion model based on the current speech output by the diffusion model after training The third Mel-spectrogram is obtained, and based on the third pitch feature and the third Mel-spectrogram in the third Mel-spectrogram, the converted speech corresponding to the current speech is generated. The beneficial effects are in two aspects. On the one hand, the third Mel-spectrogram output by the trained diffusion model based on the current speech is obtained, and based on the third pitch feature and the third Mel-spectrogram in the third Mel-spectrogram, the converted speech corresponding to the current speech is generated. Since the third pitch feature and the third Mel-spectrogram are processed, the distortion degree of the converted speech of the current speech can be reduced. Therefore, the current speech does not need to be repeatedly converted. Therefore, the conversion time of the current speech is reduced, which is beneficial to improving the conversion efficiency of the current speech. On the other hand, since the trained diffusion model will not be affected by human intervention, it is beneficial to improve the reliability of the converted speech corresponding to the current speech. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0040] Figure 1 is a schematic diagram of an application environment of a voice conversion method according to an embodiment of the present invention;
[0041] Figure 2 A flow chart of a voice conversion method provided by an embodiment of the present invention;
[0042] Figure 3 yes Figure 1 A schematic flow chart of a specific implementation of step S23;
[0043] Figure 4 yes Figure 1A schematic flow chart of a specific implementation of step S25;
[0044] Figure 5 yes Figure 1 A schematic flow chart of a specific implementation of step S26;
[0045] Figure 6 is a structural schematic diagram of a speech conversion device in one embodiment of the present invention;
[0046] Figure 7 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;
[0047] Figure 8 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION
[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0049] See also Figure 1 , Figure 1 FIG. 1 is a schematic diagram of an application environment of a voice conversion method according to an embodiment of the present invention. The voice conversion method provided by the embodiment of the present invention can be applied in the following embodiments: Figure 1 In an application environment, a client communicates with a server through a network.
[0050] The server obtains the original voice and the target voice corresponding to the original voice through the client;
[0051] Obtain a first Mel-spectrogram output by a preset diffusion model based on the original speech, and obtain a second Mel-spectrogram corresponding to the target speech;
[0052] Based on the first Mel-spectrogram, a first pitch feature and a first Mel-spectrogram feature are obtained; based on the second Mel-spectrogram, a second pitch feature and a second Mel-spectrogram feature are obtained;
[0053] Obtaining a first loss value between the first pitch feature and the second pitch feature, and obtaining a second loss value between the first Mel spectrum feature and the second Mel spectrum feature;
[0054] Based on the first loss value, the second loss value and a preset acquisition method, acquiring the trained diffusion model;
[0055] A third Mel-spectrogram outputted by the trained diffusion model based on the current speech is obtained, and a converted speech corresponding to the current speech is generated based on a third pitch feature and a third Mel-spectrogram feature in the third Mel-spectrogram.
[0056] In the scheme implemented by the above-mentioned speech conversion method, device, equipment and medium, the beneficial effects are in two aspects. On the one hand, the third Mel-spectrogram outputted by the diffusion model after training is obtained based on the current speech, and the converted speech corresponding to the current speech is generated based on the third pitch feature and the third Mel-spectrogram feature in the third Mel-spectrogram. Since the third pitch feature and the third Mel-spectrogram feature are processed, the distortion degree of the converted speech of the current speech can be reduced, so the current speech does not need to be repeatedly converted, thereby reducing the conversion time of the current speech, which is beneficial to improving the conversion efficiency of the current speech. On the other hand, since the diffusion model after training will not be affected by human intervention, it is beneficial to improve the reliability of the converted speech corresponding to the current speech.
[0057] The device running the client is referred to as a client device.
[0058] The device running the server is referred to as the server device.
[0059] The client devices may include but are not limited to various personal computers, laptops, smart phones, tablet computers, and portable wearable devices.
[0060] The server device can be implemented with an independent task database or a task database cluster composed of multiple task databases. The present invention is described in detail below through specific embodiments.
[0061] See also Figure 2 , Figure 2 A flow chart of a voice conversion method provided by an embodiment of the present invention includes the following steps:
[0062] S21, obtaining an original speech and a target speech corresponding to the original speech;
[0063] Exemplarily, obtaining an original speech and a target speech corresponding to the original speech includes:
[0064] The customer service voice of the target platform is selected as the source voice, the source voice and the interference voice are mixed to obtain the original voice, and the target voice corresponding to the original voice is obtained.
[0065] The target speech is the speech of the target speaker.
[0066] Among them, the target platforms include financial platforms, technology platforms and insurance platforms.
[0067] A financial platform is an online service platform built on advanced information technology. It integrates a variety of financial services and functions, and provides users with convenient financial transaction and management services through computer networks.
[0068] A technology platform refers to a comprehensive online platform built based on modern information technology that provides technical services, application development, data processing and resource sharing to developers, enterprises or individuals.
[0069] Insurance platform refers to an online platform built by using modern information technology to serve the insurance industry. The insurance platform not only provides a one-stop service process such as insurance product display, online insurance, claims processing, customer service, etc., but also realizes the design, pricing, risk management and data analysis of insurance products through a complex back-end system.
[0070] S22, obtaining a first mel-spectrogram output by a preset diffusion model based on the original speech, and obtaining a second mel-spectrogram corresponding to the target speech;
[0071] The step of obtaining a preset diffusion model based on a first mel-spectrogram output by the original speech to obtain a second mel-spectrogram corresponding to the target speech includes:
[0072] Convert the original speech into a first text content, perform text feature extraction on the first text content to obtain a first text vector, input the first text vector into a preset diffusion model, and obtain a first Mel-spectrogram output by the diffusion model based on the first text vector;
[0073] The target speech is converted into a second text content, text features are extracted for the second text content to obtain a second text vector, the second text vector is input into a decoder, and a second Mel-Mel spectrogram output by the decoder is obtained.
[0074] S23, based on the first Mel-spectrogram, obtaining a first pitch feature and a first Mel-spectrogram feature, and based on the second Mel-spectrogram, obtaining a second pitch feature and a second Mel-spectrogram feature;
[0075] S24, obtaining a first loss value between the first pitch feature and the second pitch feature, and obtaining a second loss value between the first Mel spectrum feature and the second Mel spectrum feature;
[0076] The step of obtaining a first loss value between the first pitch feature and the second pitch feature, and obtaining a second loss value between the first Mel spectrum feature and the second Mel spectrum feature, includes:
[0077] Obtaining a first loss value between the first pitch feature and the second pitch feature through a preset absolute loss function;
[0078] A second loss value between the first Mel frequency spectrum feature and the second Mel frequency spectrum feature is obtained through the absolute loss function.
[0079] S25, acquiring the trained diffusion model based on the first loss value, the second loss value and a preset acquisition method;
[0080] Among them, the first loss value serves as an indicator to measure the difference between the first pitch feature and the second pitch feature. By continuously minimizing the first loss value, the diffusion model can gradually adjust its internal parameters, thereby improving the diffusion model's learning ability in pitch.
[0081] The second loss value is used as an indicator to measure the difference between the first Mel-spectrum feature and the second Mel-spectrum feature. By continuously minimizing the second loss value, the diffusion model can gradually adjust its internal parameters, thereby improving the learning ability of the diffusion model in terms of frequency.
[0082] S26, obtaining a third Mel-spectrogram output by the trained diffusion model based on the current speech, and generating a converted speech corresponding to the current speech based on a third pitch feature and a third Mel-spectrogram feature in the third Mel-spectrogram.
[0083] Wherein, after acquiring the third Mel-spectrogram output by the diffusion model after training based on the current speech, and generating the converted speech corresponding to the current speech based on the third pitch feature and the third Mel-spectrogram feature in the third Mel-spectrogram, the speech conversion method includes:
[0084] A push instruction of the converted voice is obtained, the push instruction is executed, and the converted voice is pushed to a preset system.
[0085] In an embodiment of the present application, the converted speech is pushed to a preset system, and the developer can analyze the converted speech corresponding to the current speech and further optimize the speech conversion method based on the analysis results.
[0086] In the embodiment of the present invention, the beneficial effects are in two aspects. On the one hand, the third Mel-spectrogram outputted by the diffusion model after training is obtained based on the current speech, and the converted speech corresponding to the current speech is generated based on the third pitch feature and the third Mel-spectrogram feature in the third Mel-spectrogram. Since the third pitch feature and the third Mel-spectrogram feature are processed, the distortion degree of the converted speech of the current speech can be reduced, so the current speech does not need to be repeatedly converted, thereby reducing the conversion time of the current speech, which is beneficial to improving the conversion efficiency of the current speech. On the other hand, since the diffusion model after training will not be affected by human intervention, it is beneficial to improve the reliability of the converted speech corresponding to the current speech.
[0087] See also Figure 3 , Figure 3 yes Figure 1 A specific implementation flow diagram of step S23 is described in detail as follows:
[0088] S31, selecting the pitch feature of the first Mel-spectrogram as a first pitch feature, and selecting the Mel-spectrogram feature of the first Mel-spectrogram as a first Mel-spectrogram feature;
[0089] Selecting the pitch feature of the first mel-spectrogram as the first pitch feature includes:
[0090] The encoder is used to perform feature extraction on the first Mel-spectrogram to obtain a pitch feature of the first Mel-spectrogram, and the pitch feature of the first Mel-spectrogram is selected as the first pitch feature.
[0091] S32: Select the pitch feature of the second Mel-spectrogram as the second pitch feature, and select the Mel-spectrogram feature of the second Mel-spectrogram as the second Mel-spectrogram feature.
[0092] Selecting the pitch feature of the second Mel-spectrogram as the second pitch feature includes:
[0093] A feature extraction is performed on the first mel-spectrogram by using a YAAPT algorithm to obtain a pitch feature of the second mel-spectrogram, and the pitch feature of the second mel-spectrogram is selected as a second pitch feature.
[0094] Among them, the Chinese definition of YAAPT algorithm is: Another pitch tracking algorithm, and the English definition of YAAPT algorithm is: Yet Another Algorithm for Pitch Tracking.
[0095] Among them, the YAAPT algorithm is an algorithm specifically used for pitch tracking. The YAAPT algorithm is designed to accurately and quickly extract pitch information from audio signals. Therefore, in terms of pitch extraction, the YAAPT algorithm has higher professionalism and accuracy than the encoder.
[0096] In the embodiment of the present invention, the difference between the first pitch feature and the second pitch feature and the difference between the second Mel spectrum feature and the first Mel spectrum feature are used to facilitate training of the diffusion model.
[0097] See also Figure 4 , Figure 4 yes Figure 1 A specific implementation flow diagram of step S25 is described in detail as follows:
[0098] S41, training the diffusion model by using the first loss value and the second loss value;
[0099] S42: When both the first loss value and the second loss value are smaller than a preset value, stop training the diffusion model and obtain the trained diffusion model.
[0100] In the embodiment of the present invention, the training of the diffusion model is stopped, and the trained diffusion model is obtained. The trained diffusion model can meet the needs of different users and is applicable to various application scenarios.
[0101] See also Figure 5 , Figure 5 yes Figure 1 A specific implementation flow diagram of step S26 is described in detail as follows:
[0102] S51, obtaining a current speech, inputting the current speech into the trained diffusion model, obtaining a third Mel-spectrogram output by the trained diffusion model based on the current speech, selecting a pitch feature of the third Mel-spectrogram as a third pitch feature, and selecting a Mel-spectrogram feature of the third Mel-spectrogram as a third Mel-spectrogram feature;
[0103] S52, input the third pitch feature and the third Mel spectrum feature into a preset speech converter, obtain a fourth Mel spectrum graph output by the speech converter, input the fourth Mel spectrum graph into a preset vocoder, convert the fourth Mel spectrum graph through the vocoder, and generate a converted speech corresponding to the current speech.
[0104] In an embodiment of the present invention, the fourth Mel-spectrogram is converted by the voice coder to generate a converted speech corresponding to the current speech, which can reduce the distortion of the converted speech of the current speech. Therefore, the current speech does not need to be repeatedly converted, thereby reducing the conversion time of the current speech and helping to improve the conversion efficiency of the current speech.
[0105] See also Figure 6 , Figure 6 FIG. 1 is a schematic diagram of a structure of a speech conversion device according to an embodiment of the present invention. Figure 6 As shown, the speech conversion device includes a first acquisition module 101, a second acquisition module 102, a third acquisition module 103, a fourth acquisition module 104, a fifth acquisition module 105, and a conversion module 106. The functional modules are described in detail as follows:
[0106] A first acquisition module 101 is used to acquire an original speech and a target speech corresponding to the original speech;
[0107] A second acquisition module 102 is used to acquire a first Mel-spectrogram output by a preset diffusion model based on the original speech, and acquire a second Mel-spectrogram corresponding to the target speech;
[0108] A third acquisition module 103 is used to acquire a first pitch feature and a first Mel spectrum feature based on the first Mel spectrum graph, and acquire a second pitch feature and a second Mel spectrum feature based on the second Mel spectrum graph;
[0109] A fourth acquisition module 104 is used to acquire a first loss value between the first pitch feature and the second pitch feature, and acquire a second loss value between the first Mel spectrum feature and the second Mel spectrum feature;
[0110] A fifth acquisition module 105, configured to acquire the trained diffusion model based on the first loss value, the second loss value and a preset acquisition method;
[0111] The conversion module 106 is used to obtain a third Mel-spectrogram output by the trained diffusion model based on the current speech, and generate a converted speech corresponding to the current speech based on a third pitch feature and a third Mel-spectrogram feature in the third Mel-spectrogram.
[0112] In one embodiment, the second acquisition module 102 includes:
[0113] A first acquisition subunit is used to convert the original speech into a first text content, perform text feature extraction on the first text content to obtain a first text vector, input the first text vector into a preset diffusion model, and obtain a first Mel-spectrogram output by the diffusion model based on the first text vector;
[0114] The second acquisition subunit is used to convert the target speech into a second text content, perform text feature extraction on the second text content to obtain a second text vector, input the second text vector into a decoder, and obtain a second Mel-Mel spectrogram output by the decoder.
[0115] In one embodiment, the third acquisition module 103 includes:
[0116] A first selection subunit is used to select the pitch feature of the first Mel-spectrogram as the first pitch feature, and select the Mel-spectrogram feature of the first Mel-spectrogram as the first Mel-spectrogram feature;
[0117] The second selection subunit is used to select the pitch feature of the second Mel-spectrogram as the second pitch feature, and select the Mel-spectrogram feature of the second Mel-spectrogram as the second Mel-spectrogram feature.
[0118] In one embodiment, the fourth acquisition module 104 includes:
[0119] A third acquisition subunit, configured to acquire a first loss value between the first pitch feature and the second pitch feature by using a preset absolute loss function;
[0120] The fourth acquisition subunit is configured to acquire a second loss value between the first Mel-spectrum feature and the second Mel-spectrum feature by using the absolute loss function.
[0121] In one embodiment, the fifth acquisition module 105 includes:
[0122] A training subunit, configured to train the diffusion model using the first loss value and the second loss value;
[0123] The fifth acquisition subunit is used to stop training the diffusion model and acquire the trained diffusion model when both the first loss value and the second loss value are less than a preset value.
[0124] In one embodiment, the conversion module 106 includes:
[0125] An input subunit is used to obtain the current speech, input the current speech into the trained diffusion model, obtain a third Mel-spectrogram output by the trained diffusion model based on the current speech, select a pitch feature of the third Mel-spectrogram as a third pitch feature, and select a Mel-spectrogram feature of the third Mel-spectrogram as a third Mel-spectrogram feature;
[0126] The conversion subunit is used to input the third pitch feature and the third Mel spectrum feature into a preset speech converter, obtain a fourth Mel spectrum map output by the speech converter, input the fourth Mel spectrum map into a preset vocoder, convert the fourth Mel spectrum map through the vocoder, and generate a converted speech corresponding to the current speech.
[0127] In one embodiment, the speech conversion device further includes:
[0128] The push module is used to obtain a push instruction for the converted voice, execute the push instruction, and push the converted voice to a preset system.
[0129] In the embodiment of the present invention, the beneficial effects are in two aspects. On the one hand, the third Mel-spectrogram outputted by the diffusion model after training is obtained based on the current speech, and the converted speech corresponding to the current speech is generated based on the third pitch feature and the third Mel-spectrogram feature in the third Mel-spectrogram. Since the third pitch feature and the third Mel-spectrogram feature are processed, the distortion degree of the converted speech of the current speech can be reduced, so the current speech does not need to be repeatedly converted, thereby reducing the conversion time of the current speech, which is beneficial to improving the conversion efficiency of the current speech. On the other hand, since the diffusion model after training will not be affected by human intervention, it is beneficial to improve the reliability of the converted speech corresponding to the current speech.
[0130] For the specific definition of the voice conversion device, please refer to the definition of the voice conversion method above, which will not be repeated here.
[0131] Each module in the above-mentioned speech conversion device can be implemented in whole or in part by software, hardware and a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.
[0132] See also Figure 7 , Figure 7 is a schematic diagram of a computer device in one embodiment of the present invention. In one embodiment, a computer device is provided. The computer device may be a server device, and its internal structure diagram may be as shown in FIG. Figure 7 The computer device includes a processor, a memory, a network interface and a database connected through a system bus.
[0133] The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, the functions or steps of a speech conversion method based on a diffusion model can be implemented on the server device.
[0134] See also Figure 8 , Figure 8 is another structural diagram of a computer device in one embodiment of the present invention. In one embodiment, a computer device is provided. The computer device may be a client device, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external task database through a network connection. When the computer program is executed by the processor, the functions or steps of a speech conversion method based on a diffusion model can be implemented on a client device.
[0135] In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor.
[0136] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant description in the aforementioned method embodiment. In order to avoid repetition, they will not be described one by one here.
[0137] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.
[0138] The above description and accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent possible changes only. Unless explicitly required, separate components and functions are optional, and the order of operation may vary. Parts and subsamples of some embodiments may be included in or replace parts and subsamples of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of a stated sub-sample, whole, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other sub-samples, wholes, steps, operations, elements, components and / or groups of these. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method or device comprising the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.
[0139] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. Technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present disclosure. Technicians can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0140] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some sub-samples can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.
[0141] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and the block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.
Claims
1. A speech conversion method based on a diffusion model, characterized in that: include: Acquire an original voice and a target voice corresponding to the original voice; Obtain a first Mel-spectrogram output by a preset diffusion model based on the original speech, and obtain a second Mel-spectrogram corresponding to the target speech; Based on the first Mel-spectrogram, a first pitch feature and a first Mel-spectrogram feature are obtained; based on the second Mel-spectrogram, a second pitch feature and a second Mel-spectrogram feature are obtained; Obtaining a first loss value between the first pitch feature and the second pitch feature, and obtaining a second loss value between the first Mel spectrum feature and the second Mel spectrum feature; Based on the first loss value, the second loss value and a preset acquisition method, acquiring the trained diffusion model; A third Mel-spectrogram outputted by the trained diffusion model based on the current speech is obtained, and a converted speech corresponding to the current speech is generated based on a third pitch feature and a third Mel-spectrogram feature in the third Mel-spectrogram.
2. The voice conversion method according to claim 1, characterized in that: The step of obtaining a preset diffusion model based on a first Mel-spectrogram output by the original speech to obtain a second Mel-spectrogram corresponding to the target speech includes: Convert the original speech into a first text content, perform text feature extraction on the first text content to obtain a first text vector, input the first text vector into a preset diffusion model, and obtain a first Mel-spectrogram output by the diffusion model based on the first text vector; The target speech is converted into a second text content, text features are extracted for the second text content to obtain a second text vector, the second text vector is input into a decoder, and a second Mel-Mel spectrogram output by the decoder is obtained.
3. The voice conversion method according to claim 1, characterized in that: The step of acquiring a first pitch feature and a first Mel spectrum feature based on the first Mel spectrum graph, and acquiring a second pitch feature and a second Mel spectrum feature based on the second Mel spectrum graph includes: Selecting the pitch feature of the first Mel-spectrogram as the first pitch feature, and selecting the Mel-spectrogram feature of the first Mel-spectrogram as the first Mel-spectrogram feature; The pitch feature of the second Mel-spectrogram is selected as the second pitch feature, and the Mel-spectrogram feature of the second Mel-spectrogram is selected as the second Mel-spectrogram feature.
4. The method for voice conversion according to claim 1, characterized in that: The obtaining of a first loss value between the first pitch feature and the second pitch feature, and the obtaining of a second loss value between the first Mel spectrum feature and the second Mel spectrum feature, include: Obtaining a first loss value between the first pitch feature and the second pitch feature through a preset absolute loss function; A second loss value between the first Mel frequency spectrum feature and the second Mel frequency spectrum feature is obtained through the absolute loss function.
5. The voice conversion method according to claim 1, characterized in that: The acquiring the trained diffusion model based on the first loss value, the second loss value and a preset acquisition method includes: Training the diffusion model by using the first loss value and the second loss value; When both the first loss value and the second loss value are smaller than preset values, the training of the diffusion model is stopped, and the trained diffusion model is obtained.
6. The voice conversion method according to claim 1, characterized in that: The step of acquiring a third Mel-spectrogram outputted by the diffusion model after training based on the current speech, and generating a converted speech corresponding to the current speech based on a third pitch feature and a third Mel-spectrogram feature in the third Mel-spectrogram, comprises: Acquire a current speech, input the current speech into the trained diffusion model, acquire a third Mel-spectrogram output by the trained diffusion model based on the current speech, select a pitch feature of the third Mel-spectrogram as a third pitch feature, and select a Mel-spectrogram feature of the third Mel-spectrogram as a third Mel-spectrogram feature; The third pitch feature and the third Mel spectrum feature are input into a preset speech converter, a fourth Mel spectrum graph output by the speech converter is obtained, the fourth Mel spectrum graph is input into a preset vocoder, the fourth Mel spectrum graph is converted by the vocoder, and a converted speech corresponding to the current speech is generated.
7. The method for voice conversion according to any one of claims 1 to 6, characterized in that: After acquiring the third Mel-spectrogram output by the diffusion model after training based on the current speech, and generating the converted speech corresponding to the current speech based on the third pitch feature and the third Mel-spectrogram feature in the third Mel-spectrogram, the speech conversion method includes: A push instruction of the converted voice is obtained, the push instruction is executed, and the converted voice is pushed to a preset system.
8. A speech conversion device based on a diffusion model, characterized in that: include: A first acquisition module, used to acquire an original voice and a target voice corresponding to the original voice; A second acquisition module is used to acquire a first Mel-spectrogram output by a preset diffusion model based on the original speech, and acquire a second Mel-spectrogram corresponding to the target speech; A third acquisition module, configured to acquire a first pitch feature and a first Mel spectrum feature based on the first Mel spectrum graph, and acquire a second pitch feature and a second Mel spectrum feature based on the second Mel spectrum graph; a fourth acquisition module, configured to acquire a first loss value between the first pitch feature and the second pitch feature, and acquire a second loss value between the first Mel spectrum feature and the second Mel spectrum feature; a fifth acquisition module, configured to acquire the trained diffusion model based on the first loss value, the second loss value and a preset acquisition method; The conversion module is used to obtain the third Mel-spectrogram output by the trained diffusion model based on the current speech, and generate a converted speech corresponding to the current speech based on the third pitch feature and the third Mel-spectrogram feature in the third Mel-spectrogram.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the speech conversion method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the speech conversion method according to any one of claims 1 to 7 are implemented.