Singing conversion method and device, electronic equipment and storage medium

By calculating the basic frequency adjustment characteristics and adjusting the source basic frequency characteristics, the problem of inconsistent melody in singing conversion is solved, and the quality of song audio is improved.

CN119091854APending Publication Date: 2024-12-06PING AN TECH (SHENZHEN) CO LTD
View PDF 0 Cites 6 Cited by

Patent Information

Application Number
CN202411250557.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-06
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The prior art is difficult to maintain consistent melody when singing voices are converted, resulting in low audio quality of converted songs.

Method used

By obtaining the characteristics of the target voice and source song audio, the fundamental frequency adjustment characteristics are calculated, and based on this, the target song audio is generated to keep the melody unchanged.

Benefits of technology

It realizes the melody consistency during singing conversion, and improves the audio quality of the converted target song.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091854A_ABST
    Figure CN119091854A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a song conversion method and device, electronic equipment and a storage medium, and belongs to the technical field of digital medical treatment. Comprising the following steps: acquiring a target voice, and extracting a target voiceprint feature and a target fundamental frequency feature from the target voice; obtaining a source song audio, and determining a source text feature, a source fundamental frequency feature and a source audio distribution feature corresponding to the source song audio based on the source song audio; determining a fundamental frequency adjustment feature based on the target fundamental frequency feature and the source fundamental frequency feature, and adjusting the source fundamental frequency feature based on the fundamental frequency adjustment feature to obtain an adjusted fundamental frequency feature; and generating a target song audio based on the adjusted fundamental frequency feature, the target voiceprint feature, the source text feature and the source audio distribution feature. The fundamental frequency adjustment feature is determined based on the mean value difference of the target fundamental frequency feature and the source fundamental frequency feature, the source fundamental frequency feature is adjusted based on the fundamental frequency adjustment feature, singing synthesis is carried out based on the adjusted fundamental frequency feature, and therefore it is guaranteed that the audio melody obtained through singing conversion is consistent with the source audio.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of digital medical technology, and in particular to a singing voice conversion method and device, an electronic device, and a storage medium. Background Art

[0002] With the development of artificial intelligence technology, singing voice conversion has been widely used in various fields. For example, in medical scenarios, music can be used to relieve the subject's emotions, and the subject may have preferences for different singing objects. At this time, singing voice conversion can be used to convert a song audio from singing object A to singing object B to better meet the needs of the subject and thus better meet the subject's needs and link the subject's emotions.

[0003] Singing is a type of speech, and in the prior art, voice conversion technology is often used to convert singing. However, unlike conventional speech audio, song audio has an important feature, in addition to semantics and timbre. When using traditional voice conversion technology to convert singing, it is difficult for the converted song audio to keep the melody consistent with the original song audio, and the converted song audio quality is not high. Summary of the invention

[0004] The main purpose of the embodiments of the present disclosure is to propose a singing voice conversion method and device, electronic device, and computer-readable storage medium, which can keep the melody of the song unchanged during the singing voice conversion, and obtain high-quality target song audio after the singing voice conversion.

[0005] To achieve the above object, a first aspect of an embodiment of the present application provides a singing voice conversion method, the method comprising:

[0006] Acquire a target voice, and extract a target voiceprint feature and a target fundamental frequency feature from the target voice;

[0007] Acquire source song audio, and determine source text features, source fundamental frequency features, and source audio distribution features corresponding to the source song audio based on the source song audio;

[0008] Determine a fundamental frequency adjustment feature based on the target fundamental frequency feature and the source fundamental frequency feature, and adjust the source fundamental frequency feature based on the fundamental frequency adjustment feature to obtain an adjusted fundamental frequency feature, wherein the fundamental frequency adjustment feature is a difference between a mean value of the target fundamental frequency feature and a mean value of the source fundamental frequency feature;

[0009] The target song audio is generated based on the adjusted fundamental frequency feature, the target voiceprint feature, the source text feature and the source audio distribution feature.

[0010] In some embodiments, determining a fundamental frequency adjustment feature based on the target fundamental frequency feature and the source fundamental frequency feature, and adjusting the source fundamental frequency feature based on the fundamental frequency adjustment feature to obtain an adjusted fundamental frequency feature includes:

[0011] Performing logarithmic processing on the target fundamental frequency feature and the source fundamental frequency feature to obtain a logarithmic target fundamental frequency feature and a logarithmic source fundamental frequency feature;

[0012] Determine a first feature mean corresponding to the logarithmic target fundamental frequency feature and a second feature mean corresponding to the logarithmic source fundamental frequency feature, and determine the fundamental frequency adjustment feature based on the first feature mean and the second feature mean;

[0013] The logarithmic source fundamental frequency characteristic is adjusted based on the fundamental frequency adjustment characteristic, and the adjustment result is subjected to exponential processing to obtain the adjusted fundamental frequency characteristic.

[0014] In some embodiments, before determining the fundamental frequency adjustment feature based on the target fundamental frequency feature and the source fundamental frequency feature, the method further includes:

[0015] Obtaining standard melody information corresponding to the source song audio;

[0016] Determine first melody information of the source song audio based on the source fundamental frequency feature;

[0017] The feature points to be corrected in the source fundamental frequency feature are determined based on the first melody information and the standard melody information, and the feature points to be corrected are corrected to obtain the corrected source fundamental frequency feature.

[0018] In some embodiments, generating the target song audio based on the adjusted fundamental frequency feature, the target voiceprint feature, the source text feature, and the source audio distribution feature includes:

[0019] Sampling from the source audio distribution feature to obtain an acoustic sampling feature corresponding to the source audio, and superimposing the acoustic sampling feature and the source text feature to obtain a source audio superposition feature;

[0020] The source audio superposition feature and the adjusted fundamental frequency feature are concatenated, and the concatenated source audio superposition feature and the adjusted fundamental frequency feature are input into a preset decoder for decoding together with the target voiceprint feature to obtain a decoded audio feature;

[0021] The decoded audio features and the target voiceprint features are input into a preset vocoder for audio generation to obtain the target song audio corresponding to the target voiceprint features.

[0022] In some embodiments, determining the source text feature, source fundamental frequency feature, and source audio distribution feature corresponding to the source song audio based on the source song audio includes:

[0023] Convert the source song audio into corresponding linear spectrum features, and encode the linear spectrum features through a spectrum encoder to obtain initial text features and the source audio distribution features;

[0024] Quantifying the initial text features to obtain the source text features;

[0025] The source fundamental frequency feature of the source song audio is extracted by a fundamental frequency extractor.

[0026] In some embodiments, the initial text features include a plurality of initial text vectors, and the quantizing process of the initial text features to obtain the source text features includes:

[0027] Inputting the initial text features into a preset quantizer, wherein the quantizer includes a plurality of preset text vectors;

[0028] In the quantizer, for each of the initial text vectors, a target text vector corresponding to the initial text vector is determined from the multiple preset text vectors, wherein the target text vector is a preset text vector having the smallest distance from the initial text vector among the multiple preset text vectors;

[0029] The target text vectors corresponding to the initial text vectors are concatenated to obtain the source text features.

[0030] In some embodiments, the baseband extractor is trained by:

[0031] Acquire a sample source audio, and extract a first sample fundamental frequency feature from the sample source audio by the fundamental frequency extractor;

[0032] Extracting sample text features, sample audio distribution features, and sample voiceprint features from the sample source audio, sampling from the sample audio distribution features to obtain sample acoustic sampling features, and superimposing the first sample fundamental frequency features and the sample acoustic sampling features to obtain sample superimposed audio features;

[0033] splicing the sample superimposed audio feature and the first sample feature to obtain a sample spliced ​​audio feature, and inputting the sample spliced ​​audio feature and the sample voiceprint feature into a preset decoder for decoding to obtain a sample decoded audio feature;

[0034] Inputting the sample decoded feature into a vocoder to generate sample converted audio, and extracting a second sample fundamental frequency feature from the sample converted audio by the fundamental frequency extractor;

[0035] A cross entropy calculation is performed based on the first sample fundamental frequency feature and the second sample fundamental frequency feature to obtain a first loss value, and the fundamental frequency extractor is optimized based on the first loss value.

[0036] To achieve the above object, a second aspect of an embodiment of the present application provides a singing voice conversion device, the device comprising:

[0037] A first acquisition unit, configured to acquire a target voice and extract a target voiceprint feature and a target fundamental frequency feature from the target voice;

[0038] A second acquisition unit is used to acquire a source song audio, and determine a source text feature, a source fundamental frequency feature, and a source audio distribution feature corresponding to the source song audio based on the source song audio;

[0039] a fundamental frequency adjustment unit, configured to determine a fundamental frequency adjustment feature based on the target fundamental frequency feature and the source fundamental frequency feature, and adjust the source fundamental frequency feature based on the fundamental frequency adjustment feature to obtain an adjusted fundamental frequency feature, wherein the fundamental frequency adjustment feature is a difference between a mean value of the target fundamental frequency feature and a mean value of the source fundamental frequency feature;

[0040] The singing synthesis unit is used to generate target song audio based on the adjusted fundamental frequency feature, the target voiceprint feature, the source text feature and the source audio distribution feature.

[0041] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the singing voice conversion method described in the first aspect is implemented.

[0042] To achieve the above-mentioned purpose, the fourth aspect of the embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the singing voice conversion method described in the first aspect above.

[0043] The third aspect of the embodiments of the present disclosure proposes an electronic device, which includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the singing voice conversion method as described in any one of the embodiments of the first aspect of the present application is realized.

[0044] The fourth aspect of the embodiments of the present disclosure proposes a storage medium, which is a computer-readable storage medium used for computer-readable storage, and is characterized in that the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the singing voice conversion method as described in any one of the embodiments of the first aspect of the present application.

[0045] The singing voice conversion method and device, electronic device, and storage medium proposed in the embodiments of the present disclosure obtain a target voice and extract a target voiceprint feature and a target fundamental frequency feature from the target voice; obtain a source song audio, and determine a source text feature, a source fundamental frequency feature, and a source audio distribution feature corresponding to the source song audio based on the source song audio; determine a fundamental frequency adjustment feature based on the target fundamental frequency feature and the source fundamental frequency feature, and adjust the source fundamental frequency feature based on the fundamental frequency adjustment feature to obtain an adjusted fundamental frequency feature, wherein the fundamental frequency adjustment feature is the difference between the mean of the target fundamental frequency feature and the mean of the source fundamental frequency feature; generate a target song audio based on the adjusted fundamental frequency feature, the target voiceprint feature, the source text feature, and the source audio distribution feature. By extracting the fundamental frequency information and target fundamental frequency features of the source song audio, determining the fundamental frequency adjustment features based on the difference between the means of the two, and then adjusting the source fundamental frequency features as a whole based on the fundamental frequency adjustment features, so that the source fundamental frequency features rise or fall as a whole, when mapping the sound range of the source song audio to the target sound range, the overall melody of the source song audio is kept unchanged, so that the melody of the converted target song audio is consistent with the melody of the source song audio, thereby improving the quality of the converted target song audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of a singing voice conversion method provided in an embodiment of the present application;

[0047] Figure 2 yes Figure 1 Flow chart of step S103 in FIG.

[0048] Figure 3 yes Figure 1 Another flow chart of step S103 in FIG. 1 ;

[0049] Figure 4 yes Figure 1 Flow chart of step S104 in FIG.

[0050] Figure 5 yes Figure 1 Flow chart of step S102 in FIG.

[0051] Figure 6 yes Figure 1 Flow chart of step S502 in FIG.

[0052] Figure 7 is a flowchart of training a decoder according to an embodiment of the present application;

[0053] Figure 8 yes Figure 1 Flow chart of step S702 in FIG.

[0054] Fig. 9 is a structural schematic diagram of a singing voice conversion device provided in an embodiment of the present application;

[0055] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0059] The singing voice conversion method and device, electronic device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the data processing method in the embodiments of the present application is described.

[0060] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0061] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0062] The singing voice conversion method provided in the embodiment of the present application relates to the field of digital medical technology. The singing voice conversion method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the singing voice conversion method, etc., but is not limited to the above forms.

[0063] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0064] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to the identity or characteristics of the object, such as object information, object behavior data, object historical data, and object location information, the permission or consent of the object will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the personal information of the object, the separate permission or separate consent of the object will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the separate permission or separate consent of the object, the necessary object-related data for the normal operation of the embodiment of the present application will be obtained.

[0065] Figure 1 is an optional flow chart of a singing voice conversion method provided in an embodiment of the present application. Figure 1 The singing voice conversion method may include but is not limited to steps S101 to S104.

[0066] Step S101, obtaining a target voice, and extracting a target voiceprint feature and a target fundamental frequency feature from the target voice;

[0067] Step S102, obtaining source song audio, and determining source text features, source fundamental frequency features, and source audio distribution features corresponding to the source song audio based on the source song audio;

[0068] Step S103, determining a fundamental frequency adjustment feature based on the target fundamental frequency feature and the source fundamental frequency feature, and adjusting the source fundamental frequency feature based on the fundamental frequency adjustment feature to obtain an adjusted fundamental frequency feature, wherein the fundamental frequency adjustment feature is a difference between a mean value of the target fundamental frequency feature and a mean value of the source fundamental frequency feature;

[0069] Step S104, generating target song audio based on the adjusted fundamental frequency features, target voiceprint features, source text features and source audio distribution features.

[0070] In step S101, the target voice is a piece of voice data provided by the target speaker, and the target speaker can be the user himself or an object designated by the user. For example, when the user wants to convert a piece of singing into a singing with his own voiceprint characteristics, the user can input the target voice through the terminal microphone; or, in the digital medical scenario, the user wants to listen to song a sung by singing object A, and song a itself is sung by singing object B. By selecting the corresponding singing object A in the terminal, and then retrieving the corresponding voice of singing object B from the preset database as the target voice, the corresponding target voiceprint characteristics and target pitch characteristics can be extracted, so as to convert song a from singing object B to singing object A in the future, so as to meet the needs of the user and assist the user. The target fundamental frequency feature is a feature representation corresponding to the fundamental frequency of the target speech, which can reflect the basic pitch of the target speaker. Specifically, the target fundamental frequency feature can be extracted from the target speech through a fundamental frequency extraction module in a WORLD vocoder. Exemplarily, the WORLD vocoder can extract the fundamental frequency based on the harvest algorithm or the DIO algorithm. The target voiceprint feature is a feature representation used to reflect the timbre of the target speaker corresponding to the target speech. Specifically, the corresponding target voiceprint feature can be extracted from the target speech through a trained speaker encoder. The speaker encoder can be a trained speaker recognition network.

[0071] In step S102, the source song audio is an audio file of a song selected by the user, or an audio file of a song uploaded by the user; the source text feature is the text corresponding to the source song audio. It can be understood that the source song audio is a song, and the corresponding source text feature is the feature representation corresponding to the lyrics of the song, such as the feature vector corresponding to the lyrics; the source fundamental frequency feature is the feature representation corresponding to the fundamental frequency of the source song audio. The details of the source fundamental frequency feature can refer to the relevant description of the target fundamental frequency feature in step S101, which will not be repeated here. It should be noted that the source fundamental frequency feature extracted by the WORLD vocoder will carry information corresponding to the melody of the source song audio; the source audio distribution feature is a feature used to represent the distribution of acoustic information of the source song audio. The source audio distribution feature can be a Gaussian distribution. In this embodiment, the source audio distribution feature can be a standard Gaussian distribution, that is, a Gaussian distribution with a mean of 0 and a variance of 1, which can be recorded as N(0,1). Specifically, the corresponding source text features and source audio distribution features can be extracted by inputting the source song audio into a trained spectrum encoder. The spectrum encoder can be a neural network model based on the encoder structure in the variational autoencoder (VAE), which can map the input source song audio data to a latent space and represent the source song audio as a distribution. This latent space can be represented by a standard Gaussian distribution, and then the encoder outputs the mean and variance of this distribution, thereby obtaining the corresponding source audio distribution features. At the same time, the spectrum encoder can also map the source audio features into a continuous latent vector in the latent space, and the source text features can be obtained after quantizing this latent vector.

[0072] It should be noted that the singing voice conversion method of the embodiment of the present application can be used in various medical scenarios in the field of digital medicine. For example, when conducting rehabilitation training for a certain subject, the subject can initiate a singing voice conversion request based on the subject's interest preference, and change the source audio a played in the rehabilitation training from being sung by subject A to being sung by subject B. At this time, the source audio a is determined as the source song audio, and subject B is determined as the target speaker and the corresponding target voice is obtained.

[0073] In step S103, the fundamental frequency adjustment feature is the adjustment amount required to map the range of the source song audio to the range of the target speaker. Specifically, the target fundamental frequency feature is the lowest frequency component in the multiple sine wave signals obtained by decomposing the target speech, which reflects the basic pitch of the target speaker; similarly, the source fundamental frequency feature reflects the basic pitch of the singer of the source song audio; by averaging the target fundamental frequency feature and the source fundamental frequency feature respectively, and then performing a difference process on the mean of the two, the difference between the basic pitch of the target speaker and the basic pitch corresponding to the source song audio can be determined, that is, the basic pitch of the source song audio is adjusted to the adjustment amount required for the basic pitch of the target speaker, thereby obtaining the fundamental frequency adjustment feature. Afterwards, the basic fundamental frequency adjustment feature adjusts the source fundamental frequency feature, thereby obtaining the adjusted fundamental frequency feature, which represents the result obtained after mapping the basic pitch of the source song audio to the basic pitch of the target speaker. Exemplarily, the fundamental frequency adjustment feature can be superimposed on the source fundamental frequency feature, so as to adjust the source fundamental frequency feature based on the fundamental frequency adjustment feature. It is understandable that since the fundamental frequency adjustment feature is obtained based on the difference between the mean of the source fundamental frequency feature and the target fundamental frequency feature, when the source fundamental frequency feature is adjusted based on the fundamental frequency adjustment feature, the source fundamental frequency feature is pitched up or down as a whole, so it will not affect the rhythm of the source song audio itself, and the adjusted fundamental frequency feature can retain the melody consistent with the source fundamental frequency feature. In this way, the melody of the target song audio subsequently generated based on the adjusted fundamental frequency feature can be consistent with the melody of the source song audio, avoiding the situation of being out of tune after the singing is converted.

[0074] In step S104, after obtaining the adjusted fundamental frequency feature, audio synthesis is performed based on the adjusted fundamental frequency feature, the target voiceprint feature, the source text feature and the source audio distribution feature, so that the target song audio obtained has the timbre and pitch characteristics of the target speaker corresponding to the target voice, thereby realizing the conversion of the source song audio into the target song audio having the voice characteristics of the target speaker. At the same time, since the fundamental frequency adjustment feature is obtained based on the difference between the mean values ​​of the source fundamental frequency feature and the target fundamental frequency feature, when the source fundamental frequency feature is adjusted based on the fundamental frequency adjustment feature, the source fundamental frequency feature is raised or lowered as a whole, so the rhythm of the source song audio itself will not be affected, and the adjusted fundamental frequency feature can retain the melody consistent with the source fundamental frequency feature. In this way, the melody of the target song audio generated based on the adjusted fundamental frequency feature can be consistent with the melody of the source song audio, and only the voiceprint feature and the basic pitch of the song audio before and after the singing conversion are converted, and the melody of the song audio will not change. Specifically, in step S102, the source text features and source audio distribution features are extracted from the source song audio by the encoder in the variational autoencoder structure. Accordingly, audio synthesis can be performed by the decoder module in the variational autoencoder structure, that is, the source text features, source audio distribution features, adjusted fundamental frequency features and target voiceprint features are used as inputs to the decoder, so that the decoder generates corresponding outputs, and obtains a feature representation corresponding to the target song audio, and then the feature representation is converted into the corresponding target song audio through the vocoder.

[0075] In step S101 to step S104, the corresponding source fundamental frequency features, source text features and source audio distribution features are extracted from the source song audio, and the voiceprint features and target fundamental frequency features corresponding to the target speaker are extracted from the target speech. Then, the fundamental frequency adjustment feature is determined based on the difference between the mean values ​​of the source fundamental frequency feature and the target fundamental frequency feature, and the source fundamental frequency feature is adjusted based on the fundamental frequency adjustment feature, so as to perform an overall up-or-down processing on the source fundamental frequency feature, and the fundamental frequency of the source song audio is mapped to the range of the target speaker while keeping the melody information consistent, thereby obtaining the adjusted fundamental frequency feature. Then, audio is generated based on the adjusted fundamental frequency feature, the target voiceprint feature, the source audio distribution feature and the source text feature, and the target song audio having the timbre and pitch characteristics of the target speaker and the melody and lyrics information unchanged is obtained, so as to ensure that the melody of the song audio does not change before and after the singing voice conversion.

[0076] In one embodiment, Figure 2 is an optional flowchart of step S103, refer to Figure 2 Step S103 includes but is not limited to:

[0077] Step S201, logarithmically processing the target fundamental frequency feature and the source fundamental frequency feature to obtain a logarithmic target fundamental frequency feature and a logarithmic source fundamental frequency feature;

[0078] Step S202, determining a first feature mean corresponding to the logarithmic target fundamental frequency feature and a second feature mean corresponding to the logarithmic source fundamental frequency feature, and determining a fundamental frequency adjustment feature based on the first feature mean and the second feature mean;

[0079] Step S203, adjusting the logarithmic source fundamental frequency feature based on the fundamental frequency adjustment feature, and performing exponential processing on the adjustment result to obtain the adjusted fundamental frequency feature.

[0080] In step S201, the logarithmic processing is to take the logarithm of the target fundamental frequency feature and the source fundamental frequency feature respectively. For example, the source fundamental frequency feature is recorded as F0, and after the logarithmic processing, logF0 is obtained, which is recorded as LF0, that is, the logarithmic source fundamental frequency feature; similarly, the logarithmic target fundamental frequency feature can be obtained, which is recorded as By performing logarithmic processing on the source fundamental frequency characteristics and the target fundamental frequency characteristics, the source fundamental frequency characteristics and the target fundamental frequency characteristics are changed from exponential distribution to linear distribution, which can be more in line with the hearing of the human ear. In this way, when calculating the adjusted fundamental frequency characteristics and adjusting the source fundamental frequency characteristics based on the adjusted fundamental frequency characteristics, the changes in pitch increase or pitch decrease can be more in line with the user's auditory perception.

[0081] In step S202, the mean value is calculated based on the logarithmic target fundamental frequency feature to obtain the corresponding first feature mean value, which can be recorded as It can be understood that the first feature mean is a feature representation with the same format as the logarithmic target fundamental frequency feature, and the element value at each position in the first feature mean is the same. The element value at each position of the first feature mean is the mean calculation result of the logarithmic target fundamental frequency feature. It can be expressed as [0,1,2,3,4,5,6], and the corresponding first eigenvalue is It can be expressed as [3,3,3,3,3,3,3]; similarly, based on the logarithmic source fundamental frequency feature, the mean value is calculated to obtain the corresponding second feature mean, which can be recorded as E(LF0). The corresponding fundamental frequency adjustment feature is determined based on the difference between the first feature mean and the second feature mean, and the fundamental frequency adjustment feature can be expressed as It can be understood that the fundamental frequency adjustment feature is also a feature representation having the same format as the logarithmic source fundamental frequency feature and the same value of the elements at each position.

[0082] In step S203, after obtaining the fundamental frequency adjustment feature, the logarithmic source fundamental frequency feature is adjusted based on the fundamental frequency adjustment feature, so that the source fundamental frequency feature is overall raised or lowered, and the range of the source song audio is mapped to the range of the target speaker corresponding to the target voice, while keeping the overall melody of the source song audio unchanged, and obtaining the corresponding adjustment result. Afterwards, since in step 201, the source fundamental frequency feature is logarithmized and then the fundamental frequency adjustment feature is calculated and used to adjust the logarithmic source fundamental frequency feature, after obtaining the adjustment result, the adjustment result needs to be restored to the format before the logarithmic processing. Based on this, the adjustment result is indexed. It is not difficult to understand that the process of indexation and the process of logarithmization are corresponding. When the logarithmization is performed, the logarithmization is performed with 10 as the base, and when the indexation is performed, the indexation is also performed with 10 as the base. Specifically, the adjustment result is recorded as LF0′, and the adjusted fundamental frequency feature is recorded as F0′, then there is F0′=e LF0′ .

[0083] In the embodiment disclosed in step S201 to step S203, the source fundamental frequency feature and the target fundamental frequency feature are logarithmically processed, and the fundamental frequency adjustment feature is determined based on the logarithmic source fundamental frequency feature and the logarithmic target fundamental frequency feature obtained by the logarithmic processing, and the fundamental frequency adjustment feature is superimposed on the logarithmic source fundamental frequency feature and then exponentially processed to obtain the adjusted fundamental frequency feature. The source fundamental frequency feature and the target fundamental frequency feature are changed from exponential distribution to linear distribution by logarithmic processing, which can be more in line with the auditory perception of the human ear, and then the corresponding fundamental frequency adjustment feature is calculated based on this. Then, when the source fundamental frequency feature is adjusted based on the fundamental frequency adjustment feature, the overall up-or-down change of the source fundamental frequency feature can be more in line with the user's auditory perception.

[0084] In one embodiment, referring to Figure 3 , before step S103, it also includes:

[0085] Step S301, obtaining standard melody information corresponding to the source song audio;

[0086] Step S302, determining first melody information of the source audio based on the source fundamental frequency feature;

[0087] Step S303: determining feature points to be corrected in the source fundamental frequency feature based on the first melody information and the standard melody information, and correcting the feature points to be corrected to obtain corrected source fundamental frequency features.

[0088] In step S301, the standard melody information is the standard melody information of the song corresponding to the source song audio. It is understandable that the source song audio may be out of tune due to reasons such as the singer or the recording, and the melody of the source song audio itself may have some inaccuracies. At this time, the music score corresponding to the source song audio can be retrieved from a preset database, and the corresponding standard melody information can be determined based on the retrieved music score.

[0089] In step S302, the first melody information refers to the actual melody information of the source song audio itself. Specifically, the source fundamental frequency feature can reflect the pitch of each node of the source song audio itself in the time domain, and the first melody information of the source song audio itself can be determined based on the change of the source fundamental frequency feature in the time domain.

[0090] In step S303, after obtaining the standard melody information and the first melody information corresponding to the source song audio, the first melody information and the standard melody information can be compared to correct the first melody information. Specifically, the function images of the first melody information and the standard melody information with respect to time can be generated respectively, and then the curves of the two images themselves are compared to see whether they overlap. The non-overlapping positions are the positions of the feature points to be corrected in the source fundamental frequency features. These feature points to be corrected reflect the positions where the actual melody of the source song audio is inconsistent with the corresponding standard melody, that is, the positions where the source song audio is out of tune. Afterwards, the feature points to be corrected in the source fundamental frequency features are corrected to ensure that the source fundamental frequency features can accurately reflect the melody information of the source song audio. Specifically, the feature points to be corrected are corrected based on the difference between the standard melody information and the first melody information, so that the source fundamental frequency features can have melody information consistent with the standard melody information of the song corresponding to the source song audio.

[0091] In the embodiment disclosed in steps S301 to S303, by obtaining the standard melody information and the first melody information corresponding to the source song audio, the corresponding feature points to be corrected are determined based on the standard melody information and the first melody information, and the feature points to be corrected in the source fundamental frequency feature are corrected based on the difference between the standard melody information and the first melody information, so that the source fundamental frequency feature can accurately reflect the melody of the song corresponding to the source song audio. Based on this, the subsequent singing conversion according to the corrected source fundamental frequency feature can ensure that the melody of the converted target song audio is accurate, thereby improving the quality of the converted target song audio.

[0092] In another embodiment, corresponding standard melody information can be constructed based on the retrieved music score and the duration of the source song audio, and the corresponding fundamental frequency sequence can be generated as the corrected source fundamental frequency feature. In this way, the melody information of the source song audio can be directly customized to better meet the needs of the user. In the medical scenario, the user's nervousness can be better linked to help the user relax.

[0093] In one embodiment, referring to Figure 4 Step S104 includes but is not limited to:

[0094] Step S401, sampling from the source audio distribution features to obtain acoustic sampling features corresponding to the source song audio, and superimposing the acoustic sampling features and the source text features to obtain source audio superposition features;

[0095] Step S402, concatenating the source audio superposition feature and the adjusted fundamental frequency feature, and inputting them and the target voiceprint feature into a preset decoder for decoding to obtain a decoded audio feature;

[0096] Step S403, input the decoded audio features and the target voiceprint features into a preset vocoder for audio generation, and obtain the target song audio corresponding to the target voiceprint features.

[0097] In step S401, the source audio distribution feature is a probability distribution for characterizing the acoustic information of the source song audio, and sampling is performed from the probability distribution, that is, a random sample is extracted from the source audio distribution feature as the corresponding acoustic sampling feature, and the acoustic sampling feature can be recorded as z. Afterwards, the acoustic sampling feature and the source text feature are superimposed, and the source text feature is recorded as q, and the active audio superimposed feature is z+q.

[0098] In step S402, after the source audio superposition feature is obtained, the source audio superposition feature and the adjusted fundamental frequency feature are spliced. Specifically, the source audio superposition feature is z+q, and the adjusted fundamental frequency feature is F0′, then concat((z+q), F0′) is obtained after splicing, and then the splicing result is input into the decoder. At the same time, the target voiceprint feature is also input into the decoder as a conditional feature for decoding processing. The decoder can use a decoder in a conditional variational autoencoder structure, which can map the input latent variable into corresponding data in the corresponding latent space according to the given conditional feature. Thus, the splicing result of the source audio superposition feature and the adjusted fundamental frequency feature can be used as the latent variable of the input decoder, and the target voiceprint feature can be input into the decoder as an additional input conditional feature, thereby generating a decoded audio feature with timbre information of the target voiceprint feature, and this decoded audio feature can be a Mel cepstrum feature.

[0099] In step S403, after the decoded audio features are obtained, since the decoder only outputs a Mel cepstral feature corresponding to the audio, rather than directly outputting the audio, at this time, a vocoder is required to generate the audio from the decoded audio features. Specifically, the vocoder can use High Fidelity Generative Adversarial Networks (HiFiGAN), which includes a generator that can be used to generate a corresponding audio waveform from a Mel cepstral graph. The generator is composed of multiple convolutional layers and residual blocks. The corresponding features are extracted from the input through the convolutional layers and residual blocks, and up-sampled to improve the resolution, thereby generating high-quality target song audio. Specifically, the target voiceprint features and the decoded audio features can be used as the input of the vocoder together, so that the vocoder generates audio based on the decoded audio features and the target voiceprint features, thereby obtaining a target song audio with the timbre characteristics of the target voiceprint features, thereby converting the source song audio into a target song audio with the timbre of the target speaker.

[0100] In step S401 to step S403, sampling is performed in the source audio distribution features, and then the sampled acoustic sampling features and source text features are superimposed and spliced ​​with the adjusted fundamental frequency features as the input of the decoder. At the same time, the target voiceprint features are injected into the decoder as additional conditional features, so that the decoder maps the input latent variables into corresponding outputs according to the target voiceprint features to obtain corresponding decoded audio features. After that, the decoded audio features and the target voiceprint features are input into the vocoder together to obtain the target song audio with the timbre attributes of the target speaker, thereby completing the singing conversion.

[0101] In one embodiment, referring to Figure 5 Step S102 includes but is not limited to:

[0102] Step S501, converting the source song audio into corresponding linear spectrum features, and encoding the linear spectrum features through a spectrum encoder to obtain initial text features and source audio distribution features;

[0103] Step S502, quantizing the initial text features to obtain source text features;

[0104] Step S503: extracting source fundamental frequency features of the source song audio through a fundamental frequency extractor.

[0105] In step S501, the source song audio can be converted into corresponding linear spectral features by performing a sliding window Fourier transform on the source song audio. After that, the linear spectral features are directly input into the spectrum encoder for processing. The spectrum encoder adopts the encoder structure in the variational autoencoder, which can map the input linear spectral features to a high-dimensional latent space. This latent space can be represented by a Gaussian distribution. The spectrum encoder outputs the mean and variance of this Gaussian distribution to obtain the corresponding source audio distribution features; at the same time, the spectrum encoder generates a continuous latent vector based on the linear spectral features, that is, the initial text features.

[0106] In step S502, the initial text feature can be a continuous vector formed by splicing multiple initial text vectors, and the initial text feature includes multiple initial text vectors. The initial text feature is input into a trained quantizer for quantization processing. The quantizer can replace each initial text vector with a corresponding text feature code, thereby quantizing the initial text feature into a discrete text feature, thereby decoupling the text information and other acoustic information of the source song audio, avoiding the noise caused by the background acoustic information of the source song audio and causing the text information to be distorted, and improving the noise resistance and robustness in the singing voice conversion process. The training process of the quantizer will be described later, so it will not be repeated here.

[0107] In step S503, the source song audio is input into a baseband extractor, which may be a WORLD baseband extractor. In the baseband extractor, the source song audio may be subjected to baseband extraction processing based on a harvest algorithm or a DIO algorithm to obtain source baseband features.

[0108] In step S501 to step S503, the corresponding source fundamental frequency features are extracted from the source song audio by the fundamental frequency extractor. At the same time, the source song audio is converted into corresponding linear spectrum features, and then the linear spectrum features are input into the spectrum encoder, so that the spectrum encoder generates corresponding initial text features and source audio distribution features based on the linear spectrum features, and then the initial text features are quantized to obtain source text features, thereby decoupling the text information of the source song audio from other acoustic information.

[0109] In one embodiment, referring to Figure 6 Step S502 includes but is not limited to:

[0110] Step S601, inputting the initial text features into a preset quantizer, where the quantizer includes a plurality of preset text vectors;

[0111] Step S602, in the quantizer, for each initial text vector, determining a target text vector corresponding to the initial text vector from a plurality of preset text vectors, wherein the target text vector is a preset text vector having the smallest distance from the initial text vector among the plurality of preset text vectors;

[0112] Step S603: concatenate the target text vectors corresponding to the initial text vectors to obtain source text features.

[0113] In step S601, the preset quantizer is provided with a codebook, which includes a plurality of random discrete vectors with the same dimension, and these random discrete vectors are preset text vectors, wherein the preset text vectors in the codebook are determined in the process of training the quantizer. At the same time, after the initial text feature is input into the quantizer, the initial text feature is divided into a plurality of discrete sub-vectors to obtain a plurality of initial text vectors.

[0114] In step S602, in the quantizer, for each initial text vector, the Euclidean distance between it and each preset text vector in the codebook is calculated. The larger the Euclidean distance, the lower the similarity between the preset text vector and the initial text vector; conversely, the smaller the Euclidean distance, the higher the similarity between the preset text vector and the initial text vector. Based on this, for each initial text vector, the preset text vector with the smallest Euclidean distance from the initial text vector among multiple preset text vectors is used as the corresponding target text vector.

[0115] In step S603, after obtaining the target text vector corresponding to each initial text vector, these target text vectors are spliced ​​along the sequence dimension to obtain the source text feature. For example, there are m target text vectors, and each target text vector is a d-dimensional column vector. In this case, the sequence dimension is the column dimension. After splicing along the column dimension, a matrix of d rows and m columns is obtained, which is the source text feature.

[0116] In step S601 to step S603, the initial text features are divided into multiple initial text vectors, and in the quantizer, the corresponding target text vectors are determined from the quantizer codebook by calculating the Euclidean distance, and then the multiple target text vectors are spliced ​​along the sequence dimension to obtain a feature matrix as the source text feature.

[0117] In one embodiment, referring to Figure 7 , the singing voice conversion method also includes:

[0118] Step S701, obtaining a sample source audio, and extracting a first sample fundamental frequency feature from the sample source audio by a fundamental frequency extractor;

[0119] Step S702, extracting sample text features, sample audio distribution features, and sample voiceprint features from the sample source audio, and sampling from the sample audio distribution features to obtain sample acoustic sampling features, and superimposing the first sample fundamental frequency features and the sample acoustic sampling features to obtain sample superimposed audio features;

[0120] Step S703, concatenating the sample superimposed audio feature and the first sample feature to obtain a sample concatenated audio feature, and inputting the sample concatenated audio feature and the sample voiceprint feature into a preset decoder for decoding to obtain a sample decoded audio feature;

[0121] Step S704, inputting the sample decoding feature into a vocoder to generate a sample converted audio, and extracting a second sample fundamental frequency feature from the sample converted audio by a fundamental frequency extractor;

[0122] Step S705, calculating a mean square error based on the first sample fundamental frequency feature and the second sample fundamental frequency feature to obtain a first loss value, and optimizing the decoder based on the first loss value.

[0123] In step S701, the sample source audio is a training sample for training the decoder, and the sample source audio can be a song audio or a non-song speech audio. It can be understood that there are multiple sample source audios, and for each sample source audio, the process of step S701 to step S705 needs to be performed separately. After obtaining the sample source audio, the corresponding first sample fundamental frequency feature is first extracted by fundamental frequency extraction.

[0124] In step S702, after the sample source audio is converted into the corresponding sample linear spectrum feature, it is input into the spectrum encoder, the corresponding sample initial text feature is extracted by the spectrum encoder, and the sample initial text feature is quantized by the quantizer to obtain the sample text feature. At the same time, the sample linear spectrum feature is mapped to a latent space, which can be represented by a Gaussian distribution, denoted as N(μ,σ 2 ), thereby obtaining the sample audio distribution features. At the same time, the sample source audio is input into the trained speaker encoder to extract the corresponding sample voiceprint features. After that, sampling is performed from the sample audio distribution features to obtain the sample acoustic sampling features and superimposed with the sample text features to obtain the sample superimposed audio features.

[0125] In step S703, after obtaining the audio feature after sample superposition, the first sample fundamental frequency feature and the audio feature after sample superposition are spliced ​​into a sample spliced ​​audio feature, and the sample spliced ​​audio feature is input into the decoder. At the same time, the sample voiceprint feature is input into the decoder as a conditional feature of the decoder, so that the decoder maps the sample spliced ​​audio feature into corresponding data in the latent space based on the sample voiceprint feature, and obtains the corresponding sample decoded audio feature. It can be understood that the sample decoded audio feature is also a Mel-cephalogram feature.

[0126] In step S704, the sample decoded audio features and the sample voiceprint features are input into the vocoder, so that the vocoder generates speech based on the input sample decoded audio features and the sample voiceprint features to obtain the sample converted audio, and then the sample converted audio is input into the fundamental frequency extractor to extract the corresponding second sample fundamental frequency features.

[0127] In step S705, since the first sample fundamental frequency feature input to the decoder is obtained based on the sample source audio, when the decoder learns how to accurately decode according to the input fundamental frequency feature, the fundamental frequency of the sample converted audio generated based on the sample decoded audio feature should be consistent with the fundamental frequency of the input sample source audio. At this time, by calculating the mean square error between the first sample fundamental frequency feature and the second sample fundamental frequency feature, a first loss value is obtained. This first loss value represents the error between the first sample fundamental frequency feature and the second sample fundamental frequency feature, and can reflect the error when the decoder performs decoding processing based on the first sample fundamental frequency feature. Back propagation is performed based on the first loss value to optimize the parameters of the decoder, so that the decoder learns how to accurately perform decoding processing based on the input fundamental frequency feature.

[0128] Specifically, the first loss value can be expressed as:

[0129]

[0130] Among them, L F0 is the first loss value, E t Characterizes the mean square error function, F0 represents the fundamental frequency feature of the first sample, represents the second sample fundamental frequency feature, n represents the number of elements in F0, f(i) represents the i-th element in F0, y i express The i-th element in .

[0131] In addition, in the related art, when training the model for singing voice conversion, the conventional speech data set does not carry melody information, and directly using it to train the singing voice conversion model easily leads to the model being unable to learn how to keep the singing melody consistent before and after the conversion. The singing data suitable for training the singing voice conversion model is difficult to collect, and there is a lack of open source multi-speaker singing data suitable for singing voice conversion, which makes the training process of the singing voice conversion model face the problem of serious shortage of training samples. In the present disclosure, since in steps S701 to S705, the fundamental frequency features of the sample source audio are extracted separately for processing, and in the above steps S101 to S104 and other related steps, the corresponding mean is calculated based on the fundamental frequency feature as the fundamental frequency adjustment feature to adjust the pitch in the singing voice conversion process, and no explicit melody information is used in the process, based on this, in the present disclosure, the conventional speech data can be directly used to train the relevant spectrum encoder, decoder, quantizer and vocoder, thereby avoiding the problem of lack of data for training the singing voice conversion model in the related art.

[0132] In step S701 to step S705, the corresponding first sample fundamental frequency feature, sample text feature, sample audio distribution feature and sample voiceprint feature are extracted by using the sample source audio, and used as the input of the decoder to make the decoder decode, and the output of the decoder is processed by the vocoder to obtain the sample converted audio, and the corresponding second sample fundamental frequency feature is extracted from the sample converted audio and combined with the first sample fundamental frequency feature to perform mean square error calculation, so as to calculate the error between the first sample fundamental frequency feature and the second sample fundamental frequency feature, and thereby optimize the decoder so that the decoder learns how to perform decoding processing according to the given fundamental frequency data to generate the target audio with the pitch corresponding to the given fundamental frequency. At the same time, by extracting the fundamental frequency features and based on the fundamental frequency features, the decoder learns how to generate target audio with a pitch corresponding to a given fundamental frequency, so that no explicit melody information is required in the process of singing voice conversion. When training the model for singing voice conversion, conventional speech data can be directly used for training, and it can also be ensured that the trained model can generate target song audio consistent with the input song melody, without the need for training through multi-speaker singing data suitable for singing voice conversion, thereby avoiding the situation in related technologies where there are insufficient training samples suitable for singing voice conversion.

[0133] In one embodiment, referring to Figure 8 Step S702 also includes but is not limited to:

[0134] Step S801, extracting sample initial text features from the sample source audio, and quantizing the sample initial text features through a quantizer to obtain sample text features;

[0135] Step S802, obtaining a sample text corresponding to the sample source audio, and converting the sample text into a sample text embedding vector;

[0136] Step S803, performing a monotonic alignment search on the text embedding vector to obtain an aligned text vector aligned with the sample text feature;

[0137] Step S804, determining a first matching probability based on the aligned text vector and the sample text feature, determining a second matching probability based on a true matching relationship between the sample source audio and the sample text, and determining a second loss value based on the first matching probability and the second matching probability;

[0138] Step S805, calculating a mean square error based on the sample initial text features and the sample text features to obtain a third loss value;

[0139] Step S806: determine a quantization loss based on the second loss value and the third loss value, and optimize the quantizer based on the quantization loss until the quantization loss is less than a quantization loss threshold.

[0140] The details of step S801 may refer to the above step S502 and will not be described in detail here.

[0141] In step S802, the sample text is the text corresponding to the sample source audio. For example, the sample source audio is a recitation audio, and the sample text is the content of the recitation. After obtaining the sample text, the sample text is word-embedded through an embedding layer, so as to convert the sample text into a corresponding sample text embedding vector. Specifically, if there are multiple sample source audios, there are also multiple corresponding sample texts, and each sample text corresponds to one sample source audio.

[0142] In step S803, a monotonic alignment search is performed on the sample text embedding vector, so that the sample text embedding vector is aligned with the sample text feature length corresponding to the sample source audio, and an aligned text vector is obtained.

[0143] In step S804, the similarity between each aligned text vector and the sample text feature is first calculated, such as cosine similarity or Euclidean distance, so as to obtain the matching probability between each aligned text vector and each sample text feature, which is recorded as the first matching probability. It can be understood that the first matching probability is a probability distribution, and the first matching probability is recorded as p(x). At the same time, based on the true matching probability between each sample source audio and the sample text, the second matching probability between the sample text feature and the aligned text vector is determined. It can be understood that the second matching probability is also a probability distribution, and the second matching probability is recorded as q(x). In the second matching probability, the matching probability between each aligned text vector and the sample text feature of the corresponding source audio is 1, and the matching probability between the features of other sample texts is 0. For example, if the i-th sample text corresponds to the i-th sample source audio, then the matching probability between the aligned text vector of the i-th sample text and the sample text feature of the i-th sample source audio is 1. Afterwards, a cross entropy calculation is performed based on the first matching probability and the second matching probability as the second loss value. The second loss value represents the error between the sample text feature obtained after quantization and the real text information. Specifically, the second loss value can refer to the following formula:

[0144] L content =H(x align ,softmax(LP(q)));

[0145] Where L content represents the second loss value, H() represents the cross entropy function, x align represents the aligned text vector, and softmax(LP(q)) represents the sample text feature. By calculating the second loss value based on the sample text corresponding to the sample source audio, the quantizer is supervised and trained, so that the quantizer can capture the text information of the input audio more accurately, and at the same time, the goal of the quantizer in the quantization process is clearer, so that the quantizer learns how to capture the text information of the audio and ignore the acoustic information in the audio. This can improve the noise resistance of the quantization process on the one hand, and avoid retaining the original timbre of the input audio in the audio obtained after the singing voice conversion on the other hand.

[0146] The details of the mean square error calculation in step S805 can refer to the above step S705, which will not be repeated here. By calculating the mean square error, a third loss value is obtained, and the third loss value represents the error before and after quantization by the quantizer.

[0147] In step S806, the second loss value and the third loss value are added to obtain the total loss value of the quantizer, that is, the quantization loss. After that, back propagation is performed in the quantizer based on the quantization loss to optimize the parameters of the quantizer until the quantization loss is less than a given quantization loss threshold, thereby obtaining a trained quantizer.

[0148] In one embodiment, step S702 also includes but is not limited to:

[0149] A fourth loss value is determined based on the sample audio distribution characteristics and a standard Gaussian distribution, and the spectrum encoder is optimized based on the fourth loss value until the fourth loss value is less than a distribution loss threshold.

[0150] Specifically, during the training process, after the corresponding sample audio distribution feature is extracted from the sample source audio, the sample audio distribution feature is a Gaussian distribution. However, for different given audios, the Gaussian distribution generated by the spectrum encoder may be different. The sample audio distribution feature can be recorded as N(μ,σ 2 ), where μ is the mean, σ 2 is the variance, which is not conducive to the subsequent sampling and decoding process after sampling. In order to facilitate sampling from accurate source audio distribution characteristics during use, the spectrum encoder should be able to convert each input source audio into a unified distribution characteristic, that is, a standard Gaussian distribution. Based on this, in this embodiment, the fourth loss value is calculated based on the sample audio distribution characteristics and the standard Gaussian distribution. Specifically, the KL divergence is calculated based on the sample audio distribution characteristics and the standard Gaussian distribution as the fourth loss value. The formula of the fourth loss value can be expressed as:

[0151]

[0152] It is not difficult to see that when σ is close to 1 and μ is close to 0, the fourth loss value is the smallest, that is, the closer the distribution characteristics of the sample audio generated by the spectrum encoder based on the sample source audio are to the standard Gaussian distribution, the smaller the corresponding fourth loss value is, thereby enabling the spectrum encoder to learn how to map the given audio into a distribution characteristic that conforms to the standard Gaussian distribution.

[0153] Reference Fig. 9 The present disclosure also provides a singing voice conversion device 900, including:

[0154] A first acquisition unit 910 is used to acquire a target speech and extract a target voiceprint feature and a target fundamental frequency feature from the target speech;

[0155] The second acquisition unit 920 is used to acquire the source song audio, and determine the source text feature, source fundamental frequency feature and source audio distribution feature corresponding to the source song audio based on the source song audio;

[0156] A fundamental frequency adjustment unit 930 is configured to determine a fundamental frequency adjustment feature based on the target fundamental frequency feature and the source fundamental frequency feature, and adjust the source fundamental frequency feature based on the fundamental frequency adjustment feature to obtain an adjusted fundamental frequency feature, wherein the fundamental frequency adjustment feature is a difference between a mean value of the target fundamental frequency feature and a mean value of the source fundamental frequency feature;

[0157] The singing synthesis unit 940 is used to generate target song audio based on the adjusted fundamental frequency characteristics, target voiceprint characteristics, source text characteristics and source audio distribution characteristics.

[0158] The singing voice conversion device 900 of the embodiment of the present disclosure is used to execute the singing voice conversion method in the above embodiment. Its specific processing process is the same as the singing voice conversion method in the above embodiment, and will not be repeated here.

[0159] The present disclosure also provides an electronic device 1000, including:

[0160] at least one processor, and

[0161] a memory communicatively connected to at least one processor; wherein,

[0162] The memory stores instructions, and the instructions are executed by at least one processor so that when the at least one processor executes the instructions, a method as described in any one of the above embodiments of the present application is implemented.

[0163] Combine the following Fig.10 The hardware structure of the electronic device is described in detail. The electronic device includes: a processor 1010 , a memory 1020 , an input / output interface 1030 , a communication interface 1040 and a bus 1050 .

[0164] The processor 1010 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present disclosure;

[0165] The memory 1020 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 1020 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1020, and the processor 1010 is called to execute the singing voice conversion method of the embodiment of the present disclosure;

[0166] Input / output interface 1030, used to implement information input and output;

[0167] Communication interface 1040, used to realize communication interaction between the device and other devices, which can be realized by wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.); and

[0168] Bus 1050 , which transmits information between various components of the device (e.g., processor 1010 , memory 1020 , input / output interface 1030 , and communication interface 1040 );

[0169] The processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 are connected to each other in communication within the device via a bus 1050 .

[0170] An embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the above-mentioned singing voice conversion method.

[0171] The embodiments described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems.

[0172] It can be understood by those skilled in the art that Figures 1 to 10 The technical solutions shown in the figure do not constitute a limitation on the embodiments of the present disclosure, and may include more or fewer steps than those shown in the figure, or a combination of certain steps, or different steps.

[0173] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0174] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0175] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0176] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0177] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not run. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0178] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0179] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0180] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to run all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0181] The preferred embodiments of the present disclosure are described above with reference to the accompanying drawings, but the scope of the rights of the present disclosure is not limited thereto. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the present disclosure should be within the scope of the rights of the present disclosure.

Claims

1. A singing voice conversion method, characterized in that: The method comprises: Acquire a target voice, and extract a target voiceprint feature and a target fundamental frequency feature from the target voice; Acquire source song audio, and determine source text features, source fundamental frequency features, and source audio distribution features corresponding to the source song audio based on the source song audio; Determine a fundamental frequency adjustment feature based on the target fundamental frequency feature and the source fundamental frequency feature, and adjust the source fundamental frequency feature based on the fundamental frequency adjustment feature to obtain an adjusted fundamental frequency feature, wherein the fundamental frequency adjustment feature is a difference between a mean value of the target fundamental frequency feature and a mean value of the source fundamental frequency feature; The target song audio is generated based on the adjusted fundamental frequency feature, the target voiceprint feature, the source text feature and the source audio distribution feature.

2. The singing voice conversion method according to claim 1, characterized in that: The determining of the fundamental frequency adjustment feature based on the target fundamental frequency feature and the source fundamental frequency feature, and adjusting the source fundamental frequency feature based on the fundamental frequency adjustment feature to obtain an adjusted fundamental frequency feature includes: Performing logarithmic processing on the target fundamental frequency feature and the source fundamental frequency feature to obtain a logarithmic target fundamental frequency feature and a logarithmic source fundamental frequency feature; Determine a first feature mean corresponding to the logarithmic target fundamental frequency feature and a second feature mean corresponding to the logarithmic source fundamental frequency feature, and determine the fundamental frequency adjustment feature based on the first feature mean and the second feature mean; The logarithmic source fundamental frequency characteristic is adjusted based on the fundamental frequency adjustment characteristic, and the adjustment result is subjected to exponential processing to obtain the adjusted fundamental frequency characteristic.

3. The singing voice conversion method according to claim 1, characterized in that: Before determining the fundamental frequency adjustment feature based on the target fundamental frequency feature and the source fundamental frequency feature, the method further includes: Obtaining standard melody information corresponding to the source song audio; Determine first melody information of the source song audio based on the source fundamental frequency feature; The feature points to be corrected in the source fundamental frequency feature are determined based on the first melody information and the standard melody information, and the feature points to be corrected are corrected to obtain the corrected source fundamental frequency feature.

4. The singing voice conversion method according to claim 1, characterized in that: The generating target song audio based on the adjusted fundamental frequency feature, the target voiceprint feature, the source text feature and the source audio distribution feature comprises: Sampling from the source audio distribution feature to obtain an acoustic sampling feature corresponding to the source song audio, and superimposing the acoustic sampling feature and the source text feature to obtain a source audio superposition feature; The source audio superposition feature and the adjusted fundamental frequency feature are concatenated, and the concatenated source audio superposition feature and the adjusted fundamental frequency feature are input into a preset decoder for decoding together with the target voiceprint feature to obtain a decoded audio feature; The decoded audio features and the target voiceprint features are input into a preset vocoder for audio generation to obtain the target song audio corresponding to the target voiceprint features.

5. The singing voice conversion method according to claim 1, characterized in that: The determining, based on the source song audio, source text features, source fundamental frequency features, and source audio distribution features corresponding to the source song audio includes: Convert the source song audio into corresponding linear spectrum features, and encode the linear spectrum features through a spectrum encoder to obtain initial text features and the source audio distribution features; Quantifying the initial text features to obtain the source text features; The source fundamental frequency feature of the source song audio is extracted by a fundamental frequency extractor.

6. The singing voice conversion method according to claim 5, characterized in that: The initial text features include a plurality of initial text vectors, and the quantization process is performed on the initial text features to obtain the source text features, including: Inputting the initial text features into a preset quantizer, wherein the quantizer includes a plurality of preset text vectors; In the quantizer, for each of the initial text vectors, a target text vector corresponding to the initial text vector is determined from the multiple preset text vectors, wherein the target text vector is a preset text vector having the smallest distance from the initial text vector among the multiple preset text vectors; The target text vectors corresponding to the initial text vectors are concatenated to obtain the source text features.

7. The singing voice conversion method according to claim 1, characterized in that: The singing voice conversion method also includes: Acquire a sample source audio, and extract a first sample fundamental frequency feature from the sample source audio by the fundamental frequency extractor; Extracting sample text features, sample audio distribution features, and sample voiceprint features from the sample source audio, sampling from the sample audio distribution features to obtain sample acoustic sampling features, and superimposing the first sample fundamental frequency features and the sample acoustic sampling features to obtain sample superimposed audio features; splicing the sample superimposed audio feature and the first sample feature to obtain a sample spliced ​​audio feature, and inputting the sample spliced ​​audio feature and the sample voiceprint feature into a preset decoder for decoding to obtain a sample decoded audio feature; Inputting the sample decoded feature into a vocoder to generate sample converted audio, and extracting a second sample fundamental frequency feature from the sample converted audio by the fundamental frequency extractor; A mean square error is calculated based on the first sample fundamental frequency feature and the second sample fundamental frequency feature to obtain a first loss value, and the decoder is optimized based on the first loss value.

8. A singing voice conversion device, characterized in that: The device comprises: A first acquisition unit, configured to acquire a target voice and extract a target voiceprint feature and a target fundamental frequency feature from the target voice; A second acquisition unit is used to acquire a source song audio, and determine a source text feature, a source fundamental frequency feature, and a source audio distribution feature corresponding to the source song audio based on the source song audio; a fundamental frequency adjustment unit, configured to determine a fundamental frequency adjustment feature based on the target fundamental frequency feature and the source fundamental frequency feature, and adjust the source fundamental frequency feature based on the fundamental frequency adjustment feature to obtain an adjusted fundamental frequency feature, wherein the fundamental frequency adjustment feature is a difference between a mean value of the target fundamental frequency feature and a mean value of the source fundamental frequency feature; The singing synthesis unit is used to generate target song audio based on the adjusted fundamental frequency feature, the target voiceprint feature, the source text feature and the source audio distribution feature.

9. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the singing voice conversion method as described in any one of claims 1 to 7 is realized.

10. A storage medium, wherein the storage medium is a computer-readable storage medium, used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the singing voice conversion method as described in any one of claims 1 to 7.

Citation Information

Cited By

  • Music conversion method, related device, equipment and storage medium

    CN119811403A

  • Music conversion method and related device, equipment and storage medium

    CN119811403B

  • Audio processing method and device

    CN120510853A

  • Audio processing method and device

    CN120673729A

  • Voice generation method and electronic equipment

    CN121725763A