A voice timbre conversion method, device and computer readable storage medium

Abstract

The application relates to the field of voice processing, in particular to a voice timbre conversion method and device and a computer readable storage medium. The method comprises the following steps: acquiring a first frequency spectrum corresponding to a first audio. Extracting a first timbre characteristic parameter from a timbre library, wherein the timbre library comprises a plurality of preset timbre characteristic parameters. Determining a second frequency spectrum according to the first frequency spectrum and the first timbre characteristic parameter through a preset timbre conversion model. Determining a second audio of a first timbre corresponding to the first timbre characteristic parameter according to the second frequency spectrum. The method provided by the application can realize real-time conversion of voice timbre and improve the efficiency of voice timbre conversion.

Description

Technical Field

[0001] This invention relates to the field of speech processing, and more particularly to a speech timbre conversion method, apparatus, and computer-readable storage medium. Background Technology

[0002] With the rapid development of science and technology, voice timbre conversion technology has also developed rapidly. Voice timbre conversion refers to converting the timbre of the original speaker into the timbre of the target speaker while keeping the language content unchanged. Voice timbre conversion plays an important role in fields such as video voice changing, video dubbing, and human-computer interaction.

[0003] However, existing speech timbre conversion solutions require users to record a considerable amount of speech data and ensure the accuracy of the reading content before the speech synthesis model in the neural network speech synthesis system can be trained to obtain a customized timbre. This results in users spending a significant amount of time providing their speech data, leading to a poor user experience. Therefore, how to quickly and conveniently achieve real-time speech timbre conversion has become one of the urgent technical problems to be solved. Summary of the Invention

[0004] The technical problem to be solved by the embodiments of this application is that in the existing speech timbre conversion scheme, users need to provide a large amount of speech data and ensure the accuracy of the speech content, which results in users spending a lot of time and having a poor experience.

[0005] In a first aspect, embodiments of this application provide a speech timbre conversion method. The method includes: acquiring a first spectrum corresponding to a first audio frequency; extracting a first timbre feature parameter from a timbre library, wherein the timbre library includes multiple preset timbre feature parameters; determining a second spectrum based on the first spectrum and the first timbre feature parameter using a preset timbre conversion model; and determining a second audio frequency of the first timbre corresponding to the first timbre feature parameter based on the second spectrum.

[0006] In this embodiment, a first timbre feature parameter is directly extracted from a timbre library, and a preset timbre conversion model is used to convert the first audio into a second audio with the first timbre corresponding to the first timbre feature parameter. Therefore, the speech timbre conversion method provided in this embodiment directly extracts the target timbre feature parameter from the timbre library, eliminating the need for users to spend a lot of time providing accurate speech data to extract their timbre features, thereby achieving timbre conversion, saving time, improving speech timbre conversion efficiency, and enhancing user experience.

[0007] In conjunction with the first aspect, in one feasible implementation, the first spectrum includes N first sub-spectrums, and the second spectrum includes N-1 second sub-spectrums, where N is a positive integer greater than or equal to 2. The step of determining the second spectrum based on the first spectrum and the first timbre feature parameter using a preset timbre conversion model includes: performing the following spectrum synthesis operation on the N first sub-spectrums i and j: determining a first intermediate spectrum k based on the first sub-spectrum i and the first sub-spectrum j; determining a second sub-spectrum based on the first timbre feature parameter and the first sub-intermediate spectrum k using the preset timbre conversion model; and determining the N-1 second sub-spectrums based on the result of performing the spectrum synthesis operation on each of the N first sub-spectrums between two adjacent first sub-spectrums.

[0008] In conjunction with the first aspect, in one feasible implementation, determining the first intermediate spectrum k based on the first sub-spectrum i and the first sub-spectrum j includes: truncating the first sub-spectrum i according to a preset first truncation offset to obtain the first sub-intermediate spectrum k1; truncating the first sub-spectrum j according to the preset second truncation offset to obtain the first sub-intermediate spectrum k2; and merging the first sub-intermediate spectrum k1 and the first sub-intermediate spectrum k2 to obtain the first intermediate spectrum k.

[0009] In conjunction with the first aspect, in one feasible implementation, the step of truncating the first sub-spectrum i according to a preset first truncation offset to obtain a first sub-intermediate spectrum k1 includes: obtaining the termination time and the first center time of the first sub-spectrum i; determining a first intermediate time t1 based on the first center time and the preset first truncation offset, wherein the first intermediate time t1 is before the first center time and the difference between the first center time and the first intermediate time t1 is the first truncation offset; and determining the first sub-intermediate spectrum k1 in the first sub-spectrum i based on the first intermediate time t1 and the termination time.

[0010] In conjunction with the first aspect, in one feasible implementation, the step of truncating the first sub-spectrum j according to a preset second truncation offset to obtain a first sub-intermediate spectrum k2 includes: obtaining the start time and the second center time of the first sub-spectrum j; determining a first intermediate time t2 based on the second center time and the preset second truncation offset, wherein the first intermediate time t2 is after the second center time and the difference between the first intermediate time t2 and the second center time is the second truncation offset; and determining the first sub-intermediate spectrum k2 in the first sub-spectrum j based on the start time and the first intermediate time t2.

[0011] In conjunction with the first aspect, in one feasible implementation, determining the second audio of the first timbre corresponding to the first timbre feature parameter based on the second spectrum includes: performing the following audio segment restoration operation on any second sub-spectrum m among the N-1 second sub-spectrums: processing the second sub-spectrum m using a preset vocoder model to obtain a corresponding intermediate audio segment; truncating the intermediate audio segment according to a preset third truncation offset to obtain a target audio segment; and determining the second audio of the first timbre corresponding to the first timbre feature parameter based on the N-1 target audio segments obtained by performing the audio segment restoration operation on each of the N-1 second sub-spectrums.

[0012] In conjunction with the first aspect, in one feasible implementation, the step of truncating the intermediate audio segment according to a preset third truncation offset to obtain a target audio segment includes: acquiring the start time and end time of the intermediate audio segment; determining a first time and a second time according to the preset third truncation offset, wherein the difference between the first time and the start time is the third truncation offset, and the difference between the end time and the second time is the third truncation offset; and determining the target audio segment from the intermediate audio segment based on the first time and the second time.

[0013] In conjunction with the first aspect, in one feasible implementation, the method further includes: obtaining a third spectrum corresponding to a third audio frequency. A preset timbre feature model is used to obtain timbre feature parameters corresponding to the third audio frequency based on the third spectrum, and the timbre feature parameters corresponding to the third audio frequency are saved to the timbre library.

[0014] In conjunction with the first aspect, in one feasible implementation, obtaining the first spectrum from the first audio source includes: obtaining the first audio source and segmenting it into N source audio segments; obtaining N first sub-spectrums corresponding to the N source audio segments and determining the N first sub-spectrums as the first spectrum.

[0015] In conjunction with the first aspect, in one feasible implementation, obtaining the first audio includes: obtaining text information; and converting the text information to obtain the first audio.

[0016] Secondly, embodiments of this application provide a speech timbre conversion device. The device includes: an acquisition unit for acquiring a first spectrum corresponding to a first audio frequency; an extraction unit for extracting a first timbre feature parameter from a timbre library, wherein the timbre library includes multiple preset timbre feature parameters; a processing unit for determining a second spectrum based on the first spectrum and the first timbre feature parameter using a preset timbre conversion model; and a further processing unit for determining a second audio frequency of the first timbre corresponding to the first timbre feature parameter based on the second spectrum.

[0017] In conjunction with the second aspect, in one feasible implementation, the first spectrum includes N first sub-spectrums, and the second spectrum includes N-1 second sub-spectrums, where N is a positive integer greater than or equal to 2. A processing unit is configured to perform the following spectrum synthesis operation on any two adjacent first sub-spectrums i and j among the N first sub-spectrums: The processing unit is configured to determine a first intermediate spectrum k based on the first sub-spectrum i and the first sub-spectrum j. The processing unit is configured to determine a second sub-spectrum based on the first timbre feature parameter and the first intermediate spectrum k using a preset timbre conversion model. The processing unit is configured to determine the N-1 second sub-spectrums based on the result of performing the spectrum merging operation on each of the N first sub-spectrums.

[0018] In conjunction with the second aspect, in one feasible implementation, the device further includes: a truncating unit, configured to truncate the first sub-spectrum i according to a preset first truncating offset to obtain a first sub-intermediate spectrum k1; a truncating unit, configured to truncate the first sub-spectrum j according to a preset second truncating offset to obtain a first sub-intermediate spectrum k2; and a merging unit, configured to merge the first sub-intermediate spectrum k1 and the first sub-intermediate spectrum k2 to obtain a first intermediate spectrum k.

[0019] In conjunction with the second aspect, in one feasible implementation, an acquisition unit is used to acquire the termination time and the first center time of the first sub-spectrum i. A processing unit is used to determine a first intermediate time t1 based on the first center time and the preset first truncation offset, wherein the first intermediate time t1 is before the first center time and the difference between the first center time and the first intermediate time t1 is the first truncation offset. A processing unit is also used to determine the first sub-intermediate spectrum k1 in the first sub-spectrum i based on the first intermediate time t1 and the termination time.

[0020] In conjunction with the second aspect, in one feasible implementation, an acquisition unit is used to acquire the start time and the second center time of the first sub-spectrum j. A processing unit is used to determine a first intermediate time t2 based on the second center time and the preset second truncation offset, wherein the first intermediate time t2 is after the second center time and the difference between the first intermediate time t2 and the second center time is the second truncation offset. A processing unit is also used to determine the first sub-intermediate spectrum k2 in the first sub-spectrum j based on the start time and the first intermediate time t2.

[0021] In conjunction with the second aspect, in one feasible implementation, a processing unit is configured to perform the following audio segment restoration operation on any second sub-spectrum m among the N-1 second sub-spectrums. Specifically, the processing unit processes the second sub-spectrum m according to a preset vocoder model to obtain a corresponding intermediate audio segment. A truncation unit truncates the intermediate audio segment according to a preset third truncation offset to obtain a target audio segment. Finally, a processing unit determines the second audio segment corresponding to the first timbre feature parameter based on the N-1 target audio segments obtained by performing the audio segment restoration operation on each of the N-1 second sub-spectrums.

[0022] In conjunction with the second aspect, in one feasible implementation, an acquisition unit is used to acquire the start time and end time of the intermediate audio segment. A processing unit is used to determine a first time and a second time based on the preset third truncation offset, wherein the difference between the first time and the start time is the third truncation offset, and the difference between the end time and the second time is the third truncation offset. The processing unit is used to determine the target audio segment within the intermediate audio segment based on the first time and the second time.

[0023] In conjunction with the second aspect, in one feasible implementation, an acquisition unit is used to acquire the third spectrum corresponding to the third audio. A processing unit is used to acquire the timbre feature parameters corresponding to the third audio based on the third spectrum using a preset timbre feature model, and save the timbre feature parameters corresponding to the third audio to the timbre library.

[0024] In conjunction with the second aspect, in one feasible implementation, a processing unit is configured to acquire a first audio signal and segment the first audio signal into N source audio segments. The processing unit is further configured to acquire N first sub-spectrums corresponding to the N source audio segments and determine the N first sub-spectrums as the first spectrum.

[0025] In conjunction with the second aspect, in one feasible implementation, an acquisition unit is used to acquire text information, and a processing unit is used to convert the text information into the first audio.

[0026] Thirdly, embodiments of this application provide a computer-readable storage medium for storing a computer program. When the computer program is run on a computer, it enables the computer to execute the speech timbre conversion method provided by any possible implementation of the first aspect, thereby achieving the beneficial effects of the speech timbre conversion method provided in the first aspect.

[0027] Fourthly, embodiments of this application provide an electronic device that may include a processor and a memory, which are interconnected. The memory stores a computer program, and the processor is configured to execute the computer program to implement the speech timbre conversion method provided in the first aspect, thereby achieving the beneficial effects of the speech timbre conversion method provided in the first aspect.

[0028] By implementing the embodiments of the present invention, target timbre feature vectors can be extracted from the timbre library, and standard recordings can be converted into recordings of the target timbre using a timbre conversion model, thereby achieving real-time conversion of speech timbre, saving operation time, improving the efficiency of speech timbre conversion, and enhancing user experience. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the background art, the accompanying drawings used in the embodiments of the present invention or the background art will be described below.

[0030] Figure 1 This is a flowchart illustrating a speech timbre conversion method provided in an embodiment of this application;

[0031] Figure 2 This is a schematic diagram of a spectrum truncation provided in an embodiment of this application;

[0032] Figure 3 This is a schematic diagram of another spectrum truncation provided in the embodiments of this application;

[0033] Figure 4 This is a schematic diagram of an audio segment extraction method provided in an embodiment of this application;

[0034] Figure 5 This is another flowchart illustrating a speech timbre conversion method provided in an embodiment of this application;

[0035] Figure 6 This is a schematic diagram of the voice timbre conversion device provided in the embodiments of this application.

[0036] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical methods in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application. The embodiments of this application will be described below with reference to the accompanying drawings.

[0038] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this invention are used to distinguish similar objects, not to describe a specific order or sequence. It should be understood that such data or objects can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0039] It should be noted that the "spectrum" in this application refers to a three-dimensional spectrum, that is, a spectrum represented by three-dimensional coordinates of frequency, time, and amplitude.

[0040] Existing speech timbre conversion technologies require users to record a considerable amount of speech data and ensure the accuracy of the reading content before a speech synthesis model in a neural network speech system can be trained to obtain a customized timbre. However, this results in users spending a significant amount of time providing their speech data, leading to a poor user experience. Therefore, the technical problem this application aims to solve is: how to quickly and conveniently achieve real-time speech timbre conversion.

[0041] Example 1:

[0042] Please participate Figure 1 , Figure 1This is a flowchart illustrating a speech timbre conversion method provided in an embodiment of this application. This speech timbre conversion method can be specifically executed by a terminal device. In this embodiment, the terminal device can be any type of electronic device capable of speech / audio processing, such as a smartphone, laptop, desktop computer, self-service terminal, etc. This application does not impose specific limitations on the implementation form of the terminal device. Figure 1 As shown, the speech timbre conversion method may specifically include:

[0043] S101, Obtain the first spectrum corresponding to the first audio.

[0044] In some feasible implementations, after acquiring the first audio, the terminal device can process the first audio using methods such as Fourier transform and logarithmic operation to obtain the first spectrum corresponding to the first audio.

[0045] In one optional implementation, the terminal device can acquire text information input by the user, and then synthesize the first audio using a speech synthesis system based on the text information. It should be noted that the speech synthesis system can be an internal or external system capable of speech synthesis, and can take the form of software, hardware, or online services, or other systems with equivalent speech synthesis capabilities.

[0046] Optionally, the text information input by the user can be Chinese characters encoded in UTF-8 (8-bit, Universal Character Set / Unicode Transformation Format), Unicode, or other encoding languages. It should be noted that this application does not impose specific restrictions on the language type of the input text information; the language type can be Chinese, English, French, etc.

[0047] In an optional real-time mode, when the first audio is a continuous audio data segment, the terminal device can directly process the first audio using methods such as Fourier transform and logarithmic operation after acquiring the first audio to obtain the first audio.

[0048] In one optional implementation, when the first audio is a continuous audio data segment, after acquiring the first audio, the terminal device can segment the first audio to obtain N source audio segments. It should be noted that the terminal device can segment the first audio uniformly or non-uniformly to obtain N source audio segments; this application does not impose specific limitations on the segmentation method. Furthermore, the terminal device can use methods such as Fourier transform and logarithmic operations to process each of the N source audio segments to obtain N first sub-spectrums corresponding to these N source audio segments, and determine these N first sub-spectrums as the first spectrum. It should be noted that the N source audio segments obtained from the segmentation of the first audio are arranged in chronological order, and correspondingly, the N first sub-spectrums obtained from processing the N source audio segments are also arranged in the corresponding chronological order.

[0049] In a specific implementation, the terminal device can segment the first audio audio into N source audio segments based on a preset window length and frame shift. It should be noted that the window length represents the length of each source audio segment, and the frame shift represents the time difference between adjacent source audio segments. For example, assuming a window length of 25ms and a frame shift of 15ms, the first source audio segment in the N source audio segments would be 0–25ms, the second source audio segment would be 15–40ms, and so on. It should be understood that the values ​​of the window length and frame shift used for segmentation can be set according to actual needs, and this embodiment does not impose specific limitations on this. In another optional implementation, the first audio audio obtained by the terminal device is the N source audio segments that have already been segmented. Further, the terminal device can use methods such as Fourier transform and logarithmic operations to obtain the N first sub-spectrums corresponding to the N source audio segments, and determine these N first sub-spectrums as the first spectrum.

[0050] S102, extract the first timbre feature parameter from the timbre library.

[0051] In some optional implementations, the terminal device extracts a first timbre feature parameter from a timbre library. It should be noted that the timbre library includes multiple preset timbre feature parameters. For example, assuming the designated user is Xiaoming, the terminal device extracts Xiaoming's timbre feature parameter from the timbre library and uses it as the first timbre feature parameter.

[0052] In one optional implementation, the first timbre feature parameter can be specified by the user in real time, and the terminal device extracts the timbre feature of the specified timbre from the timbre library as the first timbre feature parameter. Alternatively, the first timbre feature parameter can be preset by the terminal device.

[0053] S103 determines the second spectrum based on the first spectrum and the first timbre feature parameters using a preset timbre conversion model.

[0054] In some feasible implementations, after extracting the first timbre feature parameter, the terminal device can determine the second spectrum based on the first spectrum and the first timbre feature parameter using a preset timbre conversion model. It should be noted that the preset timbre conversion model can be a convergent machine learning model trained using a large amount of speech sample data.

[0055] In one feasible implementation, after acquiring the first spectrum, the terminal device can input the first spectrum and the first timbre feature parameter into a preset timbre conversion model to obtain the second spectrum output by the preset timbre conversion model. It should be noted that the first spectrum is obtained by directly processing the first audio signal using methods such as Fourier transform and logarithmic operations.

[0056] In one feasible implementation, the first spectrum includes N first sub-spectrums, and the second spectrum includes N-1 second sub-spectrums. After acquiring the N first sub-spectrums, the terminal device can perform a spectrum synthesis operation on each pair of adjacent first sub-spectrums to obtain the N-1 second sub-spectrums. The spectrum synthesis operation is illustrated below using any two adjacent first sub-spectrums i and j from the N first sub-spectrums as an example. Specifically, the terminal device can first determine a first intermediate spectrum k based on the first sub-spectrum i and the first sub-spectrum j. Further, the terminal device can input the first intermediate spectrum k and the first timbre feature parameter into a preset timbre conversion model and obtain a second sub-spectrum output by the preset timbre conversion model. This second sub-spectrum is the second sub-spectrum synthesized from the first sub-spectrum i and the first sub-spectrum j. By performing the spectrum synthesis operation on each pair of adjacent first sub-spectrums from the N first sub-spectrums, the terminal device can determine the N-1 second sub-spectrums.

[0057] In one optional implementation, the terminal device truncates the first sub-spectrum i according to a preset first truncation offset to obtain a first sub-intermediate spectrum k1, and truncates the first sub-spectrum j according to the preset first truncation offset to obtain a first sub-intermediate spectrum k2. Further, the terminal device can perform spectrum merging on the first sub-intermediate spectrum k1 and the first sub-intermediate spectrum k2 to obtain a first intermediate spectrum k.

[0058] Optionally, the terminal device can obtain the end time and the first center time of the first sub-spectrum i, and then determine the first intermediate time t1 based on the first center time and the preset first truncation offset. It should be noted that the first intermediate time t1 is before the first center time, and the difference between the first center time and the first intermediate time t1 is the first truncation offset. Further, the terminal device can determine the first sub-intermediate spectrum k1 in the first sub-spectrum i based on the first intermediate time t1 and the end time. Specifically, the terminal device can determine the portion of the spectrum in the first sub-spectrum i located within the time interval between the first intermediate time t1 and the end time as the first sub-intermediate spectrum k1. For example, please refer to [link to relevant documentation]. Figure 2 , Figure 2 This is a schematic diagram of a spectrum truncation provided in an embodiment of this application, as shown below. Figure 2 As shown, it is assumed that the termination time of the first sub-spectrum i is 400ms, the first center time is 300ms, and the preset first truncation offset is 50ms. Then the terminal device can determine that the first intermediate time t1 is 250ms. Therefore, the terminal device can determine the part of the spectrum corresponding to the first sub-spectrum i in the time interval of 250ms to 400ms as the first sub-intermediate spectrum k1.

[0059] Optionally, the terminal device can obtain the start time and the second center time of the first sub-spectrum j, and then determine the first intermediate time t2 based on the second center time and the preset second truncation offset. It should be noted that the first intermediate time t2 is after the second center time, and the difference between the first intermediate time t2 and the second center time is the second truncation offset. Further, the first sub-intermediate spectrum k2 is determined from the first sub-spectrum j based on the start time and the first intermediate time t2. Specifically, the terminal device can determine the portion of the spectrum within the time interval between the start time and the first intermediate time t2 contained in the first sub-spectrum j as the first sub-intermediate spectrum k2. For example, please refer to [link to relevant documentation]. Figure 3 , Figure 3 This is a schematic diagram of another spectrum truncation provided in the embodiments of this application, such as... Figure 3 As shown, it is assumed that the starting time of the first sub-spectrum j is 400ms, the second center time is 500ms, and the preset second truncation offset is 50ms. Then the terminal device can determine that the first intermediate time t2 is 550ms. Therefore, the terminal device can determine the part of the spectrum corresponding to the first sub-spectrum j in the time interval of 400ms to 550ms as the first sub-intermediate spectrum k2.

[0060] In an optional implementation, after the terminal device acquires the last two segments of the first sub-spectrum in the first spectrum, the terminal device only performs the spectrum truncation operation performed on the first sub-spectrum i on the second-to-last segment of the first sub-spectrum, and does not perform the spectrum truncation operation performed on the first sub-spectrum j on the last segment of the first sub-spectrum, in order to prevent audio loss. S104, the second audio of the first timbre corresponding to the first timbre feature parameter is determined according to the second spectrum.

[0061] In some feasible implementations, the terminal device can determine the second audio of the first timbre corresponding to the first timbre feature parameter based on the second spectrum.

[0062] In one optional implementation, after obtaining the second spectrum, the terminal device can directly input the second spectrum into a preset vocoder model and obtain the second audio of the first timbre corresponding to the first timbre feature parameter output by the preset vocoder model.

[0063] In one optional implementation, the second spectrum includes N-1 second sub-spectrums. After acquiring the N-1 second sub-spectrums, the terminal device can perform audio segment restoration operations on each of these N-1 second sub-spectrums to obtain N-1 target audio segments. Further, the terminal device can determine the second audio of the first timbre based on the N-1 target audio segments. The restoration operation of the audio segment is illustrated below using any second sub-spectrum m from the N-1 second sub-spectrums as an example. Specifically, the terminal device can process any second sub-spectrum m from the N-1 second sub-spectrums using a preset vocoder model to obtain a corresponding intermediate audio segment. Then, the terminal device can perform audio truncation on the intermediate audio segment according to a preset third truncation offset to obtain a target audio segment. By performing the restoration operation on all N-1 second sub-spectrums, the terminal device can obtain N-1 target audio segments.

[0064] Furthermore, the terminal device can merge and splice the aforementioned N-1 target audio segments in chronological order to obtain a complete audio segment, and then identify this complete audio segment as the second audio of the first timbre corresponding to the first timbre feature parameter. Alternatively, after obtaining the aforementioned N-1 target audio segments, the terminal device can directly identify the aforementioned N-1 target audio segments as the second audio of the first timbre corresponding to the first timbre feature parameter. It should be noted here that the aforementioned N-1 target audio segments are arranged in chronological order.

[0065] In one optional implementation, the terminal device can obtain the start and end times of the aforementioned intermediate audio segment, and then determine the first and second times based on a preset third truncation offset. It should be noted that the difference between the first time and the start time is the third truncation offset, and the difference between the end time and the second time is also the third truncation offset. Then, the terminal device can determine a portion of the audio segment within the time interval between the first and second times as the target audio segment. For example, please refer to... Figure 4 , Figure 4 This is a schematic diagram of an audio segment extraction method provided in an embodiment of this application, such as... Figure 4 As shown, assuming the start time of the intermediate audio segment is 250ms and the end time is 550ms, and the preset third segment offset is 50ms, the terminal device can determine the first time as 300ms and the second time as 500ms. Then the terminal device can determine the corresponding part of the audio segment within the time interval of 300ms to 500ms as the target audio segment.

[0066] In one optional implementation, after the terminal device obtains an intermediate audio segment corresponding to the start time of the second audio output by the vocoder model, it fills a blank audio segment of the same length as the target audio segment before the intermediate audio segment corresponding to the start time to prevent calculation overflow.

[0067] In one optional implementation, after the terminal device obtains an intermediate audio segment corresponding to the termination time of the second audio output by the vocoder model, it does not perform the audio truncation operation on the intermediate audio segment to prevent audio loss.

[0068] In one alternative implementation, after acquiring the second audio, the terminal device can play the second audio through an output device such as a speaker.

[0069] In a specific timbre customization scenario, the terminal device can acquire the first audio recorded by the user through an input device such as a microphone. Then, the terminal device can process the first audio using methods such as Fourier transform to obtain the first spectrum corresponding to the first audio. Next, the terminal device can extract the first timbre feature parameter corresponding to the first timbre specified by the user from the aforementioned timbre library. Further, the terminal device can determine a second spectrum based on the first spectrum and the first timbre feature parameter using a preset speech timbre conversion model, then determine the second audio corresponding to the first timbre feature parameter based on the second spectrum, and play the second audio to the user through an output device such as a speaker, thereby achieving real-time speech timbre conversion.

[0070] In this embodiment, the terminal device extracts target timbre feature parameters from a timbre library, and then converts the source audio into audio with the target timbre using a preset timbre conversion model. This method of directly extracting target timbre feature parameters from the timbre library to achieve real-time speech timbre conversion saves time in obtaining target timbre feature parameters, thereby improving the efficiency of speech conversion and enhancing the user experience.

[0071] Please see Figure 5 , Figure 5 This is another schematic flowchart of a speech timbre conversion method provided in this application embodiment. It should be understood that in this application embodiment, steps S115 and S116 can be executed before or after step S104; this application embodiment does not impose specific restrictions on the execution order of the above two steps. Figure 5 As shown, the speech timbre conversion method also includes the following steps:

[0072] S115, obtain the third spectrum corresponding to the third audio.

[0073] In some feasible implementations, the terminal device can acquire a complete audio segment and use it as a third audio signal. Furthermore, it can process the audio signal using methods such as Fourier transform and logarithmic operations to obtain the corresponding third spectrum. In this application embodiment, no specific restrictions are placed on the content, language, etc., of the complete audio segment, nor on the file format, sampling rate, bit depth, volume, etc., of the complete audio segment acquired by the terminal device.

[0074] In one optional implementation, the third spectrum obtained by the terminal device from the third audio processing can be a linear spectrum, an amplitude spectrum, a Mel spectrum, or a Mel cepstral spectrum. For example, if the terminal device performs a Fourier transform on the first audio, the resulting third spectrum is a linear spectrum. Alternatively, the terminal device can first perform a Fourier transform on the first audio to convert the time-domain signal into a frequency-domain signal, and then process the frequency-domain signal using a Mel frequency-scaled filter bank; the resulting third spectrum is the Mel spectrum.

[0075] S116 uses a preset timbre feature model to obtain the timbre feature parameters corresponding to the third audio frequency based on the third spectrum, and saves the timbre feature parameters corresponding to the third audio frequency to the timbre library.

[0076] In some feasible implementations, the terminal device inputs the aforementioned third spectrum into a preset timbre feature model and obtains the third timbre feature parameter corresponding to the aforementioned third audio output by the preset timbre feature model. Furthermore, the terminal device can save the third timbre feature parameter corresponding to the aforementioned third audio into a timbre library. It should be noted that the aforementioned preset timbre feature model is a machine learning model obtained by training a large amount of sample data. For example, assuming the third audio is a recording of Xiaoming, then the third spectrum is a spectrum corresponding to a recording of Xiaoming. The terminal device inputs this third spectrum into the preset timbre feature model and can obtain the third timbre feature vector corresponding to the third audio output by the preset timbre feature model, i.e., Xiaoming's timbre feature. Furthermore, the terminal device saves the third timbre feature parameter corresponding to the third audio into a timbre library.

[0077] In the above implementation, the terminal device uses a preset timbre feature model to extract the timbre feature parameters corresponding to the audio provided by the user and saves them to the timbre library. The user provides an audio clip, without specific restrictions on the audio content, speech, accent, etc., and the user's timbre features can be obtained through the model. This model-based reasoning method not only saves operation time but also improves the user experience.

[0078] Please participate Figure 6 , Figure 6 This is a schematic diagram of the speech timbre conversion device provided in an embodiment of this application. Figure 6 The voice timbre conversion device may include: an acquisition unit 61, an extraction unit 62, a processing unit 63, a truncating unit 64, and a merging unit 65.

[0079] In a specific implementation, the acquisition unit 61 is used to acquire the first spectrum corresponding to the first audio. The extraction unit 62 is used to extract the first timbre feature parameter from the timbre library, wherein the timbre library includes multiple preset timbre feature parameters. The processing unit 63 is used to determine the second spectrum based on the first spectrum and the first timbre feature parameter using a preset timbre conversion model. The processing unit 63 is also used to determine the second audio of the first timbre corresponding to the first timbre feature parameter based on the second spectrum.

[0080] In one optional implementation, the first spectrum includes N first sub-spectrums, and the second spectrum includes N-1 second sub-spectrums, where N is a positive integer greater than or equal to 2. Processing unit 63 is configured to perform the following spectrum synthesis operation on any two adjacent first sub-spectrums i and j among the N first sub-spectrums: Processing unit 63 determines a first intermediate spectrum k based on the first sub-spectrum i and the first sub-spectrum j. Processing unit 63 determines a second sub-spectrum based on the first timbre feature parameter and the first intermediate spectrum k using a preset timbre conversion model. Processing unit 63 determines the N-1 second sub-spectrums based on the result of performing the spectrum merging operation on each of the N first sub-spectrums.

[0081] In an optional embodiment, the device further includes: a truncating unit 64, configured to truncate the first sub-spectrum i according to a preset first truncating offset to obtain a first sub-intermediate spectrum k1; a truncating unit 64, configured to truncate the first sub-spectrum j according to a preset second truncating offset to obtain a first sub-intermediate spectrum k2; and a merging unit 65, configured to merge the first sub-intermediate spectrum k1 and the first sub-intermediate spectrum k2 to obtain a first intermediate spectrum k.

[0082] In one optional implementation, the acquisition unit 61 is configured to acquire the termination time and the first center time of the first sub-spectrum i. The processing unit 63 is configured to determine a first intermediate time t1 based on the first center time and the preset first truncation offset, wherein the first intermediate time t1 is before the first center time and the difference between the first center time and the first intermediate time t1 is the first truncation offset. The processing unit 63 is configured to determine the first sub-intermediate spectrum k1 in the first sub-spectrum i based on the first intermediate time t1 and the termination time.

[0083] In one optional implementation, the acquisition unit 61 is configured to acquire the start time and the second center time of the first sub-spectrum j. The processing unit 63 is configured to determine a first intermediate time t2 based on the second center time and the preset second truncation offset, wherein the first intermediate time t2 is after the second center time and the difference between the first intermediate time t2 and the second center time is the second truncation offset. The processing unit 63 is configured to determine the first sub-intermediate spectrum k2 in the first sub-spectrum j based on the start time and the first intermediate time t2.

[0084] In one optional implementation, processing unit 63 is configured to perform the following audio segment restoration operation on any second sub-spectrum m among the N-1 second sub-spectrums: Processing unit 63 processes the second sub-spectrum m according to a preset vocoder model to obtain a corresponding intermediate audio segment. Truncation unit 64 truncates the intermediate audio segment according to a preset third truncation offset to obtain a target audio segment. Processing unit 63 then determines the second audio segment corresponding to the first timbre of the first timbre feature parameter based on the N-1 target audio segments obtained by performing the audio segment restoration operation on each of the N-1 second sub-spectrums.

[0085] In one optional implementation, the acquisition unit 61 is configured to acquire the start time and end time of the intermediate audio segment. The processing unit 63 is configured to determine a first time and a second time based on the preset third truncation offset, wherein the difference between the first time and the start time is the third truncation offset, and the difference between the end time and the second time is the third truncation offset. The processing unit 63 is configured to determine the target audio segment in the intermediate audio segment based on the first time and the second time.

[0086] In one optional implementation, the acquisition unit 61 is used to acquire the third spectrum corresponding to the third audio. The processing unit 63 is used to acquire the timbre feature parameters corresponding to the third audio based on the third spectrum using a preset timbre feature model, and save the timbre feature parameters corresponding to the third audio to the timbre library.

[0087] In one optional implementation, processing unit 63 is configured to acquire a first audio signal and segment the first audio signal into N source audio segments. Processing unit 63 is also configured to acquire N first sub-spectrums corresponding to the N source audio segments and determine the N first sub-spectrums as the first spectrum.

[0088] In one optional implementation, the acquisition unit 61 is used to acquire text information. The processing unit 63 is used to convert the text information into the first audio.

[0089] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device can be the terminal device in Embodiment 1, and can be used to implement the steps of the voice timbre conversion method performed by the terminal device as described in Embodiment 1. The electronic device may include: a processor 71, a memory 72, and a bus system 73.

[0090] The memory 72 includes, but is not limited to, RAM, ROM, EPROM, or CD-ROM, and is used to store related instructions and data. The memory 72 stores executable modules or data structures, or subsets thereof, or extended sets thereof:

[0091] Operation instructions: This includes various operation instructions used to perform various operations.

[0092] Operating system: includes various system programs used to implement various basic business functions and handle hardware-based tasks.

[0093] Figure 7 Only one memory is shown in the image; of course, multiple memory can be configured as needed.

[0094] like Figure 7 As shown, the electronic device may further include an input / output device 74, which may be a communication module or a transceiver circuit. In this embodiment, the input / output device 74 is used to perform the transmission and reception process of data or signaling such as apartment type identification parameters involved in Embodiment 1.

[0095] Processor 71 may be a controller, CPU, general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in connection with the embodiments of this application. Processor 71 may also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of DSP and microprocessor, etc.

[0096] In practical applications, the various components of an electronic device are coupled together through a bus system 73. This bus system 73 includes not only a data bus but may also include a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 7 The various buses are all labeled as Bus System 73. For ease of representation, in... Figure 7 The image shown is only schematic.

[0097] It should be noted that in practical applications, the processor in the embodiments of this application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0098] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memory.

[0099] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer, implements the method or steps executed by the terminal device in the first embodiment described above.

[0100] This application also provides a computer program product that, when executed by a computer, implements the method or steps executed by the terminal device in the first embodiment described above.

[0101] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this application can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of a computer program from one place to another. Storage media can be any available medium accessible to a general-purpose or special-purpose computer.

[0102] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of this application should be included within the scope of protection of this application.

Claims

1. A method for converting voice timbre, characterized in that, include: Obtain the first spectrum corresponding to the first audio; Extract the first timbre feature parameter from the timbre library, wherein the timbre library includes multiple preset timbre feature parameters; The second spectrum is determined based on the first spectrum and the first timbre feature parameters using a preset timbre conversion model. The second audio frequency of the first timbre corresponding to the first timbre feature parameter is determined based on the second spectrum; Wherein, the first spectrum includes N first sub-spectrums, and the second spectrum includes N-1 second sub-spectrums, where N is a positive integer greater than or equal to 2; The step of determining the second spectrum based on the first spectrum and the first timbre feature parameters using a preset timbre conversion model includes: For any two adjacent first sub-spectrums i and j among the N first sub-spectrums, the following spectrum synthesis operation is performed: The first intermediate spectrum k is determined based on the first sub-spectrum i and the first sub-spectrum j; A second sub-spectrum is determined by using a preset timbre conversion model based on the first timbre feature parameter and the first intermediate spectrum k. The N-1 second sub-spectrums are determined based on the result of performing the spectrum synthesis operation on each of the N first sub-spectrums between two adjacent first sub-spectrums.

2. The method according to claim 1, characterized in that, The step of determining the first intermediate spectrum k based on the first sub-spectrum i and the first sub-spectrum j includes: The first sub-spectrum i is truncated according to a preset first truncation offset to obtain the first sub-intermediate spectrum k1; The first sub-spectrum j is truncated according to the preset second truncation offset to obtain the first sub-intermediate spectrum k2; The first sub-intermediate spectrum k1 and the first sub-intermediate spectrum k2 are combined to obtain the first intermediate spectrum k.

3. The method according to claim 2, characterized in that, The step of truncating the first sub-spectrum i according to a preset first truncation offset to obtain the first sub-intermediate spectrum k1 includes: Obtain the termination time and the first center time of the first sub-spectrum i; The first intermediate time t1 is determined based on the first center time and the preset first truncation offset, wherein the first intermediate time t1 is before the first center time and the difference between the first center time and the first intermediate time t1 is the first truncation offset. The first sub-intermediate spectrum k1 is extracted from the first sub-spectrum i based on the first intermediate time t1 and the termination time.

4. The method according to claim 2 or 3, characterized in that, The step of truncating the first sub-spectrum j according to the preset second truncation offset to obtain the first sub-intermediate spectrum k2 includes: Obtain the start time and the second center time of the first sub-spectrum j; The first intermediate time t2 is determined based on the second center time and the preset second truncation offset, wherein the first intermediate time t2 is after the second center time and the difference between the first intermediate time t2 and the second center time is the second truncation offset. Based on the start time and the first intermediate time t2, the first sub-intermediate spectrum k2 is extracted from the first sub-spectrum j.

5. The method according to claim 1 or 2, characterized in that, The step of determining the second audio frequency corresponding to the first timbre feature parameter based on the second spectrum includes: Perform the following audio segment restoration operation on any second sub-spectrum m among the N-1 second sub-spectrums: The second sub-spectrum m is processed by a preset vocoder model to obtain the corresponding intermediate audio segment; The intermediate audio segment is truncated according to the preset three-truncation offset to obtain the target audio segment; The second audio of the first timbre corresponding to the first timbre feature parameter is determined by performing the audio segment restoration operation on each of the N-1 second sub-spectrums.

6. The method according to claim 5, characterized in that, The step of truncating the intermediate audio segment according to a preset third truncation offset to obtain the target audio segment includes: Obtain the start and end times of the intermediate audio segment; The first time and the second time are determined according to the preset third truncation offset, wherein the difference between the first time and the start time is the third truncation offset, and the difference between the end time and the second time is the third truncation offset. The target audio segment is determined in the intermediate audio segment based on the first time and the second time.

7. A voice timbre conversion device, characterized in that, The device includes: The acquisition unit is used to acquire the first spectrum corresponding to the first audio. An extraction unit is used to extract a first timbre feature parameter from a timbre library, wherein the timbre library includes multiple preset timbre feature parameters; The processing unit is used to determine the second spectrum based on the first spectrum and the first timbre feature parameters using a preset timbre conversion model; The processing unit is configured to determine the second audio of the first timbre corresponding to the first timbre feature parameter based on the second spectrum; Wherein, the first spectrum includes N first sub-spectrums, and the second spectrum includes N-1 second sub-spectrums, where N is a positive integer greater than or equal to 2; The step of determining the second spectrum based on the first spectrum and the first timbre feature parameters using a preset timbre conversion model includes: For any two adjacent first sub-spectrums i and j among the N first sub-spectrums, the following spectrum synthesis operation is performed: The first intermediate spectrum k is determined based on the first sub-spectrum i and the first sub-spectrum j; A second sub-spectrum is determined by using a preset timbre conversion model based on the first timbre feature parameter and the first intermediate spectrum k. The N-1 second sub-spectrums are determined based on the result of performing the spectrum synthesis operation on each of the N first sub-spectrums between two adjacent first sub-spectrums.

8. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6.

9. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method according to any one of claims 1 to 6.

Images