Robot sound real-time conversion method, device, computer equipment and storage medium

Abstract

The present invention discloses a method, device, computer equipment and storage medium for real-time conversion of robot voices. The method comprises: collecting real-time audio and converting the real-time audio into real-time audio in frame-level units; extracting fundamental frequency features from the real-time audio through a vocoder; extracting spectrum envelope features from the fundamental frequency features, and extracting non-periodic features from the fundamental frequency features; adjusting the fundamental frequency features, and setting the target fundamental frequency value for the adjusted value; combining the spectrum envelope features, the non-periodic features and the target fundamental frequency value as a conversion target, and decoding the conversion target into voice using a decoder. The present invention extracts fundamental frequency features from real-time audio, and further extracts spectrum envelope features and non-periodic features, and then combines and decodes the spectrum envelope features and the non-periodic features and the target fundamental frequency value after feature conversion to achieve high naturalness conversion of voices, thereby providing users with different types of robot voice options.

Description

Technical Field

[0001] The present invention relates to the field of computer software technology, and in particular to a robot voice real-time conversion method, device, computer equipment and storage medium. Background Art

[0002] When creating videos, users may be reluctant to use their real voices for some reasons, and therefore hope to use other voices to replace their own voices. Robot voices can eliminate most of their own voice information and enrich the diversity of users' video creations. Therefore, robot voices are widely used in the video creation process.

[0003] At present, robot voice conversion exists in some audio and video editing software, such as audio director, iMyFoneFilme and other audio and video editing software, but different editing software has different definitions of robot voice, so the effects of robot voice are also different. In the prior art, the method of realizing robot voice conversion is usually similar to the short-time Fourier transform method, that is, first extract each frame of audio by framing, perform a windowing operation on the frame of audio, then use the fast Fourier transform on the frame and take the absolute value, move the zero frequency point to the middle of the spectrum, then use the inverse fast Fourier transform and add a window, and finally move the frame a certain distance to repeat the above operation, and superimpose the current result with the above result according to a certain ratio. However, after the implementation of the above method, a hissing electric current sound will be generated or it does not have the characteristics of the machine and is not real-time. Summary of the invention

[0004] The embodiments of the present invention provide a method, apparatus, computer device and storage medium for real-time conversion of robot voices, aiming to achieve high-naturalness conversion of voices and provide users with different types of robot voices to choose from.

[0005] In a first aspect, an embodiment of the present invention provides a method for real-time conversion of robot voices, comprising:

[0006] Collecting real-time audio and converting the real-time audio into real-time audio in frame-level units;

[0007] Extracting fundamental frequency features from the real-time audio through a vocoder;

[0008] Extracting spectrum envelope features from the fundamental frequency features, and extracting non-periodic features from the fundamental frequency features;

[0009] Performing feature adjustment on the fundamental frequency feature, and setting the value after the feature adjustment as the target fundamental frequency value;

[0010] The spectrum envelope feature, the non-periodic feature and the target fundamental frequency value are combined as a conversion target, and a decoder is used to decode the conversion target into sound.

[0011] In a second aspect, an embodiment of the present invention provides a robot voice real-time conversion device, comprising:

[0012] An audio acquisition unit, used to acquire real-time audio and convert the real-time audio into real-time audio in frame-level units;

[0013] A first feature extraction unit, configured to extract fundamental frequency features from the real-time audio through a vocoder;

[0014] A second feature extraction unit, configured to extract a spectrum envelope feature from the fundamental frequency feature, and extract a non-periodic feature from the fundamental frequency feature;

[0015] A feature adjustment unit, used to perform feature adjustment on the fundamental frequency feature and set the value after feature adjustment as a target fundamental frequency value;

[0016] A combined decoding unit is used to combine the spectrum envelope feature, the non-periodic feature and the target fundamental frequency value into a conversion target, and use a decoder to decode the conversion target into sound.

[0017] In a third aspect, an embodiment of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the robot sound real-time conversion method as described in the first aspect is implemented.

[0018] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for real-time conversion of robot sounds as described in the first aspect is implemented.

[0019] The embodiment of the present invention provides a method, device, computer equipment and storage medium for real-time conversion of robot voices. The method includes: collecting real-time audio and converting the real-time audio into real-time audio in frame-level units; extracting fundamental frequency features from the real-time audio through a vocoder; extracting spectrum envelope features from the fundamental frequency features, and extracting non-periodic features from the fundamental frequency features; adjusting the fundamental frequency features, and setting the target fundamental frequency value for the adjusted value; combining the spectrum envelope features, non-periodic features and target fundamental frequency value as a conversion target, and decoding the conversion target into voice using a decoder. The embodiment of the present invention extracts fundamental frequency features from real-time audio, and further extracts spectrum envelope features and non-periodic features, and then combines and decodes the spectrum envelope features and non-periodic features and the target fundamental frequency value after feature conversion to achieve high naturalness conversion of voices, thereby providing users with different types of robot voice options. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0021] Figure 1 A schematic diagram of a flow chart of a method for real-time conversion of robot voices provided by an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of a sub-process of a method for real-time conversion of robot voices provided by an embodiment of the present invention;

[0023] Figure 3 A schematic block diagram of a robot sound real-time conversion device provided by an embodiment of the present invention;

[0024] Figure 4 A sub-schematic block diagram of a robot sound real-time conversion device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0027] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0028] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0029] See below Figure 1 , Figure 1 A schematic flow chart of a method for real-time conversion of robot voices provided by an embodiment of the present invention specifically includes steps S101 to S105.

[0030] S101, collecting real-time audio, and converting the real-time audio into real-time audio in frame-level units;

[0031] S102, extracting fundamental frequency features from the real-time audio through a vocoder;

[0032] S103, extracting spectrum envelope features from the fundamental frequency features, and extracting non-periodic features from the fundamental frequency features;

[0033] S104, performing feature adjustment on the fundamental frequency feature, and setting the value after feature adjustment as a target fundamental frequency value;

[0034] S105: combining the spectrum envelope feature, the non-periodic feature and the target fundamental frequency value as a conversion target, and using a decoder to decode the conversion target into sound.

[0035] In this embodiment, the collected real-time audio is first converted into real-time audio at the frame level, and then the fundamental frequency feature is extracted through the vocoder, and the spectrum envelope feature and the non-periodic feature are further extracted from the fundamental frequency feature. At the same time, the fundamental frequency feature is feature-adjusted, and the fundamental frequency feature after feature adjustment (i.e., the target fundamental frequency value) is combined with the spectrum envelope feature and the non-periodic feature, and then the combined conversion target is decoded by the decoder, so that the robot voice after real-time conversion of the real-time audio can be obtained.

[0036] This embodiment extracts the fundamental frequency features of real-time audio, and further extracts the spectrum envelope features and non-periodic features, and then combines and decodes the spectrum envelope features and the non-periodic features as well as the target fundamental frequency value after feature conversion to achieve a highly natural conversion of the sound, thereby providing users with different types of robot voice options.

[0037] In one embodiment, the step S102 includes:

[0038] Using low-pass waves with different cutoff frequencies to obtain different sine waves of the real-time audio;

[0039] The fundamental frequency values ​​and credibility of all sine waves are calculated, and the fundamental frequency value with the highest credibility is selected as the fundamental frequency feature of the real-time audio.

[0040] In this embodiment, the DIO function in the WORLD vocoder is used to extract the audio fundamental frequency features in real time. Specifically, different sine waves are first found using low-pass rates with different cutoff frequencies, and then the fundamental frequencies and credibility of these sine waves are calculated, and the fundamental frequency with the highest credibility is selected as the final fundamental frequency feature.

[0041] In one embodiment, if Figure 2 As shown, the step S103 includes: steps S201 to S204.

[0042] S201, according to the following formula, use the Hanning window of 3 times the period length T0 and the fundamental frequency feature to perform a windowing operation and calculate the power:

[0043]

[0044]

[0045] Where N is the window length, n = 1, 2, 3, ..., N, y(t) represents the tth fundamental frequency feature sampling point value;

[0046] S202, performing fast Fourier transform on the window after windowing, taking the logarithm of the non-zero value, and performing smoothing using a triangular window according to the following formula:

[0047]

[0048] In the formula, w0=2π / T0, which represents the frequency, P s (w) represents power;

[0049] S203, according to the following formula, perform inverse fast Fourier transform on the smoothed base frequency feature, and use the sinc function to obtain the low frequency feature:

[0050]

[0051] Where τ represents time, ls(τ) represents the variation function used to smooth the logarithmic power spectrum and remove the time-varying components;

[0052] S204, eliminating the distortion after smoothing according to the following formula, and performing spectrum recovery to obtain the spectrum envelope feature:

[0053]

[0054] In the formula, l q represents the variation function used for spectrum recovery.

[0055] In this embodiment, when extracting the spectrum envelope feature, the fundamental frequency feature is first windowed by a Hanning window, and then the windowed window is subjected to a fast Fourier transform, and a triangular window is used for smoothing. Then, the fundamental frequency feature is subjected to an inverse fast Fourier transform, and a sinc function is used to obtain the low-frequency feature corresponding to the fundamental frequency feature. Subsequently, a variation function for spectrum recovery is used to eliminate the distortion after smoothing, so that the spectrum is restored, thereby obtaining the spectrum envelope feature.

[0056] In one embodiment, the step S103 further includes:

[0057] Calculate group delay based on phase change and frequency change;

[0058] Modifying the parameters of the fundamental frequency characteristics according to the group delay;

[0059] The non-periodic feature ap(w c ):

[0060] ap(w c )=-log 10 (P s (2w bw ,w c ))

[0061] In the formula, w bw Represents the main lobe width of the window function.

[0062] In this embodiment, the non-periodic features are extracted by using the D4C function in the WORLD vocoder. The specific steps are: calculating group delay, correcting parameters and calculating non-periodic features.

[0063] In one embodiment, the robot voice conversion method further includes:

[0064] A plurality of frame-level points are extracted from the real-time audio of the frame-level unit, and interpolation is performed accordingly to smooth the real-time audio of the frame-level unit.

[0065] In this embodiment, in order to make each frame cascade of real-time audio smoother, multiple frame-level points are extracted and interpolated to make it smoother. Specifically, three points are extracted and interpolated, and these three points are the 25th last sampling point at the end of the previous frame level, the last sampling point, and the point closest to the last sampling point of the previous frame within the first 220 sampling points at the beginning of the next frame cascade. Of course, the three points selected here are actually trend points, so the three points can also be switched to other arbitrary values.

[0066] In one embodiment, the step S104 includes:

[0067] Determine whether the fundamental frequency feature is 0;

[0068] If the fundamental frequency feature is 0, the fundamental frequency feature is directly used as the target fundamental frequency value;

[0069] If the fundamental frequency feature is not 0, the fundamental frequency feature is set to a fixed value, and the fixed value is used as the target fundamental frequency value.

[0070] In this embodiment, if the fundamental frequency feature is 0, it remains unchanged. If the fundamental frequency feature is not 0, it is adjusted to a fixed value, such as 100. Here, in this embodiment, the fundamental frequency feature is adjusted to a fixed target fundamental frequency value in order to make the real-time audio have no variance, thereby making the robot sound smoother.

[0071] In one embodiment, the step S104 further includes:

[0072] The logarithm of the fundamental frequency feature is taken, and the fundamental frequency feature after taking the logarithm is set as the target fundamental frequency value.

[0073] In this embodiment, in order to make the robot voice have a hoarse feeling, the logarithm of the fundamental frequency feature is taken, and then the fundamental frequency feature after taking the logarithm is combined with the spectrum envelope feature and the non-periodic feature and decoded. Here, since the logarithm of the fundamental frequency feature is taken, the fundamental frequency value becomes smaller and the variance becomes smaller, so that the voice has a deeper feeling. Of course, in other embodiments, it is also possible to choose not to smooth the fundamental frequency feature and not to adjust the feature of the fundamental frequency feature, but to directly combine the fundamental frequency feature with the spectrum envelope feature and the non-periodic feature and decode it, so that a hoarse robot voice can also be produced.

[0074] Figure 3 A schematic block diagram of a robot sound real-time conversion device 300 provided in an embodiment of the present invention, the device 300 includes:

[0075] The audio collection unit 301 is used to collect real-time audio and convert the real-time audio into real-time audio in frame-level units;

[0076] A first feature extraction unit 302 is used to extract fundamental frequency features from the real-time audio through a vocoder;

[0077] A second feature extraction unit 303 is used to extract a spectrum envelope feature from the fundamental frequency feature, and to extract a non-periodic feature from the fundamental frequency feature;

[0078] A feature adjustment unit 304 is used to perform feature adjustment on the fundamental frequency feature and set the value after feature adjustment as a target fundamental frequency value;

[0079] The combined decoding unit 305 is used to combine the spectrum envelope feature, the non-periodic feature and the target fundamental frequency value into a conversion target, and use a decoder to decode the conversion target into sound.

[0080] In one embodiment, the first feature extraction unit 302 includes:

[0081] A sine wave acquisition unit, used for acquiring different sine waves of the real-time audio using low-pass waves with different cut-off frequencies;

[0082] The credibility calculation unit is used to calculate the fundamental frequency values ​​and credibility of all sine waves, and select the fundamental frequency value with the highest credibility as the fundamental frequency feature of the real-time audio.

[0083] In one embodiment, if Figure 4 As shown, the second feature extraction unit 303 includes:

[0084] The windowing unit 401 is used to perform a windowing operation and calculate the power using a Hanning window of 3 times the period length T0 and the fundamental frequency feature according to the following formula:

[0085]

[0086]

[0087] Where N is the window length, n = 1, 2, 3, ..., N, y(t) represents the tth fundamental frequency feature sampling point value;

[0088] The Fourier transform unit 402 is used to perform fast Fourier transform on the window after windowing, take the logarithm of the non-zero value, and perform smoothing using a triangular window according to the following formula:

[0089]

[0090] In the formula, w0=2π / T0, which represents the frequency, P s (w) represents power;

[0091] The inverse Fourier transform unit 403 is used to perform inverse fast Fourier transform on the smoothed baseband feature according to the following formula, and obtain the low-frequency feature using the sinc function:

[0092]

[0093] Where τ represents time, ls(τ) represents the variation function used to smooth the logarithmic power spectrum and remove the time-varying components;

[0094] The spectrum recovery unit 404 is used to eliminate the distortion after smoothing according to the following formula and perform spectrum recovery to obtain the spectrum envelope feature:

[0095]

[0096] In the formula, l q represents the variation function used for spectrum recovery

[0097] In one embodiment, the second feature extraction unit 303 further includes:

[0098] A group delay calculation unit, used for calculating the group delay based on the phase change and the frequency change;

[0099] A parameter correction unit, used for correcting the parameters of the fundamental frequency characteristics according to the group delay;

[0100] The non-periodic feature extraction unit is used to extract the non-periodic feature ap(w c ):

[0101] ap(w c )=-log 10 (P s (2w bw ,w c ))

[0102] In the formula, w bw Represents the main lobe width of the window function.

[0103] In one embodiment, the robot sound conversion device 300 further includes:

[0104] The interpolation unit is used to extract multiple frame-level points from the real-time audio of the frame-level unit and perform corresponding interpolation to smooth the real-time audio of the frame-level unit.

[0105] In one embodiment, the feature adjustment unit 304 includes:

[0106] A judging unit, used to judge whether the fundamental frequency feature is 0;

[0107] A first setting unit, configured to use the fundamental frequency feature directly as a target fundamental frequency value if the fundamental frequency feature is 0;

[0108] The second setting unit is used to set the fundamental frequency feature to a fixed value if the fundamental frequency feature is not 0, and use the fixed value as the target fundamental frequency value.

[0109] In one embodiment, the feature adjustment unit 304 further includes:

[0110] The third setting unit is used to take the logarithm of the fundamental frequency feature and set the fundamental frequency feature after taking the logarithm as a target fundamental frequency value.

[0111] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0112] The embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed, the steps provided in the above embodiment can be implemented. The storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0113] The embodiment of the present invention also provides a computer device, which may include a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps provided in the above embodiment may be implemented. Of course, the computer device may also include various network interfaces, power supplies and other components.

[0114] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

[0115] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A method for real-time conversion of robot voice, characterized in that: include: Collecting real-time audio and converting the real-time audio into real-time audio in frame-level units; Extracting fundamental frequency features from the real-time audio through a vocoder; Extracting spectrum envelope features from the fundamental frequency features, and extracting non-periodic features from the fundamental frequency features; Performing feature adjustment on the fundamental frequency feature, and setting the value after the feature adjustment as the target fundamental frequency value; Combining the spectrum envelope feature, the non-periodic feature and the target fundamental frequency value as a conversion target, and decoding the conversion target into sound using a decoder; The step of extracting a spectrum envelope feature from the baseband feature comprises: According to the following formula, a Hanning window with a period length of 3 times T0 is used to perform a windowing operation and calculate the power with the fundamental frequency characteristics: Where N is the window length, n = 1, 2, 3, ..., N, y(t) represents the tth fundamental frequency feature sampling point value; Perform fast Fourier transform on the windowed window, take the logarithm of the non-zero value, and use triangular window for smoothing according to the following formula: In the formula, w0=2π / T0, which represents the frequency, P s (w) represents power; According to the following formula, the base frequency feature after smoothing is inversely transformed by fast Fourier transform, and the low-frequency feature is obtained by using the sinc function: Where τ represents time, ls(τ) represents the variation function used to smooth the logarithmic power spectrum and remove the time-varying components; The distortion after smoothing is eliminated according to the following formula, and the spectrum is restored to obtain the spectrum envelope feature: In the formula, l q represents the variation function used for spectrum recovery.

2. The robot voice real-time conversion method according to claim 1, characterized in that: The extracting fundamental frequency features of the real-time audio through a vocoder comprises: Using low-pass waves with different cutoff frequencies to obtain different sine waves of the real-time audio; The fundamental frequency values ​​and credibility of all sine waves are calculated, and the fundamental frequency value with the highest credibility is selected as the fundamental frequency feature of the real-time audio.

3. The robot voice real-time conversion method according to claim 1, characterized in that: The extracting of non-periodic features from the fundamental frequency features includes: Calculate group delay based on phase change and frequency change; Modifying the parameters of the fundamental frequency characteristics according to the group delay; The non-periodic feature ap(w c ): ap(w c )=-log 10 (P s (2w bw ,w c )) In the formula, w bw Represents the main lobe width of the window function.

4. The robot voice real-time conversion method according to claim 1, characterized in that: Also includes: A plurality of frame-level points are extracted from the real-time audio of the frame-level unit, and interpolation is performed accordingly to smooth the real-time audio of the frame-level unit.

5. The robot voice real-time conversion method according to claim 1, characterized in that: The step of adjusting the fundamental frequency feature and setting the value after the feature adjustment as the target fundamental frequency value includes: Determine whether the fundamental frequency feature is 0; If the fundamental frequency feature is 0, the fundamental frequency feature is directly used as the target fundamental frequency value; If the fundamental frequency feature is not 0, the fundamental frequency feature is set to a fixed value, and the fixed value is used as the target fundamental frequency value.

6. The method for real-time conversion of robot voice according to claim 1, characterized in that: The step of adjusting the fundamental frequency feature and setting the adjusted value as the target fundamental frequency value further includes: The logarithm of the fundamental frequency feature is taken, and the fundamental frequency feature after taking the logarithm is set as the target fundamental frequency value.

7. A robot voice real-time conversion device, characterized in that: include: An audio acquisition unit, used to acquire real-time audio and convert the real-time audio into real-time audio in frame-level units; A first feature extraction unit, configured to extract fundamental frequency features from the real-time audio through a vocoder; A second feature extraction unit, configured to extract a spectrum envelope feature from the fundamental frequency feature, and extract a non-periodic feature from the fundamental frequency feature; A feature adjustment unit, used to perform feature adjustment on the fundamental frequency feature and set the value after feature adjustment as a target fundamental frequency value; A combined decoding unit is used to combine the spectrum envelope feature, the non-periodic feature and the target fundamental frequency value into a conversion target, and decode the conversion target into sound using a decoder; The second feature extraction unit comprises: The windowing unit is used to perform a windowing operation and calculate the power using a Hanning window of 3 times the period length T0 and the fundamental frequency characteristic according to the following formula: Where N is the window length, n = 1, 2, 3, ..., N, y(t) represents the tth fundamental frequency feature sampling point value; The Fourier transform unit is used to perform fast Fourier transform on the window after windowing, take the logarithm of non-zero values, and use a triangular window for smoothing according to the following formula: In the formula, w0=2π / T0, which represents the frequency, P s (w) represents power; The inverse Fourier transform unit is used to perform a fast inverse Fourier transform on the smoothed base frequency feature according to the following formula, and obtain the low frequency feature using the sinc function: Where τ represents time, ls(τ) represents the variation function used to smooth the logarithmic power spectrum and remove the time-varying components; The spectrum recovery unit is used to eliminate the distortion after smoothing according to the following formula and perform spectrum recovery to obtain the spectrum envelope feature: In the formula, l q represents the variation function used for spectrum recovery.

8. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the robot sound real-time conversion method as claimed in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for real-time conversion of robot sounds according to any one of claims 1 to 6 is implemented.

Images