Speech processing method, apparatus, device, and computer-readable storage medium

Abstract

The application discloses a voice processing method, device and equipment and a computer readable storage medium. Specifically, first voiceprint features of a first voice and second voiceprint features of a second voice are extracted, the first voiceprint features and the second voiceprint features are fused according to a preset fusion weight to obtain third voiceprint features, a similarity ratio between the third voiceprint features and the first voiceprint features is determined, and finally a fused voice is generated according to the third voiceprint features. Thus, the first voiceprint features and the second voiceprint features corresponding to the first voice and the second voice can be extracted first, the third voiceprint feature parameters are obtained after weighted calculation, and then the first voice is subjected to frequency domain conversion and time domain conversion by using the parameters of the third voiceprint features, so that the fused voice is obtained. In this way, on the premise of retaining the voice features, multiple voices are fused into one voice, the influence of the mixing phenomenon on the auditory effect of the user is reduced, and the experience of the user is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a speech processing method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] With the development of voice technology, voice optimization technology has been applied to all aspects of life. Among them, voice optimization technology can combine the voice that needs to be optimized with a second voice to optimize the voice. For example, when a user is singing, the relevant technology can use real-time tuning to achieve the purpose of integrating the voice and improving the user's auditory experience.

[0003] However, while related technologies can optimize a user's voice through real-time tuning, the coexistence of other sounds, such as the original singer's voice while the user is singing, can create a mix of multiple sounds in the same space, resulting in severe mixing and affecting the user's auditory experience, thus degrading the user's overall experience. Summary of the Invention

[0004] This application provides a voice processing method, apparatus, device, and computer-readable storage medium that can fuse different sounds into one sound and integrate different voiceprint features to reduce the impact of mixing on the user's auditory experience and improve the user's experience.

[0005] This application provides a speech processing method, including:

[0006] Extract the first voiceprint feature of the first speech and the second voiceprint feature of the second speech;

[0007] Obtain a preset fusion weight between the first speech and the second speech, and fuse the first voiceprint feature and the second voiceprint feature according to the preset fusion weight to obtain the third voiceprint feature;

[0008] Determine a first feature similarity between the third voiceprint feature and the first voiceprint feature, and determine a second feature similarity between the third voiceprint feature and the second voiceprint feature;

[0009] Calculate the similarity ratio between the first feature similarity and the second feature similarity, and determine the fusion weight difference between the similarity ratio and the preset fusion weight;

[0010] When the difference in fusion weights is less than or equal to a preset weight threshold, the third voiceprint feature is converted into fused speech.

[0011] Accordingly, embodiments of this application provide a voice processing device, including:

[0012] An extraction unit is used to extract the first voiceprint feature of the first speech and the second voiceprint feature of the second speech.

[0013] A fusion unit is used to obtain a preset fusion weight between the first speech and the second speech, and to fuse the first voiceprint feature and the second voiceprint feature according to the preset fusion weight to obtain a third voiceprint feature.

[0014] A determining unit is configured to determine a first feature similarity between the third voiceprint feature and the first voiceprint feature, and to determine a second feature similarity between the third voiceprint feature and the second voiceprint feature;

[0015] The calculation unit is used to calculate the similarity ratio between the first feature similarity and the second feature similarity, and to determine the fusion weight difference between the similarity ratio and the preset fusion weight;

[0016] The conversion unit is used to generate fused speech based on the third voiceprint feature when the fusion weight difference is less than or equal to a preset weight threshold.

[0017] In some embodiments, the extraction unit is further configured to:

[0018] The spectral information corresponding to the first speech is processed by frame segmentation, and the fundamental frequency value and formant parameters of each frame of the first speech segment obtained by frame segmentation are calculated to obtain the first voiceprint feature.

[0019] The spectral information corresponding to the second speech is processed by frame segmentation, and the fundamental frequency value and formant parameters of each frame of the second speech segment obtained by frame segmentation are calculated to obtain the second voiceprint feature.

[0020] In some embodiments, the fusion unit is further configured to:

[0021] Read the first fundamental frequency value and the first resonance peak value in the first voiceprint feature, and read the second fundamental frequency value and the second resonance peak value in the second voiceprint feature;

[0022] According to the preset fusion weight, a weighted calculation is performed between the first fundamental frequency value and the second fundamental frequency value to obtain the third fundamental frequency value;

[0023] According to the preset fusion weight, the first resonance peak and the second resonance peak are transformed to obtain the third resonance peak;

[0024] The third voiceprint feature is determined based on the third fundamental frequency and the third resonance peak.

[0025] In some embodiments, the conversion unit is further configured to:

[0026] Read the spectrum information corresponding to the first voice;

[0027] The first fundamental frequency value and the first resonance peak value in the spectrum information are updated according to the third voiceprint feature to obtain the updated target spectrum information;

[0028] The target spectral information is converted in the time domain to obtain fused speech.

[0029] In some embodiments, the voice processing apparatus further includes an update unit for:

[0030] If the difference in fusion weights is greater than a preset weight threshold, the preset fusion weights are updated based on the difference in fusion weights to obtain the updated target fusion weights.

[0031] The first voiceprint feature and the second voiceprint feature are fused according to the target fusion weight to obtain the target voiceprint feature;

[0032] Generate fused speech based on the target voiceprint features.

[0033] In some embodiments, the updating unit is further configured to:

[0034] Obtain the target sampling frequency of the first speech;

[0035] The fundamental frequency correction step size of the third fundamental frequency value is determined based on the ratio between the third fundamental frequency value and the target sampling frequency;

[0036] The resonant peak correction step size of the third resonant peak is determined based on the ratio between the third resonant peak value and the target sampling frequency.

[0037] Based on the fundamental frequency correction step size and the resonant peak correction step size, the preset fusion weight is updated until the difference between the updated preset fusion weight and the fusion weight is less than or equal to the preset weight threshold. The updated preset fusion weight is then determined as the target fusion weight.

[0038] In some embodiments, the voice processing apparatus further includes a sampling unit for:

[0039] If the first sampling frequency is equal to the second sampling frequency, then the first sampling frequency is determined as the target sampling frequency, and speech is sampled according to the target sampling frequency to obtain the first speech and the second speech.

[0040] If the first sampling frequency is greater than the second sampling frequency, then the second sampling frequency is determined as the target sampling frequency, and speech is sampled according to the target sampling frequency to obtain the first speech and the second speech.

[0041] Furthermore, this application also provides a computer device, including a processor and a memory, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to implement the steps in the voice processing method provided in this application.

[0042] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the speech processing methods provided in embodiments of this application.

[0043] Furthermore, embodiments of this application also provide a computer program product, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of any of the speech processing methods provided in embodiments of this application.

[0044] This application embodiment extracts the first voiceprint feature of the first speech and the second voiceprint feature of the second speech. Then, it obtains the preset fusion weight between the first speech and the second speech, and fuses the first voiceprint feature and the second voiceprint feature according to the preset fusion weight to obtain the third voiceprint feature. Then, it determines the first feature similarity between the third voiceprint feature and the first voiceprint feature, and determines the second feature similarity between the third voiceprint feature and the second voiceprint feature. Then, it calculates the similarity ratio between the first feature similarity and the second feature similarity, and determines the fusion weight difference between the similarity ratio and the preset fusion weight. Finally, when the fusion weight difference is less than or equal to the preset weight threshold, it generates fused speech based on the third voiceprint feature. Therefore, this solution first extracts the first and second voices that need to be fused, then extracts the voice features of the first and second voices, namely the corresponding first and second voiceprint features. After weighting the parameters of the first and second voiceprint features, a third voiceprint feature parameter is obtained. Then, the first speech is transformed in both the frequency domain and the time domain using the parameters of the third voiceprint feature to obtain the fused speech. In this way, multiple voices are fused into one voice while preserving the voice features, reducing the impact of mixing on the user's auditory experience and improving the user's experience. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a schematic diagram of a scenario for the voice processing system provided in an embodiment of this application;

[0047] Figure 2 This is a flowchart illustrating the steps of the speech processing method provided in the embodiments of this application;

[0048] Figure 3 This is a schematic flowchart of another step of the speech processing method provided in the embodiments of this application;

[0049] Figure 4 This is a schematic diagram of the structure of the voice processing device provided in the embodiments of this application;

[0050] Figure 5 This is a schematic diagram of the structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] This application provides a voice processing method, apparatus, device, and computer-readable storage medium. This application will describe the voice processing apparatus from the perspective of the voice processing apparatus itself. Specifically, the voice processing apparatus can be integrated into a computer device, which can be a terminal device, specifically a terminal device mounted on a voice processing tool, i.e., a voice terminal. Furthermore, the terminal device can also be other types of devices, such as a television, smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, smart wearable device, etc.; however, it is not limited to these.

[0053] For example, see Figure 1 This is a schematic diagram of a scenario for a voice processing system provided in an embodiment of this application. The scenario includes a terminal or a server.

[0054] Specifically, the terminal can be a voice terminal, used to extract the first voiceprint feature of the first speech and the second voiceprint feature of the second speech, then obtain a preset fusion weight between the first speech and the second speech, and fuse the first voiceprint feature and the second voiceprint feature according to the preset fusion weight to obtain the third voiceprint feature, then determine the first feature similarity between the third voiceprint feature and the first voiceprint feature, and determine the second feature similarity between the third voiceprint feature and the second voiceprint feature, then calculate the similarity ratio between the first feature similarity and the second feature similarity, and determine the fusion weight difference between the similarity ratio and the preset fusion weight. Finally, when the fusion weight difference is less than or equal to the preset weight threshold, fused speech is generated based on the third voiceprint feature.

[0055] It should be noted that when the voice processing system includes a server, a communication connection can be established between the voice terminal and the server. The voice terminal can extract the first voiceprint feature of the first voice and the second voiceprint feature of the second voice, obtain a preset fusion weight between the first and second voices, and fuse the first and second voiceprint features according to the preset fusion weight to obtain a third voiceprint feature. The obtained first, second, and third voiceprint features are then sent to the server. At this time, the server can receive the first, second, and third voiceprint features sent by the voice terminal. Based on the third voiceprint feature, it determines the first feature similarity between the third and first voiceprint features, and the second feature similarity between the third and second voiceprint features. It calculates the similarity ratio between the first and second feature similarities and determines the fusion weight difference between the similarity ratio and the preset fusion weight. When the fusion weight difference is less than or equal to the preset weight threshold, fused voice is generated based on the third voiceprint feature. At this time, the server can send a control strategy or instruction for generating fused voice based on the third voiceprint feature to the voice terminal. This enables the voice terminal to execute the lane change control policy or instruction received from the server and generate fused speech based on the third voiceprint feature.

[0056] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the preferred order of the embodiments.

[0057] In this embodiment, the description will be from the perspective of a voice processing device, which can be specifically integrated into a computer device such as a terminal device or a server. See also Figure 2 , Figure 2 This is a flowchart illustrating the steps of a voice processing method provided in an embodiment of this application. Taking a terminal device as an example, the terminal device is a terminal mounted on a voice processing tool. When the processor on the terminal device executes the program corresponding to the voice processing method, the specific flow of the voice processing method is as follows:

[0058] 101. Extract the first voiceprint feature of the first speech and the second voiceprint feature of the second speech.

[0059] In this embodiment of the application, in order to combine the sound characteristics of multiple sounds and merge them into one sound to avoid mixing, the voiceprint features corresponding to each sound can be obtained in advance, and then the voiceprint features can be used to merge multiple sounds to reduce the impact of mixing on the user's auditory experience.

[0060] The first voice can be the sound produced by the user, which can be acquired using a sound acquisition device. For example, when a user sings in a karaoke room, the user's voice can be recorded using a microphone device.

[0061] The second voice can be another voice that exists synchronously with the user's voice, such as the original voice of the singer when the user is singing.

[0062] The first voiceprint feature can be a visualized sound feature exhibited by the speech characteristics in the first speech, including but not limited to changes in the intensity, wavelength, frequency, and rhythm of the electrical signal converted from changes in sound waves.

[0063] The second voiceprint feature can be a visual sound feature exhibited by the speech characteristics in the second speech, including but not limited to changes in the intensity, wavelength, frequency, and rhythm of the electrical signal converted from changes in sound waves.

[0064] Specifically, after acquiring the first and second speech samples to be fused, speech is not confined to a fixed frequency range once the pitch is determined. Instead, regardless of pitch, speech is distributed across a wide frequency range. In other words, speech is actually synthesized from sounds of several frequencies. Therefore, it is necessary to perform speech analysis on the acquired first and second speech samples. This includes acoustic features related to the anatomical structure of the user's articulation mechanism (such as spectrum, cepstral, formants, fundamental tone, reflection coefficient, etc.) as well as feature extraction for nasality, deep breathing sounds, hoarseness, laughter, etc. In particular, speech is represented by the sound wave spectrum carrying speech information as an electrical signal, such as the intensity, wavelength, frequency, and rhythmic variations of the audio frequency domain signal. After extracting the sound features of the first speech sample, these are used as the speech features of the first speech sample, i.e., the first voiceprint features. Similarly, the second speech sample is also extracted using the same method to obtain the second voiceprint features.

[0065] In some implementations, the fundamental frequency value and formant parameters of the speech segments can be obtained by processing the spectral information corresponding to the first and second speech segments, thereby determining the first and second voiceprint features. For example, step 101, "extracting the first voiceprint feature of the first speech and the second voiceprint feature of the second speech," may include:

[0066] (101.1) Perform frame-segmentation processing on the spectral information corresponding to the first speech, and calculate the fundamental frequency value and formant parameters of each frame of the first speech segment obtained by the frame-segmentation processing to obtain the first voiceprint feature;

[0067] (101.2) Perform frame-by-frame processing on the spectral information corresponding to the second speech, and calculate the fundamental frequency value and formant parameters of each frame of the second speech segment obtained by the frame-by-frame processing to obtain the second voiceprint feature.

[0068] Among them, the spectrum information can indicate which frequency sine waves a speech signal is composed of, and can also show information such as the magnitude and phase of each frequency sine wave.

[0069] Specifically, after obtaining the first and second speech samples to be fused, a Fourier transform can be performed on the first speech signal to obtain its spectrogram. The resulting spectrogram will be plotted with amplitude and phase as the vertical axis and frequency as the horizontal axis. Then, based on the obtained spectrogram, the corresponding spectral information in the spectrogram is processed into frames to divide the speech into multiple average speech segments, i.e., the first speech segments. The fundamental frequency value and formant parameters of each frame of the first speech segment obtained by the frame processing are calculated to obtain the first voiceprint feature. The methods for calculating the fundamental frequency include, but are not limited to, the cepstral method, the autocorrelation method, and LPC (linear prediction method). For example, taking the cepstral method, since the first speech is obtained by filtering the vocal tract response after glottal pulse excitation, and the first speech and vocal tract response are relatively separated in the cepstral domain (i.e., the acoustic pulse cepstral containing fundamental frequency information can be separated from the vocal tract response cepstral), the glottal pulse excitation and vocal tract response can be separated from the cepstral domain to obtain the cepstral. The maximum value of the cepstral function is then found within the cepstral frequency range. The number of sample points corresponding to the maximum value of the cepstral function is the fundamental period of the first speech signal in the current frame, thus obtaining the fundamental frequency value. The formant parameters of the first speech can be obtained using methods such as the cepstral method and LPC (linear prediction method). The fundamental frequency value and formants constitute the first voiceprint feature. Similarly, the second voiceprint feature corresponding to the second speech can also be determined using the above method.

[0070] In some implementations, the speech processing method may further include first determining a uniform speech sampling frequency, and then sampling multiple speech samples in combination with the uniform sampling frequency. For example, before "extracting the first voiceprint feature of the first speech and the second voiceprint feature of the second speech" in step (101), the method may further include:

[0071] (101.a.1) Obtain the first sampling frequency of the first speech and the second sampling frequency of the second speech;

[0072] (101.a.2) If the first sampling frequency is equal to the second sampling frequency, the first sampling frequency is determined as the target sampling frequency, and speech is sampled according to the target sampling frequency to obtain the first speech and the second speech.

[0073] (101.a.3) If the first sampling frequency is greater than the second sampling frequency, the second sampling frequency is determined as the target sampling frequency, and speech is sampled according to the target sampling frequency to obtain the first speech and the second speech.

[0074] The first sampling frequency can be defined as the number of samples that the computer can continuously extract from the first speech and assemble into a discrete signal per unit time, expressed in Hertz (Hz). In simpler terms, the first sampling frequency refers to how many signal samples of the first speech can be collected by the computer per unit time.

[0075] The second sampling frequency can be defined as the number of samples that the computer can continuously extract from the second speech and assemble into a discrete signal per unit time, expressed in Hertz (Hz). In simpler terms, the second sampling frequency refers to how many signal samples of the second speech can be collected by the computer per unit time.

[0076] The target sampling frequency can be the smaller of the first and second sampling frequencies. The target sampling frequency will be used to collect signal samples from the first and second speech.

[0077] Specifically, after acquiring the audio corresponding to the first and second speech, the number of samples that the computer can continuously extract from the first and second speech to form discrete signals per unit time is determined, namely the first sampling frequency and the second sampling frequency. The first sampling frequency and the second sampling frequency are compared. If the sampling frequencies are equal, either the first sampling frequency or the second sampling frequency is selected as the target sampling frequency. If the sampling frequencies are not equal, the smaller sampling frequency is selected as the target sampling frequency. The other speech with a higher sampling rate is then transsampled to the target sampling frequency, thereby determining the sampling of the first and second speech.

[0078] By using the above methods, the sound features of different voices can be obtained by extracting the first voiceprint feature of the first voice and the second voiceprint feature of the second voice. This provides a fusion data set for subsequent sound fusion, so as to achieve the effect of fusion of multiple voices into one voice while preserving the sound features, thereby reducing the impact of mixing on the user's auditory experience.

[0079] 102. Obtain the preset fusion weight between the first speech and the second speech, and fuse the first voiceprint feature and the second voiceprint feature according to the preset fusion weight to obtain the third voiceprint feature.

[0080] The preset fusion weight can be the proportion of the first voiceprint feature and the second voiceprint feature contained in the third voiceprint feature after the first voiceprint feature and the second voiceprint feature are fused. The greater the weight of the speech, the closer the fused speech is to the original speech.

[0081] Among them, the third voiceprint feature can be the visualized sound features shown by the speech characteristics in the fused speech, including but not limited to the changes in the intensity, wavelength, frequency, rhythm, etc. of the electrical signals converted from changes in sound waves.

[0082] Specifically, after obtaining the first and second voiceprint features corresponding to the first and second speech to be fused, the preset fusion weights between the first and second speech are determined. For example, if the fused voice is required to sound more like the first speech, the fusion weight corresponding to the first speech is increased and the fusion weight corresponding to the second speech is decreased. Based on the fusion weights corresponding to the first and second speech, the first and second voiceprint features are weighted and fused to obtain a third voiceprint feature that retains both the first and second voiceprint features. This third voiceprint feature is the voiceprint feature of the fused speech.

[0083] In some implementations, a third fundamental frequency value and a third resonance peak value can be obtained by weighting the fundamental frequency value and resonance peak value corresponding to the first and second voiceprint features, and then determining the third voiceprint feature based on the third fundamental frequency value and the third resonance peak value. For example, step 102, "fusing the first and second voiceprint features according to a preset fusion weight to obtain the third voiceprint feature," may include:

[0084] (102.1) Read the first fundamental frequency value and the first resonance peak value in the first voiceprint feature, and read the second fundamental frequency value and the second resonance peak value in the second voiceprint feature;

[0085] (102.2) According to the preset fusion weight, the first fundamental frequency value and the second fundamental frequency value are weighted and calculated to obtain the third fundamental frequency value;

[0086] (102.3) According to the preset fusion weight, the first resonance peak and the second resonance peak are transformed to obtain the third resonance peak;

[0087] (102.4) Determine the third voiceprint feature based on the third fundamental frequency and the third resonance peak.

[0088] The first fundamental frequency value can be the first fundamental tone frequency, which is a parameter value extracted from the fundamental frequency value of the first speech. It is also the most important criterion for speech recognition. The first fundamental frequency reflects the frequency of vocal cord vibration when the first speech is produced.

[0089] The first resonance peak can be a parameter value extracted from the first speech resonance peak parameter. It refers to some regions in the speech spectrum where energy is relatively concentrated. The resonance peak is not only a determining factor of sound quality, but also reflects the physical characteristics of the vocal tract (resonance cavity).

[0090] The second fundamental frequency value can be the second fundamental tone frequency, which is a parameter value extracted from the fundamental frequency value of the second speech. It is also the most important discrimination criterion for speech recognition. The second fundamental frequency reflects the frequency of vocal cord vibration when the second speech is pronounced.

[0091] The second resonance peak can be a parameter value extracted from the second speech resonance peak parameter. It refers to some regions in the speech spectrum where energy is relatively concentrated. The resonance peak is not only a determining factor of sound quality, but also reflects the physical characteristics of the vocal tract (resonance cavity).

[0092] The third fundamental frequency value can be the second fundamental frequency, which is the fundamental frequency obtained by weighting the first and second fundamental frequency values. The third fundamental frequency reflects the vocal cord vibration frequency during fused speech pronunciation.

[0093] The third resonant peak can be obtained by weighting the first and second resonant peaks. It refers to some regions in the speech spectrum where energy is relatively concentrated. The resonant peaks are not only a determining factor of sound quality, but also reflect the physical characteristics of the vocal tract (resonance cavity).

[0094] Specifically, after obtaining the first and second voiceprint features corresponding to the first and second speech, the fundamental frequency value and formant parameters in the frequency domain data corresponding to the first and second voiceprint features are read. Specific methods for obtaining the fundamental frequency value include, but are not limited to, cepstral method, autocorrelation method, and LPC (linear prediction method). Specific methods for obtaining the formant parameters include, but are not limited to, cepstral method and LPC (linear prediction method). This determines the first fundamental frequency value and first formant peak value of the first voiceprint feature, and the second fundamental frequency value and second formant peak value of the second voiceprint feature. Next, the fusion weights for fusing the first and second voiceprint features are obtained, i.e., preset fusion weights. The fused voiceprint feature, i.e., the third voiceprint feature, will have a speech feature that is closer to the voiceprint feature with the larger preset fusion weight. For example, if user A is singing in a karaoke bar with the original vocals enabled, user A prefers the fused vocals to be closer to their own voice; therefore, the weight of user A's own voice will be greater than the weight of the original vocals.

[0095] Specifically, using preset fusion weights, the first and second fundamental frequency values ​​are calculated to obtain the third fundamental frequency value. The specific expression for calculating the fundamental frequency value is: F0 = a% * F o0 +b%*F r0Where a>0, b>0, a+b=100, F0 is the third fundamental frequency value, a is the preset fusion weight of the first voiceprint feature, and F o0 Let b be the first fundamental frequency value, b be the preset fusion weight of the first voiceprint feature, and F be the second fundamental frequency value. r0 This is the second fundamental frequency value. Similarly, using preset fusion weights, the first and second resonance peak values ​​are calculated to obtain the third resonance peak value. The specific expression for calculating the resonance peak value is: F1 = a * F o1 +b%*F r1 Where a>0, b>0, a+b=100, F1 is the third resonance peak, a is the preset fusion weight of the first voiceprint feature, and F o1 Let b be the first resonance peak value, b be the preset fusion weight of the first voiceprint feature, and F be the second resonance peak value. r1 This is the second resonance peak.

[0096] It's worth noting that since a speech can have multiple formants, these formants together determine the timbre of the speech. For example, a user's speech might have the following characteristics: the size of the tongue opening (also called the height of the tongue), the front and back of the tongue, and the rounding of the lips. These features are basically sufficient to describe the vowel timbre of a speech, and they are closely related to the values ​​of the formants. Therefore, the first formant peak corresponding to the first voiceprint feature consists of four formants, meaning the first formant peak corresponds to four formant peaks. Similarly, the second formant peak corresponding to the second voiceprint feature also includes four formant peaks. After weighting the four formant peaks of the first and second formant peaks, the third formant peak of the third voiceprint feature also has four formant peaks, as specifically expressed in the following expression:

[0097] F1 = a% * F o1 +b%*F r1

[0098] F2 = a% * F o2 +b%*F r2

[0099] F3 = a% * F o3 +b%*F r3

[0100] F4 = a% * F o4 +b%*F r4 ,

[0101] Among them, the four resonance peaks corresponding to the first resonance peak of the first voiceprint feature are: F o1 F o2 F o3 F o4 The four resonant peaks corresponding to the second resonance peak of the second voiceprint feature are: F r1 Fr2 F r3 F r4 The four resonant peaks corresponding to the third resonant peak of the third voiceprint feature are F1, F2, F, and F4, respectively. a and b are preset fusion weights, and a>0, b>0, a+b=100.

[0102] By using the above method, a third voiceprint feature can be obtained by obtaining the preset fusion weight between the first and second voiceprints and fusing the first and second voiceprint features according to the preset fusion weight. This achieves the effect of fusing multiple voices into one voice while preserving the voice features, thereby reducing the impact of mixing on the user's auditory experience.

[0103] 103. Determine the first feature similarity between the third voiceprint feature and the first voiceprint feature, and determine the second feature similarity between the third voiceprint feature and the second voiceprint feature.

[0104] In this embodiment of the application, in order to ensure that the third voiceprint feature obtained by fusion meets the expected preset fusion weight, the similarity between the fused third voiceprint feature and the first and second voiceprint features is calculated respectively, so as to ensure that the third voiceprint feature is successfully fused, so that the subsequent fused speech achieves the expected fusion effect and improves the user experience.

[0105] The first feature similarity can be the degree of difference between the third voiceprint feature and the first voiceprint feature after voiceprint comparison, including but not limited to the similarity of intensity, wavelength, frequency, and rhythm changes.

[0106] The second feature similarity can be the degree of difference between the third voiceprint feature and the second voiceprint feature after voiceprint comparison, including but not limited to the similarity of intensity, wavelength, frequency, and rhythm changes.

[0107] Specifically, after fusing the first voiceprint feature of the first speech and the second voiceprint feature of the second speech, a third voiceprint feature is obtained. Then, voiceprint comparison or speech similarity scoring methods can be used to compare the third voiceprint feature with the first and second voiceprint features respectively, and score them in a score system to determine the similarity, namely the first feature similarity and the second feature similarity. For example, when comparing the third voiceprint feature and the first voiceprint feature using a speech similarity scoring algorithm, the third voiceprint feature and the first voiceprint feature can first be frequency domain shaped, including level adjustment and filtering. Then, the third voiceprint feature and the first voiceprint feature can be time-converted for auditory conversion. After perturbation and distortion calculations are performed on the converted speech, asymmetric processing is applied, and the similarity can be scored to obtain the first feature similarity. The method for obtaining the second feature similarity can also be the same.

[0108] By using the above methods, the first feature similarity between the third voiceprint feature and the first voiceprint feature, as well as the second feature similarity between the third voiceprint feature and the second voiceprint feature, can be determined to ensure successful fusion of the third voiceprint feature, so that the subsequent fused speech achieves the expected fusion effect and improves the user experience.

[0109] 104. Calculate the similarity ratio between the first feature similarity and the second feature similarity, and determine the fusion weight difference between the similarity ratio and the preset fusion weight.

[0110] In this embodiment of the application, in order to ensure that the third voiceprint feature obtained by fusion meets the expected preset fusion weight, the ratio between the first feature similarity and the second feature similarity is used to determine the fusion weight difference between it and the preset fusion weight, thereby ensuring that the third voiceprint feature is successfully fused, so that the subsequent fused speech achieves the expected fusion effect and improves the user experience.

[0111] Among them, the fusion weight difference can be the difference between the similarity ratio and the preset fusion weight, which is one of the judgment conditions for determining whether the third voiceprint feature has achieved the expected fusion.

[0112] Specifically, after determining the similarity between the third voiceprint feature and the first and second voiceprint features respectively, the similarity ratio between the first and second features is calculated using the first feature similarity as the denominator and the second feature similarity as the numerator. Then, the fusion weight ratio is calculated using the preset fusion weight value of the first voiceprint feature as the denominator and the preset fusion weight of the second voiceprint feature as the numerator. The difference between the similarity ratio and the preset fusion weight is the fusion weight difference between the similarity ratio and the preset fusion weight. This fusion weight difference can be used to determine whether the fused third voiceprint feature has reached the expected weight ratio.

[0113] By calculating the similarity ratio between the first feature similarity and the second feature similarity, and determining the fusion weight difference between the similarity ratio and the preset fusion weight, the fusion weight difference between the third voiceprint feature and the preset fusion weight can be determined, thereby ensuring the successful fusion of the third voiceprint feature and enabling the subsequent fused speech to achieve the expected fusion effect, thus improving the user experience.

[0114] 105. When the difference in fusion weights is less than or equal to the preset weight threshold, fused speech is generated based on the third voiceprint feature.

[0115] In this embodiment of the application, in order to obtain fused speech through the third voiceprint feature, the frequency domain data corresponding to the third voiceprint feature can be used to perform time domain conversion, thereby obtaining the audio of the fused speech, achieving the effect of fusing multiple sounds into one sound while preserving sound features, reducing the impact of mixing on the user's auditory effect, and improving the user's experience.

[0116] The preset weight threshold can be a fusion weight ratio calculated by using the preset fusion weight value of the first voiceprint feature as the denominator and the preset fusion weight of the second voiceprint feature as the numerator. This ratio is one of the judgment conditions for determining whether the third voiceprint feature has reached the expected fusion.

[0117] Among them, fused speech can be speech that combines and retains the sound features of the first speech and the second speech.

[0118] Specifically, after obtaining the fusion weight difference between the similarity ratio and the preset fusion weight, the preset weight threshold is obtained by using the preset fusion weights of the first and second voiceprint features. Then, the absolute value of the fusion weight difference is compared with the preset weight threshold. If the fusion weight difference is less than or equal to the preset weight threshold, it means that the fused third voiceprint feature has reached the expected weight ratio, indicating that the frequency domain parameters of the third voiceprint feature are correct and the fusion of voiceprint features between the first and second speech has achieved the effect of the preset fusion weight. Then, the third voiceprint feature is converted into the audio voiceprint of the speech audio and the corresponding speech audio is generated, that is, the fused speech is generated based on the third voiceprint feature.

[0119] Specifically, the expression for the judgment condition of the fusion expectation is as follows:

[0120]

[0121] Where α and β are preset parameters, with α having a default value of 1 and β having a default value of 0.2, S o S represents the first feature similarity. r denoted as the second feature similarity, where a is the preset fusion weight corresponding to the first voiceprint feature, and b is the preset fusion weight corresponding to the second voiceprint feature.

[0122] In some implementations, the corresponding spectral information can be obtained based on the first speech, and the fundamental frequency value and resonant peak value in the spectral information can be updated according to the third voiceprint features, thereby updating the frequency domain data. Then, the fused speech is obtained by converting between the frequency domain and the time domain. For example, step 105, "generating fused speech based on the third voiceprint features," may include:

[0123] (105.1) Read the spectrum information corresponding to the first speech;

[0124] (105.2) Update the first fundamental frequency value and the first resonance peak value in the spectrum information according to the third voiceprint feature to obtain the updated target spectrum information;

[0125] (105.3) The target spectrum information is converted in the time domain to obtain the fused speech.

[0126] The target spectrum information can be the spectrum information corresponding to the fused speech, which is the frequency domain data obtained by replacing and adjusting the frequency domain data in the first speech based on the audio of the first speech and using the frequency domain data corresponding to the third voiceprint feature.

[0127] Specifically, after determining that the difference in fusion weights is less than or equal to a preset weight threshold, the spectral information of the first speech is read based on the audio of the first speech. Then, the frequency domain data corresponding to the third voiceprint feature, namely the third fundamental frequency and the third resonant peak, is used to replace and adjust the frequency domain data in the first speech to obtain the target spectral information. Since signal processing has two domains, one called the time domain and the other called the frequency domain, playing audio uses time domain data, while processing audio usually uses frequency data. Therefore, after replacing and adjusting the frequency domain data in the first speech, it is necessary to convert the frequency domain data to the time domain. The converted frequency domain data can be converted to the time domain using Fourier transform to obtain the fused speech.

[0128] By using the above methods, fused speech can be generated based on the third voiceprint features, thereby obtaining the audio of the fused speech. This achieves the effect of merging multiple sounds into one sound while preserving their sound characteristics, reducing the impact of mixing on the user's auditory experience and improving the user's experience.

[0129] In this embodiment of the application, the voice processing method further includes step 106, which may include:

[0130] (106.1) If the difference in fusion weights is greater than the preset weight threshold, the preset fusion weights are updated according to the difference in fusion weights to obtain the updated target fusion weights;

[0131] (106.2) The first voiceprint feature and the second voiceprint feature are fused according to the target fusion weight to obtain the target voiceprint feature;

[0132] (106.3) Generate fused speech based on the target voiceprint features.

[0133] The target fusion weight can be the fusion weight obtained after adjusting the preset fusion weight because the third voiceprint feature does not meet the fusion expectation.

[0134] The target voiceprint features can be the first voiceprint features and the second voiceprint features. The voiceprint features obtained by the updated target fusion weights include, but are not limited to, changes in the intensity, wavelength, frequency, and rhythm of the electrical signal converted from changes in sound waves.

[0135] Specifically, after calculating the fusion weight difference between the similarity ratio and the preset fusion weight, the preset weight threshold is obtained by using the preset fusion weights of the first and second voiceprint features. Then, the absolute value of the fusion weight difference is compared with the preset weight threshold. If the fusion weight difference is greater than the preset weight threshold, it indicates that the fused third voiceprint feature has not reached the expected weight ratio, indicating that the frequency domain parameters of the third voiceprint feature are problematic. The fusion of voiceprint features between the first and second speech has not achieved the effect of the preset fusion weight. At this time, the preset fusion weight is updated according to the positive or negative value of the fusion weight difference to obtain the target fusion weight. Then, the first and second voiceprint features are weighted and fused to obtain the target voiceprint feature that retains both the first and second voiceprint features. This target voiceprint feature is the refused voiceprint feature. The similarity between the target voiceprint features and the first and second voiceprint features is calculated respectively to obtain the similarity ratio. The new fusion weight difference between the similarity ratio and the target fusion weight is determined. If the new fusion weight difference is less than or equal to the preset weight threshold, the fused speech is generated based on the target voiceprint features.

[0136] In some implementations, the correction step size can be determined based on the target sampling frequency of the first speech, thereby updating the preset fusion weights. For example, step (106.1), "updating the preset fusion weights based on the fusion weight difference to obtain the updated target fusion weights," may include:

[0137] (106.1.1) Obtain the target sampling frequency of the first speech;

[0138] (106.1.2) Determine the fundamental frequency correction step size of the third fundamental frequency value based on the ratio between the third fundamental frequency value and the target sampling frequency;

[0139] (106.1.3) Determine the resonant peak correction step size of the third resonant peak based on the ratio between the third resonant peak value and the target sampling frequency;

[0140] (106.1.4) Based on the fundamental frequency correction step size and the resonant peak correction step size, the preset fusion weight is updated until the difference between the updated preset fusion weight and the fusion weight is less than or equal to the preset weight threshold. The updated preset fusion weight is then determined as the target fusion weight.

[0141] The base frequency correction step size can be the value that the preset fusion weight corresponding to the base frequency value needs to be updated in each update, and is used to update the fusion weight corresponding to the base frequency value.

[0142] Among them, the resonant peak correction step size can be the value that the preset fusion weight corresponding to the resonant peak needs to be updated in each update, and is used to update the fusion weight corresponding to the resonant peak.

[0143] Specifically, after determining that the preset fusion weights need to be updated, the sampling frequency corresponding to the first speech is obtained, i.e., the target sampling frequency. Then, based on the ratio between the third fundamental frequency value corresponding to the third voiceprint feature and the target sampling frequency, the correction step size for updating the preset fusion weights corresponding to the fundamental frequency value is determined. Similarly, based on the ratio between the third resonant peak value corresponding to the third voiceprint feature and the target sampling frequency, the correction step size for updating the preset fusion weights corresponding to the resonant peak value is determined. The specific expression is as follows:

[0144] μ0 = 0.03 * F0 / (0.5 * F s )

[0145] μ 01 =0.02*F 01 / (0.5*F s )

[0146] Where μ0 is the fundamental frequency correction step size, μ 01 The step size for resonant correction is F0, where F is the third fundamental frequency value. 01 The third resonance peak, F s The target frequency is used. It's worth noting that since the third resonance peak consists of four resonance peaks, the specific expression for the correction step size corresponding to the third resonance peak can be refined as follows:

[0147] μ1=0.02*F1 / (0.5*F s )

[0148] μ2=0.03*F2 / (0.5*F s )

[0149] μ3=0.04*F3 / (0.5*F s )

[0150] μ4 = 0.06 * F4 / (0.5 * F s );

[0151] Where F1, F2, F, and F4 are the four resonance peaks corresponding to the third resonance peak, and μ1, μ2, μ3, and μ4 are the resonance peak correction step sizes corresponding to the four resonance peaks, F s Frequency is used for the target.

[0152] Furthermore, after obtaining the fundamental frequency correction step size and formant correction step size, the preset fusion weights are updated using addition and subtraction. The updated fusion weights are the target fusion weights. Specifically, first, based on the obtained fundamental frequency correction step size and formant correction step size, the weight correction values ​​for the fundamental frequency and formant peak values ​​are updated. If the previously obtained fusion weight difference is less than or equal to 0, the current weight correction value (initially 0) is added to the corresponding correction step size to obtain the corresponding weight correction value. Then, the preset fusion weight corresponding to the first voiceprint feature is added to the weight correction value, and the preset fusion weight corresponding to the second voiceprint feature is subtracted by the same weight correction value to obtain the target fusion weight. If the previously obtained fusion weight difference is greater than 0, the current weight correction value (initially 0) is subtracted from the corresponding correction step size to obtain the corresponding weight correction value. Then, the preset fusion weight corresponding to the first voiceprint feature is added to the weight correction value, and the preset fusion weight corresponding to the second voiceprint feature is subtracted by the same weight correction value to obtain the target fusion weight. When the fusion weight difference is less than or equal to 0, the specific expression for the weight correction value is:

[0153] δ0=δ0+μ0

[0154] δ1=δ1+μ1

[0155] δ2=δ2+μ2

[0156] δ3=δ3+μ3

[0157] δ4 = δ4 + μ4;

[0158] When the difference in fusion weights is less than or equal to 0, update the weight correction value:

[0159] δ0=δ0-μ0

[0160] δ1=δ1-μ1

[0161] δ2=δ2-μ2

[0162] δ3=δ3-μ3

[0163] δ4 = δ4 - μ4;

[0164] Where μ1, μ2, μ3, and μ4 are the resonant peak correction step sizes corresponding to the four resonant peaks, δ1, δ2, δ3, and δ4 are the weight correction values ​​of the four resonant peaks (initially 0), and δ0 is the weight correction value of the fundamental frequency value. Using the weight correction values, the preset fusion weights of the fundamental frequency value and the resonant peak value are updated respectively. Taking the fusion weight update of the fundamental frequency as an example: a+δ0>0, b-δ0>0, (a+δ0)+(b-δ0)=100, where a+δ0 is the target fusion weight of the updated first fundamental frequency value, and b-δ0 is the target fusion weight of the updated second fundamental frequency value.

[0165] If the difference between the updated fusion weights and the associated fusion weights is less than or equal to the preset weight threshold, the updated preset fusion weights will be determined as the target fusion weights that meet the fusion expectations.

[0166] The above method can be used to update the preset fusion weight by the fusion weight difference, obtain the updated target fusion weight, and then fuse the first voiceprint feature and the second voiceprint feature according to the target fusion weight to obtain the target voiceprint feature, which is finally converted into fused speech. This achieves the effect of merging multiple sounds into one sound while preserving the sound features, reducing the impact of mixing on the user's auditory experience and improving the user's experience.

[0167] By implementing any one or a combination of implementation methods in the embodiments of this application, the application scenarios of the voice processing process can be realized.

[0168] As can be seen from the above, the embodiments of this application can extract the first voiceprint feature of the first speech and the second voiceprint feature of the second speech, then obtain the preset fusion weight between the first speech and the second speech, and fuse the first voiceprint feature and the second voiceprint feature according to the preset fusion weight to obtain the third voiceprint feature. Then, the first feature similarity between the third voiceprint feature and the first voiceprint feature is determined, and the second feature similarity between the third voiceprint feature and the second voiceprint feature is determined. Then, the similarity ratio between the first feature similarity and the second feature similarity is calculated, and the fusion weight difference between the similarity ratio and the preset fusion weight is determined. Finally, when the fusion weight difference is less than or equal to the preset weight threshold, the fused speech is generated according to the third voiceprint feature. Therefore, this solution first extracts the first and second voices that need to be fused, then extracts the voice features of the first and second voices, namely the corresponding first and second voiceprint features. After weighting the parameters of the first and second voiceprint features, a third voiceprint feature parameter is obtained. Then, the first speech is transformed in both the frequency domain and the time domain using the parameters of the third voiceprint feature to obtain the fused speech. In this way, multiple voices are fused into one voice while preserving the voice features, reducing the impact of mixing on the user's auditory experience and improving the user's experience.

[0169] Based on the method described in the above embodiments, the following examples will provide further detailed explanations.

[0170] This application uses a speech processing device as an example to further describe the speech processing method provided in this application. Wherein, Figure 3 This is a schematic flowchart of another step in the speech processing method provided in this application embodiment. For ease of understanding, this application embodiment combines... Figure 3 Describe it.

[0171] In this embodiment, the description will focus on a voice processing device, which can be integrated into a computer device such as an in-vehicle terminal. When the processor on the in-vehicle terminal executes the program instructions corresponding to the data transmission method, the specific flow of the voice processing method is as follows:

[0172] 201. The server performs frame-by-frame processing on the spectrum information corresponding to the first speech, and calculates the fundamental frequency value and formant parameters of each frame of the first speech segment obtained by the frame-by-frame processing to obtain the first voiceprint feature.

[0173] The first voice can be the sound produced by the user, which can be acquired using a sound acquisition device. For example, when a user sings in a karaoke room, the user's voice can be recorded using a microphone device.

[0174] Among them, the spectrum information can indicate which frequency sine waves a speech signal is composed of, and can also show information such as the magnitude and phase of each frequency sine wave.

[0175] The first voiceprint feature can be a visualized sound feature exhibited by the speech characteristics in the first speech, including but not limited to changes in the intensity, wavelength, frequency, and rhythm of the electrical signal converted from changes in sound waves.

[0176] Specifically, after the server acquires the first and second speech samples to be fused, it performs a Fourier transform on the first speech signal to obtain a spectrogram of the first speech. The resulting spectrogram is plotted with amplitude and phase as the vertical axis and frequency as the horizontal axis. Then, based on the obtained spectrogram, the server performs frame-segmentation processing on the corresponding spectral information in the spectrogram, dividing the speech into multiple average speech segments, i.e., the first speech segments. The server then calculates the fundamental frequency value and formant parameters for each frame of the first speech segment obtained from the frame-segmentation processing to obtain the first voiceprint feature. The methods for calculating the fundamental frequency include, but are not limited to, the cepstral method, the autocorrelation method, and LPC (linear prediction method). For example, taking the cepstral method, since the first speech is obtained by filtering the vocal tract response after glottal pulse excitation, and the first speech and vocal tract response are relatively separated in the cepstral domain (i.e., the acoustic pulse cepstral containing fundamental frequency information can be separated from the vocal tract response cepstral), the glottal pulse excitation and vocal tract response can be separated from the cepstral domain to obtain the cepstral. The maximum value of the cepstral function is then found within the cepstral frequency range. The number of sample points corresponding to the maximum value of the cepstral function is the fundamental period of the first speech signal in the current frame, thus obtaining the fundamental frequency value. The formant parameters of the first speech can be obtained using methods such as the cepstral method and LPC (linear prediction method). The fundamental frequency value and formants constitute the first voiceprint feature.

[0177] 202. The server performs frame-by-frame processing on the spectrum information corresponding to the acquired second speech, and calculates the fundamental frequency value and formant parameters for each frame of the second speech segment obtained by the frame-by-frame processing to obtain the second voiceprint feature.

[0178] The second voice can be another voice that exists synchronously with the user's voice, such as the original voice of the singer when the user is singing.

[0179] The second voiceprint feature can be a visual sound feature exhibited by the speech characteristics in the second speech, including but not limited to changes in the intensity, wavelength, frequency, and rhythm of the electrical signal converted from changes in sound waves.

[0180] Specifically, after the server obtains the first and second speech samples to be fused, it can perform a Fourier transform on the second speech signal to obtain the spectrum of the second speech. The obtained spectrum will be a graph with amplitude and phase as the vertical axis and frequency as the horizontal axis. Then, based on the obtained spectrum, the corresponding spectral information in the spectrum is processed by frame segmentation to divide the speech into multiple average speech segments, i.e., the second speech segments. The fundamental frequency value and formant parameters of each frame of the second speech segment obtained by frame segmentation are calculated to obtain the second voiceprint feature. The methods for calculating the fundamental frequency include, but are not limited to, cepstral method, autocorrelation method, LPC (linear prediction method), etc. For example, taking the cepstral method, since the second speech is obtained by filtering the vocal tract response after glottal pulse excitation, and the second speech and vocal tract response are relatively separated in the cepstral domain (i.e., the acoustic pulse cepstral containing fundamental frequency information can be separated from the vocal tract response cepstral), the glottal pulse excitation and vocal tract response can be separated from the cepstral domain to obtain the cepstral. The maximum value of the cepstral function is then found within the cepstral frequency range. The number of sample points corresponding to the maximum value of the cepstral function is the fundamental period of the second speech signal in the current frame, thus obtaining the fundamental frequency value. The formant parameters of the second speech can be obtained using methods such as the cepstral method and LPC (linear prediction method). The fundamental frequency value and formants constitute the second voiceprint feature.

[0181] 203. The server reads the first fundamental frequency value and the first resonance peak value from the first voiceprint feature, and reads the second fundamental frequency value and the second resonance peak value from the second voiceprint feature.

[0182] The first fundamental frequency value can be the first fundamental tone frequency, which is a parameter value extracted from the fundamental frequency value of the first speech. It is also the most important criterion for speech recognition. The first fundamental frequency reflects the frequency of vocal cord vibration when the first speech is produced.

[0183] The first resonance peak can be a parameter value extracted from the first speech resonance peak parameter. It refers to some regions in the speech spectrum where energy is relatively concentrated. The resonance peak is not only a determining factor of sound quality, but also reflects the physical characteristics of the vocal tract (resonance cavity).

[0184] The second fundamental frequency value can be the second fundamental tone frequency, which is a parameter value extracted from the fundamental frequency value of the second speech. It is also the most important discrimination criterion for speech recognition. The second fundamental frequency reflects the frequency of vocal cord vibration when the second speech is pronounced.

[0185] The second resonance peak can be a parameter value extracted from the second speech resonance peak parameter. It refers to some regions in the speech spectrum where energy is relatively concentrated. The resonance peak is not only a determining factor of sound quality, but also reflects the physical characteristics of the vocal tract (resonance cavity).

[0186] Specifically, after obtaining the first voiceprint features and the second voiceprint features corresponding to the first and second speech, the server reads the fundamental frequency value and formant parameters in the frequency domain data corresponding to the first and second voiceprint features. The specific methods for obtaining the fundamental frequency value include, but are not limited to, cepstral method, autocorrelation method, LPC (linear prediction method), etc. The specific methods for obtaining the formant parameters include, but are not limited to, cepstral method, LPC (linear prediction method), etc., thereby determining the first fundamental frequency value and the first formant peak value of the first voiceprint feature, and the second fundamental frequency value and the second formant peak value of the second voiceprint feature.

[0187] 204. The server performs a weighted calculation between the first fundamental frequency value and the second fundamental frequency value according to the preset fusion weight to obtain the third fundamental frequency value, and transforms the first resonance peak value and the second resonance peak value to obtain the third resonance peak value.

[0188] The third fundamental frequency value can be the second fundamental frequency, which is the fundamental frequency obtained by weighting the first and second fundamental frequency values. The third fundamental frequency reflects the vocal cord vibration frequency during fused speech pronunciation.

[0189] The third resonant peak can be obtained by weighting the first and second resonant peaks. It refers to some regions in the speech spectrum where energy is relatively concentrated. The resonant peaks are not only a determining factor of sound quality, but also reflect the physical characteristics of the vocal tract (resonance cavity).

[0190] Specifically, the server uses preset fusion weights to calculate the first and second fundamental frequency values ​​to obtain the third fundamental frequency value. The specific expression for calculating the fundamental frequency value is: F0 = a% * F o0 +b%*F r0 Where a>0, b>0, a+b=100, F0 is the third fundamental frequency value, a is the preset fusion weight of the first voiceprint feature, and F o0 Let b be the first fundamental frequency value, b be the preset fusion weight of the first voiceprint feature, and F be the second fundamental frequency value. r0This is the second fundamental frequency value. Similarly, using preset fusion weights, the first and second resonance peak values ​​are calculated to obtain the third resonance peak value. The specific expression for calculating the resonance peak value is: F1 = a * F o1 +b%*F r1 Where a>0, b>0, a+b=100, F1 is the third resonance peak, a is the preset fusion weight of the first voiceprint feature, and F o1 Let b be the first resonance peak value, b be the preset fusion weight of the first voiceprint feature, and F be the second resonance peak value. r1 This is the second resonance peak.

[0191] It's worth noting that since a speech particle can have multiple formants, these formants together determine the timbre of the speech. For example, a user's speech has the following characteristics: the degree of tongue opening (also called tongue position), the front-back position of the tongue, and the rounding of the lips. These features are basically sufficient to describe the vowel timbre of a speech particle, and they are closely related to the values ​​of the formants.

[0192] The first resonance peak consists of four resonance peaks, meaning the first resonance peak corresponds to four resonance peaks. Similarly, the second resonance peak also includes four resonance peaks, and the third resonance peak also has four resonance peaks.

[0193] 205. The server determines the third voiceprint feature based on the third fundamental frequency and the third resonant peak.

[0194] Among them, the third voiceprint feature can be the visualized sound features shown by the speech characteristics in the fused speech, including but not limited to the changes in the intensity, wavelength, frequency, rhythm, etc. of the electrical signals converted from changes in sound waves.

[0195] Specifically, the server obtains the first voiceprint feature and the fusion weights when fusing the second voiceprint feature, i.e., the preset fusion weights. The fused voiceprint feature is the third voiceprint feature. It is worth noting that the speech features corresponding to the third voiceprint feature will be closer to the voiceprint feature with the larger preset fusion weight. For example, when user A sings in a karaoke bar with the original vocals enabled, user A would prefer the fused voice to be closer to their own voice. Therefore, the weight of user A's own voice will be greater than the weight of the original vocals.

[0196] 206. The server calculates the first feature similarity between the third voiceprint feature and the first voiceprint feature, and determines the second feature similarity between the third voiceprint feature and the second voiceprint feature.

[0197] The first feature similarity can be the degree of difference between the third voiceprint feature and the first voiceprint feature after voiceprint comparison, including but not limited to the similarity of intensity, wavelength, frequency, and rhythm changes.

[0198] The second feature similarity can be the degree of difference between the third voiceprint feature and the second voiceprint feature after voiceprint comparison, including but not limited to the similarity of intensity, wavelength, frequency, and rhythm changes.

[0199] Specifically, after fusing the first voiceprint feature of the first speech and the second voiceprint feature of the second speech, the server obtains the third voiceprint feature. Then, it can use voiceprint comparison or speech similarity scoring methods to compare the third voiceprint feature with the first and second voiceprint features respectively, and score them in a score system to determine the similarity, namely the first feature similarity and the second feature similarity. For example, when using a speech similarity scoring algorithm to compare the third voiceprint feature and the first voiceprint feature, the frequency domain shaping of the third voiceprint feature and the first voiceprint feature can be performed first, including level adjustment and filtering. Then, the third voiceprint feature and the first voiceprint feature can be time-converted for auditory conversion. After perturbation and distortion calculations are performed on the converted speech, asymmetric processing is applied, and the similarity can be scored to obtain the first feature similarity. The method for obtaining the second feature similarity can also be the same.

[0200] 207. The server calculates the similarity ratio between the first feature similarity and the second feature similarity, and determines the fusion weight difference between the similarity ratio and the preset fusion weight.

[0201] Among them, the fusion weight difference can be the difference between the similarity ratio and the preset fusion weight, which is one of the judgment conditions for determining whether the third voiceprint feature has achieved the expected fusion.

[0202] Specifically, after determining the similarity between the third voiceprint feature and the first and second voiceprint features respectively, the server calculates the similarity ratio between the first and second features using the first feature similarity as the denominator and the second feature similarity as the numerator. Then, it calculates the fusion weight ratio using the preset fusion weight value of the first voiceprint feature as the denominator and the preset fusion weight of the second voiceprint feature as the numerator. The difference between the similarity ratio and the preset fusion weight is the fusion weight difference between the similarity ratio and the preset fusion weight. This fusion weight difference can be used to determine whether the fused third voiceprint feature has reached the expected weight ratio.

[0203] 208. When the server determines that the difference in fusion weights is less than the preset weight threshold, it updates the spectral information of the first speech based on the third fundamental frequency value and the third resonant peak value in the third voiceprint feature to obtain the target spectral information.

[0204] The preset weight threshold can be a fusion weight ratio calculated by using the preset fusion weight value of the first voiceprint feature as the denominator and the preset fusion weight of the second voiceprint feature as the numerator. This ratio is one of the judgment conditions for determining whether the third voiceprint feature has reached the expected fusion.

[0205] Specifically, after determining that the difference in fusion weights is less than or equal to a preset weight threshold, the server reads the spectral information of the first speech based on the audio of the first speech. Then, using the frequency domain data corresponding to the third voiceprint feature, namely the third fundamental frequency and the third resonant peak, the server replaces and adjusts the frequency domain data in the first speech to obtain the target spectral information.

[0206] 209. The server performs time-domain conversion on the target spectrum information to obtain the fused speech.

[0207] Among them, fused speech can be speech that combines and retains the sound features of the first speech and the second speech.

[0208] Specifically, after the server obtains the target spectrum information, since signal processing involves two domains, one called the time domain and the other the frequency domain, playing audio uses time domain data, while processing audio usually uses frequency data. Therefore, after replacing and adjusting the frequency domain data in the first speech, it is necessary to convert the frequency domain data to the time domain. The converted frequency domain data can be converted back to the time domain using Fourier transform to obtain the fused speech.

[0209] Through the above application scenarios, the following effects can be achieved: If the server detects that there are multiple sounds in the current space, it can extract the sound features of different sounds, and merge multiple sounds into one sound while preserving the sound features, thereby reducing the impact of mixing on the user's auditory experience and improving the user's experience.

[0210] Therefore, this solution first extracts the first and second voices that need to be fused, then extracts the voice features of the first and second voices, namely the corresponding first and second voiceprint features. After weighting the parameters of the first and second voiceprint features, a third voiceprint feature parameter is obtained. Then, the first speech is transformed in both the frequency domain and the time domain using the parameters of the third voiceprint feature to obtain the fused speech. In this way, multiple voices are fused into one voice while preserving the voice features, reducing the impact of mixing on the user's auditory experience and improving the user's experience.

[0211] To better implement the above methods, this application also provides a voice processing device that can be integrated into computer equipment, such as a voice terminal or other computer equipment.

[0212] For example, such as Figure 4 As shown, the speech processing device may include an extraction unit 301, a fusion unit 302, a determination unit 303, a calculation unit 304, and a conversion unit 305.

[0213] Extraction unit 301 is used to extract the first voiceprint feature of the first speech and the second voiceprint feature of the second speech;

[0214] The fusion unit 302 is used to obtain a preset fusion weight between the first speech and the second speech, and to fuse the first voiceprint feature and the second voiceprint feature according to the preset fusion weight to obtain the third voiceprint feature.

[0215] The determining unit 303 is used to determine the first feature similarity between the third voiceprint feature and the first voiceprint feature, and to determine the second feature similarity between the third voiceprint feature and the second voiceprint feature;

[0216] The calculation unit 304 is used to calculate the similarity ratio between the first feature similarity and the second feature similarity, and to determine the fusion weight difference between the similarity ratio and the preset fusion weight.

[0217] The conversion unit 305 is used to generate fused speech based on the third voiceprint feature when the fusion weight difference is less than or equal to a preset weight threshold.

[0218] In some embodiments, the extraction unit 301 is further configured to:

[0219] The spectral information corresponding to the first speech is processed by frame segmentation, and the fundamental frequency value and formant parameters of each frame of the first speech segment obtained by frame segmentation are calculated to obtain the first voiceprint feature;

[0220] The spectral information corresponding to the second speech is processed by frame segmentation, and the fundamental frequency value and formant parameters of each frame of the second speech segment obtained by frame segmentation are calculated to obtain the second voiceprint feature.

[0221] In some embodiments, the fusion unit 302 is further configured to:

[0222] Read the first fundamental frequency value and the first resonance peak value in the first voiceprint feature, and read the second fundamental frequency value and the second resonance peak value in the second voiceprint feature;

[0223] Based on the preset fusion weights, the first fundamental frequency value and the second fundamental frequency value are weighted and calculated to obtain the third fundamental frequency value;

[0224] Based on the preset fusion weights, the first resonance peak and the second resonance peak are transformed to obtain the third resonance peak;

[0225] The third voiceprint feature is determined based on the third fundamental frequency and the third resonance peak.

[0226] In some embodiments, the conversion unit 305 is further configured to:

[0227] Read the spectrum information corresponding to the first speech;

[0228] The first fundamental frequency value and the first resonant peak value in the spectrum information are updated according to the third voiceprint feature to obtain the updated target spectrum information;

[0229] The target spectral information is converted in the time domain to obtain the fused speech.

[0230] In some embodiments, the voice processing apparatus further includes an update unit for:

[0231] If the difference in fusion weights is greater than the preset weight threshold, the preset fusion weights are updated based on the difference in fusion weights to obtain the updated target fusion weights.

[0232] The first and second voiceprint features are fused according to the target fusion weights to obtain the target voiceprint features;

[0233] Generate fused speech based on the target voiceprint features.

[0234] In some embodiments, the updating unit is further configured to:

[0235] Obtain the target sampling frequency of the first speech;

[0236] The fundamental frequency correction step size of the third fundamental frequency value is determined based on the ratio between the third fundamental frequency value and the target sampling frequency.

[0237] The resonant peak correction step size of the third resonant peak is determined based on the ratio between the third resonant peak value and the target sampling frequency.

[0238] Based on the fundamental frequency correction step size and the resonant peak correction step size, the preset fusion weights are updated until the difference between the updated preset fusion weights and the fusion weights associated with them is less than or equal to the preset weight threshold. The updated preset fusion weights are then determined as the target fusion weights.

[0239] In some embodiments, the voice processing apparatus further includes a sampling unit for:

[0240] If the first sampling frequency is equal to the second sampling frequency, then the first sampling frequency is determined as the target sampling frequency, and speech is sampled according to the target sampling frequency to obtain the first speech and the second speech.

[0241] If the first sampling frequency is greater than the second sampling frequency, then the second sampling frequency is determined as the target sampling frequency, and speech is sampled according to the target sampling frequency to obtain the first speech and the second speech.

[0242] As can be seen from the above, the embodiments of this application can first extract the first and second voices that need to be fused, then extract the sound features of the first and second voices, namely the corresponding first voiceprint features and second voiceprint features. After weighted calculation of the parameters of the first and second voiceprint features, the third voiceprint feature parameters are obtained. Then, the first speech is converted in the frequency domain and in the time domain using the parameters of the third voiceprint features to obtain the fused speech. In this way, multiple voices are fused into one voice while retaining the sound features, reducing the impact of mixing on the user's auditory effect and improving the user's experience.

[0243] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0244] This application also provides a computer device, such as... Figure 5 As shown, it illustrates a structural schematic diagram of the computer device involved in the embodiments of this application, specifically:

[0245] The computer device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will understand that... Figure 5 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0246] The processor 401 is the control center of the computer device. It connects various parts of the computer device via various interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in the memory 402, and by calling data stored in the memory 402, thereby providing overall monitoring of the computer device. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 401.

[0247] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and voice processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as voice playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0248] The computer device also includes a power supply 403 that supplies power to the various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0249] The computer device may also include an input unit 404, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0250] Although not shown, the computer device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the computer device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402 to realize various functions, as follows:

[0251] The process involves extracting the first voiceprint feature of the first speech and the second voiceprint feature of the second speech, then obtaining the preset fusion weight between the first and second speech, and fusing the first and second voiceprint features according to the preset fusion weight to obtain the third voiceprint feature. Next, the first feature similarity between the third and first voiceprint features is determined, as well as the second feature similarity between the third and second voiceprint features. Then, the similarity ratio between the first and second feature similarities is calculated, and the fusion weight difference between the similarity ratio and the preset fusion weight is determined. Finally, when the fusion weight difference is less than or equal to the preset weight threshold, fused speech is generated based on the third voiceprint feature.

[0252] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0253] As can be seen from the above, the embodiments of this application can first extract the first and second voices that need to be fused, then extract the sound features of the first and second voices, namely the corresponding first voiceprint features and second voiceprint features. After weighted calculation of the parameters of the first and second voiceprint features, the third voiceprint feature parameters are obtained. Then, the first speech is converted in the frequency domain and in the time domain using the parameters of the third voiceprint features to obtain the fused speech. In this way, multiple voices are fused into one voice while retaining the sound features, reducing the impact of mixing on the user's auditory effect and improving the user's experience.

[0254] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0255] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the speech processing methods provided in embodiments of this application. For example, the instructions can execute the following steps:

[0256] The process involves extracting the first voiceprint feature of the first speech and the second voiceprint feature of the second speech, then obtaining the preset fusion weight between the first and second speech, and fusing the first and second voiceprint features according to the preset fusion weight to obtain the third voiceprint feature. Next, the first feature similarity between the third and first voiceprint features is determined, as well as the second feature similarity between the third and second voiceprint features. Then, the similarity ratio between the first and second feature similarities is calculated, and the fusion weight difference between the similarity ratio and the preset fusion weight is determined. Finally, when the fusion weight difference is less than or equal to the preset weight threshold, fused speech is generated based on the third voiceprint feature.

[0257] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0258] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0259] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the speech processing methods provided in the various optional implementations of the above embodiments.

[0260] Since the instructions stored in the computer-readable storage medium can execute the steps of any of the speech processing methods provided in the embodiments of this application, the beneficial effects that any of the speech processing methods provided in the embodiments of this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0261] The above provides a detailed description of a speech processing method, apparatus, device, and computer-readable storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A speech processing method, characterized in that, include: Extract the first voiceprint feature of the first speech and the second voiceprint feature of the second speech; Obtain a preset fusion weight between the first speech and the second speech, and fuse the first voiceprint feature and the second voiceprint feature according to the preset fusion weight to obtain the third voiceprint feature; Determine a first feature similarity between the third voiceprint feature and the first voiceprint feature, and determine a second feature similarity between the third voiceprint feature and the second voiceprint feature; Calculate the similarity ratio between the first feature similarity and the second feature similarity, and determine the fusion weight difference between the similarity ratio and the preset fusion weight; When the difference in fusion weights is less than or equal to a preset weight threshold, fused speech is generated based on the third voiceprint feature.

2. The method according to claim 1, characterized in that, The extraction of the first voiceprint feature of the first speech and the second voiceprint feature of the second speech includes: The spectral information corresponding to the first speech is processed by frame segmentation, and the fundamental frequency value and formant parameters of each frame of the first speech segment obtained by frame segmentation are calculated to obtain the first voiceprint feature. The spectral information corresponding to the second speech is processed by frame segmentation, and the fundamental frequency value and formant parameters of each frame of the second speech segment obtained by frame segmentation are calculated to obtain the second voiceprint feature.

3. The method according to claim 1, characterized in that, The process of fusing the first and second voiceprint features according to the preset fusion weights to obtain the third voiceprint feature includes: Read the first fundamental frequency value and the first resonance peak value from the first voiceprint feature, and read the second fundamental frequency value and the second resonance peak value from the second voiceprint feature; According to the preset fusion weight, a weighted calculation is performed between the first fundamental frequency value and the second fundamental frequency value to obtain the third fundamental frequency value; According to the preset fusion weight, the first resonance peak and the second resonance peak are transformed to obtain the third resonance peak; The third voiceprint feature is determined based on the third fundamental frequency value and the third resonance peak value.

4. The method according to claim 1, characterized in that, The conversion of the third voiceprint feature into fused speech includes: Read the spectrum information corresponding to the first voice; The first fundamental frequency value and the first resonance peak value in the spectrum information are updated according to the third voiceprint feature to obtain the updated target spectrum information; The target spectral information is converted in the time domain to obtain fused speech.

5. The method according to claim 3, characterized in that, Also includes: If the difference in fusion weights is greater than a preset weight threshold, the preset fusion weights are updated based on the difference in fusion weights to obtain the updated target fusion weights. The first voiceprint feature and the second voiceprint feature are fused according to the target fusion weight to obtain the target voiceprint feature; Generate fused speech based on the target voiceprint features.

6. The method according to claim 5, characterized in that, The step of updating the preset fusion weights based on the fusion weight difference to obtain the updated target fusion weights includes: Obtain the target sampling frequency of the first speech; The fundamental frequency correction step size of the third fundamental frequency value is determined based on the ratio between the third fundamental frequency value and the target sampling frequency; The resonant peak correction step size of the third resonant peak is determined based on the ratio between the third resonant peak value and the target sampling frequency. Based on the fundamental frequency correction step size and the resonant peak correction step size, the preset fusion weight is updated until the difference between the updated preset fusion weight and the fusion weight is less than or equal to the preset weight threshold. The updated preset fusion weight is then determined as the target fusion weight.

7. The method according to any one of claims 1 to 6, characterized in that, Before extracting the first voiceprint feature of the first speech and the second voiceprint feature of the second speech, the method further includes: Obtain the first sampling frequency of the first speech and the second sampling frequency of the second speech; If the first sampling frequency is equal to the second sampling frequency, then the first sampling frequency is determined as the target sampling frequency, and speech is sampled according to the target sampling frequency to obtain the first speech and the second speech. If the first sampling frequency is greater than the second sampling frequency, then the second sampling frequency is determined as the target sampling frequency, and speech is sampled according to the target sampling frequency to obtain the first speech and the second speech.

8. A voice processing device, characterized in that, include: An extraction unit is used to extract the first voiceprint feature of the first speech and the second voiceprint feature of the second speech. A fusion unit is used to obtain a preset fusion weight between the first speech and the second speech, and to fuse the first voiceprint feature and the second voiceprint feature according to the preset fusion weight to obtain a third voiceprint feature. A determining unit is configured to determine a first feature similarity between the third voiceprint feature and the first voiceprint feature, and to determine a second feature similarity between the third voiceprint feature and the second voiceprint feature; The calculation unit is used to calculate the similarity ratio between the first feature similarity and the second feature similarity, and to determine the fusion weight difference between the similarity ratio and the preset fusion weight; The conversion unit is used to generate fused speech based on the third voiceprint feature when the fusion weight difference is less than or equal to a preset weight threshold.

9. A computer device, characterized in that, It includes a processor and a memory, the memory storing a computer program, and the processor running the computer program in the memory to implement the steps of the speech processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is computer-readable and stores a plurality of instructions adapted for loading by a processor to perform the steps of the speech processing method according to any one of claims 1 to 7.

Images