Speech synthesis method and device with adjustable spectrum intensity, computer equipment and medium

Abstract

The invention is suitable for the field of artificial intelligence, and particularly relates to a frequency spectrum intensity adjustable speech synthesis method and device, computer equipment and a medium. The method comprises the following steps of: randomly disturbing the frequencies of K formants in an original frequency spectrum to obtain a disturbance frequency of the corresponding formant, replacing the frequency of the disturbance frequency in the original frequency spectrum to obtain a disturbance frequency spectrum, coding to obtain a disturbance frequency spectrum characteristic, calculating the correlation between the disturbance frequency spectrum characteristic and N preset intensity matrixes, and determining the weight of the corresponding intensity matrix, performing weighted summation on the N intensity matrixes according to the weights to obtain intensity features, performing feature fusion on the intensity features and disturbance spectrum features to obtain spectrum intensity features, obtaining target speech according to the spectrum intensity features, a target text and target speech features, and disturbing formants to destroy speaker information represented in an original spectrum. Effective emotion and intensity information is reserved, the weight of the intensity matrix is calculated, and the accuracy of the emotion intensity in the target voice is improved.

Description

Technical Field

[0001] The present invention is applicable to the field of artificial intelligence, and in particular relates to a speech synthesis method, device, computer equipment and medium with adjustable spectrum intensity. Background Art

[0002] The goal of speech synthesis technology is to convert text information into speech signals that are understandable, clear, natural, expressive, and close to human speech. Since human speech carries linguistic and emotional information, modeling emotional information is crucial for synthesizing natural and expressive speech.

[0003] In the current research on emotional speech synthesis, most of the existing models learn the average emotional characteristics of all speakers in the dataset. However, in general, the emotional intensity of each speaker when using the same emotion to express the same meaning is often different. The existing models are unable to capture this difference information, resulting in low reliability of the emotional intensity information and low emotional accuracy of the synthesized speech.

[0004] Therefore, in the field of speech synthesis technology, how to improve the emotional accuracy of synthesized speech has become an urgent problem to be solved. Summary of the invention

[0005] In view of this, the embodiments of the present invention provide a speech synthesis method, apparatus, computer equipment and medium with adjustable spectral intensity to solve the problem that the reliability of emotion intensity information in the current emotional speech synthesis research is low, resulting in low emotion accuracy of the synthesized speech.

[0006] In a first aspect, an embodiment of the present invention provides a speech synthesis method with adjustable spectrum intensity, the speech synthesis method comprising: Acquire an original spectrum of the original speech, determine K formants in the original spectrum, randomly perturb the frequencies of the K formants to obtain perturbation frequencies of the corresponding formants, and use the perturbation frequency of each formant to replace the frequency of the corresponding formant in the original spectrum to obtain a perturbation spectrum, where K is an integer greater than 0; Performing feature encoding on the disturbance spectrum to obtain disturbance spectrum features, calculating the correlation between the disturbance spectrum features and N preset intensity matrices, and determining the weight of the corresponding intensity matrix according to each correlation, where N is an integer greater than 0; Performing weighted summation on the N intensity matrices according to weights corresponding to the N intensity matrices to obtain an intensity feature, and performing feature fusion on the intensity feature and the disturbance spectrum feature to obtain a spectrum intensity feature; The target text to be synthesized and the target speech features are obtained, speech synthesis is performed on the target text according to the spectrum intensity features, and the synthesized speech is adjusted using the target speech features to obtain the corresponding target speech.

[0007] In a second aspect, an embodiment of the present invention provides a speech synthesis device with adjustable spectrum intensity, the speech synthesis device comprising: A frequency perturbation module is used to obtain an original spectrum of the original speech, determine K resonance peaks in the original spectrum, randomly perturb the frequencies of the K resonance peaks to obtain perturbation frequencies of the corresponding resonance peaks, and use the perturbation frequency of each resonance peak to replace the frequency of the corresponding resonance peak in the original spectrum to obtain a perturbation spectrum, where K is an integer greater than 0; A weight determination module is used to perform feature encoding on the disturbance spectrum to obtain disturbance spectrum features, calculate the correlation between the disturbance spectrum features and N preset intensity matrices, and determine the weight of the corresponding intensity matrix according to each correlation, where N is an integer greater than 0; A feature fusion module is used to perform weighted summation on the N intensity matrices according to the weights corresponding to the N intensity matrices to obtain an intensity feature, and perform feature fusion on the intensity feature and the disturbance spectrum feature to obtain a spectrum intensity feature; The speech synthesis module is used to obtain the target text to be synthesized and the target speech features, perform speech synthesis on the target text according to the spectrum intensity features, and adjust the synthesized speech using the target speech features to obtain the corresponding target speech.

[0008] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the speech synthesis method as described in the first aspect when executing the computer program.

[0009] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the speech synthesis method as described in the first aspect is implemented.

[0010] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: the embodiments of the present invention obtain an original spectrum of an original speech, determine K resonance peaks in the original spectrum, randomly perturb the frequencies of the K resonance peaks to obtain the perturbation frequencies of the corresponding resonance peaks, use the perturbation frequency of each resonance peak to replace the frequency of the corresponding resonance peak in the original spectrum to obtain a perturbation spectrum, perform feature encoding on the perturbation spectrum to obtain perturbation spectrum features, calculate the correlation between the perturbation spectrum features and N preset intensity matrices, determine the weight of the corresponding intensity matrix according to each correlation, perform weighted summation on the N intensity matrices according to the weights corresponding to the N intensity matrices to obtain an intensity feature, perform feature fusion on the intensity feature and the perturbation spectrum feature to obtain a spectrum intensity feature, perform feature fusion on the intensity feature and the perturbation spectrum feature The dynamic spectrum features are fused to obtain the spectrum intensity features, and the target text and target speech features to be synthesized are obtained. According to the spectrum intensity features, the target text is synthesized, and the synthesized speech is adjusted using the target speech features to obtain the corresponding target speech. The speaker information represented in the original spectrum is destroyed by perturbing the resonance peaks, and only the effective emotional information and the intensity information of the emotion in the original spectrum are retained. The weight of the corresponding intensity matrix is ​​determined by calculating the correlation between the perturbed spectrum features and the preset N intensity matrices, which improves the accuracy of the emotional intensity in the target speech. In the intelligent customer service scenario in the financial field, a natural and expressive target speech is synthesized for the robot customer service, which improves the service quality and efficiency of the financial business. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0012] Figure 1 This is a schematic diagram of an application environment of a speech synthesis method with adjustable spectrum intensity provided in Embodiment 1 of the present invention; Figure 2 It is a flowchart of a speech synthesis method with adjustable spectrum intensity provided in Embodiment 1 of the present invention; Figure 3 It is a structural schematic diagram of a speech synthesis device with adjustable spectrum intensity provided in Embodiment 2 of the present invention; Figure 4 It is a structural diagram of a computer device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0013] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0014] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0015] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0016] As used in the present specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]", depending on the context.

[0017] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0018] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0019] The embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0020] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0021] It should be understood that the order of execution of the steps in the following embodiments does not imply a precedence of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0022] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.

[0023] A speech synthesis method with adjustable spectrum intensity provided in the first embodiment of the present invention can be applied in the following aspects: Figure 1 In an application environment, the client communicates with the server. The client includes but is not limited to PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud computing devices, personal digital assistants (PDAs), and other computer devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers. The speech synthesis method can be applied to multiple fields such as the Internet, finance, medical care, education, and transportation. For example, due to the complexity and diversity of financial services, a large number of simple tasks such as consulting services and after-sales services will seriously occupy the energy and time of business personnel, reducing the work efficiency and work quality of business personnel. The use of an intelligent conversation method based on automatic speech synthesis can save a lot of labor costs. At the same time, it can provide customers with more natural, expressive, and rich voice customer service services by effectively controlling the content of the voice conversation, so as to improve customer experience and thus improve the service quality of financial services.

[0024] See also Figure 2 , is a flow chart of a speech synthesis method with adjustable spectrum intensity provided in the first embodiment of the present invention. The speech synthesis method can be applied to Figure 1The client in the speech synthesis method may include the following steps: Step S201, obtain the original spectrum of the original speech, determine K resonance peaks in the original spectrum, randomly perturb the frequencies of the K resonance peaks to obtain the perturbation frequencies of the corresponding resonance peaks, and use the perturbation frequency of each resonance peak to replace the frequency of the corresponding resonance peak in the original spectrum to obtain the perturbation spectrum.

[0025] Among them, the goal of speech synthesis technology is to convert text information into speech signals that are understandable, clear, natural, expressive, and close to human speech. Since human speech carries language information and emotional information, modeling emotional information is crucial for synthesizing natural and expressive speech.

[0026] The spectrum of speech is the distribution relationship between the energy and frequency of speech signals in the frequency domain, which can reflect some important speech features. The formant is the location where the sound energy is concentrated, and it is an envelope peak. In the two-dimensional structure of the spectrum, each envelope has a large amplitude harmonic, that is, the peak of the formant. The formant, as the location where the sound energy is concentrated, is an important feature reflecting the resonance characteristics of the vocal tract and represents the most direct source of pronunciation information. People widely use spectrum information and formant information in speech perception. Therefore, spectrum and formant are very important feature parameters in speech signal processing, and are widely used as the main features of speech recognition and basic information for speech coding transmission.

[0027] In current clinical applications, acoustic detection parameters of voice include fundamental frequency, sound intensity, harmonic-to-noise ratio, frequency perturbation, amplitude perturbation, resonance peak, contact rate, contact rate perturbation, contact power, contact power perturbation, etc. The above acoustic detection parameters can be used to objectively acoustically evaluate the speaker's voice. Therefore, when the frequency of the resonance peak in the spectrum changes, the speaker information represented by the spectrum changes accordingly. Therefore, in order to achieve the goal of speech synthesis with controllable emotion intensity, this embodiment randomly perturbs the frequencies of K resonance peaks in the original spectrum of the original speech to destroy the speaker information represented in the original spectrum, and only retains the effective emotion information and emotion intensity information in the original spectrum to improve the reliability of the emotion information and intensity information.

[0028] Specifically, this embodiment randomly perturbs the frequencies of K resonance peaks in the original spectrum of the original speech to obtain the perturbation frequencies of the corresponding resonance peaks, and uses the perturbation frequencies of each resonance peak to replace the frequencies of the corresponding resonance peaks in the original spectrum, so as to destroy the speaker information represented in the original spectrum, and only retain the effective emotional information and the intensity information of the emotion in the original spectrum, and obtain the perturbation spectrum, which is a spectrum that is independent of the speaker but related to the emotion, and can effectively represent the emotional information and the intensity information of the emotion in the original spectrum. Wherein, K is an integer greater than 0. For example, in the intelligent customer service scenario in the financial field, the original speech can be the original speech of manual customer service or robot customer service to communicate with customers, which is used as the synthesis basis of speech with controllable emotion intensity to improve the naturalness and expressiveness of the speech synthesized by the robot customer service, thereby improving the service quality of financial services.

[0029] Optionally, determining K resonance peaks in the original spectrum includes: The spectrum envelope of the original spectrum is obtained, K local peaks in the spectrum envelope are determined, and the K local peaks are determined as K resonance peaks in the original spectrum.

[0030] Among them, the resonance peak information is contained in the spectrum envelope. It is generally believed that the maximum value in the spectrum envelope is the resonance peak. Therefore, this embodiment obtains the spectrum envelope of the original spectrum, and determines K local peaks of the frequency in the spectrum envelope according to the frequency values ​​at various locations in the spectrum envelope, and determines the K local peaks as the K resonance peaks in the original spectrum.

[0031] This embodiment determines K local peaks in the spectrum envelope of the original spectrum as K resonance peaks in the original spectrum, thereby improving the accuracy of resonance peak extraction.

[0032] Optionally, the frequencies of the K resonance peaks are randomly perturbed to obtain the perturbation frequencies of the corresponding resonance peaks, including: For any formant, the original frequency of the formant is determined according to the spectrum of the original speech, an interference frequency is randomly obtained within a preset frequency range, and the sum of the original frequency and the interference frequency is determined as the disturbance frequency of the formant. The preset frequency range is [0, F], where F is the fundamental frequency of the spectrum; Traverse all the resonance peaks and obtain the perturbation frequencies of all the resonance peaks.

[0033] Among them, the fundamental frequency is the frequency of the fundamental tone in the spectrum, which can be regarded as the basic pitch of the sound. Therefore, in this embodiment, when the resonance peak is disturbed, the preset frequency range for randomly taking the interference frequency is set to [0, F], where F is the fundamental frequency of the spectrum.

[0034] For any resonance peak, the original frequency of the resonance peak is determined, and an interference frequency is randomly obtained within a preset frequency range. The sum of the original frequency and the interference frequency is determined as the disturbance frequency of the resonance peak to serve as the frequency basis for disturbing the resonance peak.

[0035] In this embodiment, the preset frequency range for randomly taking the interference frequency is set to [0, F], where F is the fundamental frequency of the spectrum, thereby improving the rationality of disturbing the resonance peak.

[0036] The above steps of obtaining the original spectrum of the original speech, determining K resonance peaks in the original spectrum, randomly perturbing the frequencies of the K resonance peaks to obtain the perturbation frequencies of the corresponding resonance peaks, and using the perturbation frequency of each resonance peak to replace the frequency of the corresponding resonance peak in the original spectrum to obtain the perturbed spectrum, destroy the speaker information represented in the original spectrum by perturbing the resonance peaks, and only retain the effective emotion information and emotion intensity information in the original spectrum, thereby improving the reliability of the emotion information and intensity information.

[0037] Step S202, feature encoding is performed on the disturbance spectrum to obtain disturbance spectrum features, correlations between the disturbance spectrum features and N preset intensity matrices are calculated, and the weight of the corresponding intensity matrix is ​​determined according to each correlation.

[0038] Among them, the preset N intensity matrices can be used to measure the intensity information of the emotion in the corresponding disturbance spectrum characteristics, N is an integer greater than 0, for example, N=3, and 3 intensity matrices S, M and L are preset, S represents the low intensity of the emotion in the corresponding disturbance spectrum characteristics, and is used to synthesize speech with lower emotion intensity; M represents the medium intensity of the emotion in the corresponding disturbance spectrum characteristics, and is used to synthesize speech with medium emotion intensity; L represents the high intensity of the emotion in the corresponding disturbance spectrum characteristics, and is used to synthesize speech with higher emotion intensity.

[0039] In order to further determine the emotional information contained in the disturbance spectrum and improve the accuracy of the emotional intensity in the synthesized speech, this embodiment first performs feature encoding on the disturbance spectrum to obtain the disturbance spectrum feature, and calculates the correlation between the disturbance spectrum feature and the preset N intensity matrices, and determines the weight of the corresponding intensity matrix according to each correlation, so as to combine the N intensity matrices and the corresponding N weights to improve the accuracy of the emotional intensity in the synthesized speech. Correspondingly, the higher the correlation between the intensity matrix and the disturbance spectrum feature, the greater the weight corresponding to the intensity matrix, so as to dominate the emotional intensity in the synthesized speech.

[0040] Optionally, calculating the correlation between the disturbance spectrum feature and the preset N intensity matrices, and determining the weight of the corresponding intensity matrix according to each correlation includes: Use the trained attention mechanism model to determine the correlation between the perturbation spectrum features and the preset N intensity matrices; The correlation between the disturbance spectrum features and the corresponding intensity matrix is ​​determined as the weight of the corresponding intensity matrix.

[0041] Among them, the attention mechanism model is an information screening method. It introduces a query vector related to the task as the basis for feature selection, selects a scoring function to calculate the correlation between the input feature and the query vector, obtains the probability distribution of the input feature being selected, and finally screens out the feature information related to the task by taking the weighted average of the input features according to the probability distribution. Among them, there are four common forms of scoring functions: additive model, dot product model, scaled dot product model and bilinear model, which can be selected according to actual conditions.

[0042] Correspondingly, in this embodiment, the query vector is the disturbance spectrum feature, and the input feature is the preset N intensity matrices. The correlation between the disturbance spectrum feature and the preset N intensity matrices is calculated according to the scoring function, and the correlation between the disturbance spectrum feature and the corresponding intensity matrix is ​​determined as the weight of the corresponding intensity matrix, which serves as the basis for screening the intensity features in the disturbance spectrum feature.

[0043] This embodiment uses a trained attention mechanism model to determine the correlation between the disturbance spectrum characteristics and the preset N intensity matrices, and determines the correlation between the disturbance spectrum characteristics and the corresponding intensity matrix as the weight of the corresponding intensity matrix, thereby improving the accuracy of extracting intensity information.

[0044] Optionally, the training process of the attention mechanism model includes: Obtaining N preset intensity matrices, a number of original speech samples, M disturbance spectrum feature samples corresponding to each original speech sample, and a target text sample and a target speech feature sample to be synthesized, where M is an integer greater than 0; For any perturbation spectrum feature sample, use the attention mechanism model to determine the correlation between the perturbation spectrum feature sample and N intensity matrices, and determine the weight sample of the corresponding intensity matrix based on the correlation between the perturbation spectrum feature sample and the corresponding intensity matrix; Perform weighted summation on the N intensity matrices according to the weight samples corresponding to the N intensity matrices to obtain intensity feature samples; Perform feature fusion on the disturbance spectrum feature samples and the intensity feature samples to obtain spectrum intensity feature samples, perform speech synthesis on the target text samples according to the spectrum intensity feature samples, and adjust the synthesized speech using the target speech features to obtain the corresponding target speech samples; Calculate the emotional similarity between the original speech sample and the target speech sample corresponding to the perturbed spectral feature sample, and determine the difference between the emotional similarity and the preset value as the corresponding model sub-loss; Traverse all perturbation spectrum feature samples, obtain all model sub-losses, calculate the sum of all model sub-losses, and obtain the model loss; The parameters of the attention mechanism model are modified according to the model loss until the model loss converges to obtain a trained attention mechanism model.

[0045] In order to improve the accuracy of the weights corresponding to the intensity matrix, the attention mechanism model needs to be trained. In this embodiment, the preset N intensity matrices, a number of original speech samples, M perturbation spectrum feature samples corresponding to each original speech sample, and the target text samples and target speech feature samples to be synthesized are used as training samples, and the original speech samples are used as training labels to train the attention mechanism model.

[0046] Specifically, the attention mechanism model is used to determine the correlation between the perturbation spectrum feature samples and N intensity matrices, the correlation between the perturbation spectrum feature samples and the corresponding intensity matrices is determined, the weight samples of the corresponding intensity matrices are determined, and the N intensity matrices are weighted and summed according to the weight samples corresponding to the N intensity matrices to obtain intensity feature samples to characterize the emotion intensity information in the original speech; the perturbation spectrum feature samples and the intensity feature samples are feature fused to obtain spectrum intensity feature samples to characterize the emotion and emotion intensity information in the original speech; based on the spectrum intensity feature samples, the target text sample is speech synthesized, and the synthesized speech is adjusted using the target speech features to obtain the corresponding target speech sample.

[0047] The higher the accuracy of the attention mechanism model, the higher the emotional similarity between the target synthesized speech and the corresponding original speech. Therefore, the emotional similarity between the original speech sample and the target speech sample corresponding to the perturbation spectrum feature sample is calculated, and the difference between the emotional similarity and the preset value is determined as the corresponding model sub-loss, where the preset value can be set according to actual conditions. Correspondingly, the larger the sub-loss, the lower the accuracy of the attention mechanism model.

[0048] Then, all perturbation spectrum feature samples are traversed, the sum of all model sub-losses is calculated, the model loss is obtained, and the parameters of the attention mechanism model are corrected according to the model loss until the model loss converges. The trained attention mechanism model is obtained to determine the correlation between the perturbation spectrum features and the preset N intensity matrices to improve the accuracy of the weights of the corresponding intensity matrices.

[0049] This embodiment uses the preset N intensity matrices, several original speech samples, M perturbation spectrum feature samples corresponding to each original speech sample, and the target text samples and target speech feature samples to be synthesized as training samples, and the original speech samples as training labels to train the attention mechanism model, and obtains the trained attention mechanism model to determine the correlation between the perturbation spectrum features and the preset N intensity matrices, thereby improving the accuracy of the weights of the corresponding intensity matrices.

[0050] Optionally, obtaining M disturbance spectrum feature samples corresponding to each original speech sample includes: For any original speech sample, an original spectrum sample of the original speech sample is obtained, K resonance peaks in the original spectrum sample are determined, the frequencies of the K resonance peaks are randomly perturbed to obtain the perturbation frequencies of the corresponding resonance peaks, and the perturbation frequency of each resonance peak is used to replace the frequency of the corresponding resonance peak in the original spectrum sample to obtain a perturbed spectrum sample; Repeat the above steps of randomly perturbing the frequencies of the K resonance peaks M times to obtain the perturbation frequencies of the corresponding resonance peaks, and use the perturbation frequencies of each resonance peak to replace the frequencies of the corresponding resonance peaks in the original spectrum samples to obtain perturbed spectrum samples, and obtain M perturbation spectrum samples corresponding to the original speech samples; Encode the M disturbance spectrum samples respectively to obtain the corresponding M disturbance spectrum feature samples; All original speech samples are traversed to obtain M disturbance spectrum feature samples corresponding to each original speech sample.

[0051] Among them, in order to improve the diversity of training samples, for any original speech sample, this embodiment randomly perturbs the frequencies of the K resonance peaks of the corresponding original spectrum sample M times, and obtains M perturbed spectrum samples accordingly, and encodes the M perturbed spectrum samples respectively to obtain M corresponding perturbed spectrum feature samples. The M perturbed spectrum feature samples can be combined with the preset N intensity matrices, the target text samples to be synthesized, and the target speech feature samples as training samples to train the attention mechanism model. On the basis of a small number of original speech samples, a large number of training samples can be provided, thereby improving the training accuracy of the attention mechanism model.

[0052] In this embodiment, for any original speech sample, the frequencies of K resonance peaks of the corresponding original spectrum sample are randomly perturbed M times, and corresponding M perturbed spectrum feature samples are obtained. Based on a small number of original speech samples, a large number of training samples are obtained, thereby improving the training accuracy of the attention mechanism model.

[0053] Optionally, calculating the emotion similarity between the original speech sample and the target speech sample corresponding to the perturbation spectrum feature sample includes: Encode the original speech sample and the target speech sample using the trained emotion encoder to obtain a first emotion feature sequence corresponding to the original speech sample and a second emotion feature sequence corresponding to the target speech sample; The similarity between the first emotion feature sequence and the second emotion feature sequence is calculated as the emotion similarity between the original speech sample and the target speech sample.

[0054] Among them, the accuracy of the attention mechanism model is measured based on the emotional similarity between the target synthesized speech and the corresponding original speech. In order to accurately calculate the similarity between the emotions, this embodiment uses a trained emotion encoder to encode the original speech sample and the target speech sample to obtain a first emotion feature sequence corresponding to the original speech sample and a second emotion feature sequence corresponding to the target speech sample. The similarity between the first emotion feature sequence and the second emotion feature sequence is calculated, which can be used as the emotion similarity between the original speech sample and the target speech sample.

[0055] This embodiment uses a trained emotion encoder to encode the original speech sample and the target speech sample to obtain a first emotion feature sequence corresponding to the original speech sample and a second emotion feature sequence corresponding to the target speech sample, and converts the calculation of the emotion similarity between the original speech sample and the target speech sample into the calculation of the similarity between the corresponding first emotion feature sequence and the second emotion feature sequence, thereby improving the reliability and accuracy of the emotion similarity calculation.

[0056] The above-mentioned steps of encoding the disturbance spectrum features to obtain the disturbance spectrum features, calculating the correlation between the disturbance spectrum features and the preset N intensity matrices, and determining the weight of the corresponding intensity matrix according to each correlation, by calculating the correlation between the disturbance spectrum features and the preset N intensity matrices, to combine the N intensity matrices and the corresponding N weights, thereby improving the accuracy of the emotion intensity in the synthesized speech.

[0057] Step S203, performing weighted summation on the N intensity matrices according to the weights corresponding to the N intensity matrices to obtain intensity features, and performing feature fusion on the intensity features and the disturbance spectrum features to obtain spectrum intensity features.

[0058] The weight corresponding to the intensity matrix can represent the dominance of the intensity matrix in the emotional intensity of the speech to be synthesized. Therefore, the N intensity matrices are weighted and summed according to the weights corresponding to the N intensity matrices to obtain the intensity feature to represent the emotional intensity information in the speech to be synthesized. The intensity feature and the disturbance spectrum feature are fused to obtain the spectrum intensity feature to simultaneously represent the emotional information and emotional intensity information in the speech to be synthesized.

[0059] The above steps of performing weighted summation of N intensity matrices according to the weights corresponding to the N intensity matrices to obtain intensity features, performing feature fusion of the intensity features and the disturbance spectrum features to obtain spectrum intensity features, performing weighted summation of the intensity matrix and the corresponding weights, and performing feature fusion of the intensity features and the disturbance spectrum features, improve the accuracy of the emotional information and emotional intensity information in the speech to be synthesized.

[0060] Step S204, obtaining the target text to be synthesized and the target speech features, performing speech synthesis on the target text according to the spectrum intensity features, and adjusting the synthesized speech using the target speech features to obtain the corresponding target speech.

[0061] Among them, the target text to be synthesized is used to characterize the text information in the speech to be synthesized, the target speech features to be synthesized can be used to characterize the timbre information in the speech to be synthesized, and the spectral intensity features can be used to characterize the emotional information and emotional intensity information in the speech to be synthesized. The target text can be speech synthesized according to the spectral intensity features to synthesize speech containing text information, emotional information and emotional intensity information, and then the target speech features are used to adjust the synthesized speech, give the synthesized speech the target timbre information, and obtain the corresponding target speech.

[0062] For example, in the intelligent customer service scenario in the financial field, the target text can be the speech text, and the target speech features can be the timbre information extracted from the human customer service. The spectral intensity features, target text and target speech features can be combined to synthesize a natural and expressive target speech for the robot customer service, thereby assisting the human customer service to communicate with customers and improving the service quality and efficiency of financial services.

[0063] The above steps include obtaining the target text to be synthesized and the target speech features, performing speech synthesis on the target text according to the spectral intensity features, adjusting the synthesized speech using the target speech features, and obtaining the corresponding target speech. The target text is speech synthesized according to the spectral intensity features to synthesize speech containing text information, emotional information and emotional intensity information, and then the target speech features are used to adjust the synthesized speech to give the synthesized speech target timbre information to obtain the corresponding target speech, thereby improving the emotional accuracy of the synthesized target speech.

[0064] The embodiment of the present invention obtains the original spectrum of the original speech, determines K resonance peaks in the original spectrum, randomly perturbs the frequencies of the K resonance peaks to obtain the perturbation frequencies of the corresponding resonance peaks, uses the perturbation frequency of each resonance peak to replace the frequency of the corresponding resonance peak in the original spectrum to obtain the perturbation spectrum, performs feature encoding on the perturbation spectrum to obtain the perturbation spectrum feature, calculates the correlation between the perturbation spectrum feature and N preset intensity matrices, determines the weight of the corresponding intensity matrix according to each correlation, performs weighted summation on the N intensity matrices according to the weights corresponding to the N intensity matrices to obtain the intensity feature, performs feature fusion on the intensity feature and the perturbation spectrum feature to obtain the spectrum intensity feature, and performs feature fusion on the intensity feature and the perturbation spectrum feature , obtain the spectrum intensity feature, obtain the target text and target speech features to be synthesized, perform speech synthesis on the target text according to the spectrum intensity feature, adjust the synthesized speech using the target speech feature, and obtain the corresponding target speech. The speaker information represented in the original spectrum is destroyed by perturbing the resonance peak, and only the effective emotional information and the intensity information of the emotion in the original spectrum are retained. The weight of the corresponding intensity matrix is ​​determined by calculating the correlation between the perturbed spectrum feature and the preset N intensity matrices, which improves the accuracy of the emotional intensity in the target speech. In the intelligent customer service scenario in the financial field, a natural and expressive target speech is synthesized for the robot customer service, which improves the service quality and efficiency of the financial business.

[0065] Corresponding to the speech synthesis method of the above embodiment, Figure 3 A structural block diagram of a speech synthesis device with adjustable spectrum intensity provided by Embodiment 2 of the present invention is given. For the sake of convenience, only the parts related to the embodiment of the present invention are shown.

[0066] See also Figure 3 , the speech synthesis device comprises: The frequency perturbation module 31 is used to obtain the original spectrum of the original speech, determine K resonance peaks in the original spectrum, randomly perturb the frequencies of the K resonance peaks to obtain the perturbation frequencies of the corresponding resonance peaks, and use the perturbation frequency of each resonance peak to replace the frequency of the corresponding resonance peak in the original spectrum to obtain the perturbation spectrum, where K is an integer greater than 0; The weight determination module 32 is used to perform feature encoding on the disturbance spectrum to obtain the disturbance spectrum feature, calculate the correlation between the disturbance spectrum feature and the preset N intensity matrices, and determine the weight of the corresponding intensity matrix according to each correlation, where N is an integer greater than 0; A feature fusion module 33 is used to perform weighted summation on the N intensity matrices according to the weights corresponding to the N intensity matrices to obtain an intensity feature, and to perform feature fusion on the intensity feature and the disturbance spectrum feature to obtain a spectrum intensity feature; The speech synthesis module 34 is used to obtain the target text to be synthesized and the target speech features, perform speech synthesis on the target text according to the spectrum intensity features, and adjust the synthesized speech using the target speech features to obtain the corresponding target speech.

[0067] Optionally, the frequency disturbance module 31 includes: The first frequency perturbation submodule is used to determine the original frequency of the formant according to the spectrum of the original speech for any formant, randomly obtain the interference frequency within a preset frequency range, and determine the sum of the original frequency and the interference frequency as the perturbation frequency of the formant. The preset frequency range is [0, F], where F is the fundamental frequency of the spectrum; The second frequency perturbation submodule is used to traverse all the resonance peaks and obtain the perturbation frequencies of all the resonance peaks.

[0068] Optionally, the weight determination module 32 includes: A correlation determination submodule is used to determine the correlation between the perturbation spectrum features and the preset N intensity matrices using the trained attention mechanism model; The weight determination submodule is used to determine the correlation between the disturbance spectrum characteristics and the corresponding intensity matrix as the weight of the corresponding intensity matrix.

[0069] Optionally, the above correlation determination submodule includes: A data acquisition unit, used to acquire N preset intensity matrices, a number of original speech samples, M disturbance spectrum feature samples corresponding to each original speech sample, and a target text sample and a target speech feature sample to be synthesized, where M is an integer greater than 0; A weight determination unit is used to determine the correlation between the perturbation spectrum feature sample and N intensity matrices using an attention mechanism model for any perturbation spectrum feature sample, and to determine the weight sample of the corresponding intensity matrix based on the correlation between the perturbation spectrum feature sample and the corresponding intensity matrix; A weighted summation unit, used for performing weighted summation on the N intensity matrices according to weight samples corresponding to the N intensity matrices to obtain an intensity feature sample; A speech synthesis unit is used to perform feature fusion on the disturbance spectrum feature samples and the intensity feature samples to obtain spectrum intensity feature samples, perform speech synthesis on the target text samples according to the spectrum intensity feature samples, and adjust the synthesized speech using the target speech features to obtain the corresponding target speech samples; A sub-loss calculation unit, used to calculate the emotional similarity between the original speech sample and the target speech sample corresponding to the perturbed spectral feature sample, and determine the difference between the emotional similarity and a preset value as the corresponding model sub-loss; The model loss calculation unit is used to traverse all perturbation spectrum feature samples, obtain all model sub-losses, calculate the sum of all model sub-losses, and obtain the model loss; The parameter correction unit is used to correct the parameters of the attention mechanism model according to the model loss until the model loss converges to obtain a trained attention mechanism model.

[0070] Optionally, the data acquisition unit includes: The first frequency perturbation subunit is used to obtain an original spectrum sample of the original speech sample for any original speech sample, determine K resonance peaks in the original spectrum sample, randomly perturb the frequencies of the K resonance peaks to obtain perturbation frequencies of corresponding resonance peaks, and use the perturbation frequency of each resonance peak to replace the frequency of the corresponding resonance peak in the original spectrum sample to obtain a perturbed spectrum sample; The second frequency perturbation subunit is used to repeatedly perform the above steps of randomly perturbing the frequencies of the K resonance peaks M times to obtain the perturbation frequencies of the corresponding resonance peaks, and using the perturbation frequency of each resonance peak to replace the frequency of the corresponding resonance peak in the original spectrum sample to obtain the perturbed spectrum sample, thereby obtaining M perturbation spectrum samples corresponding to the original speech sample; A feature encoding subunit is used to encode the M disturbance spectrum samples respectively to obtain corresponding M disturbance spectrum feature samples; The data acquisition subunit is used to traverse all original speech samples to obtain M disturbance spectrum feature samples corresponding to each original speech sample.

[0071] Optionally, the above-mentioned sub-loss calculation unit includes: The feature encoding subunit is used to encode the original speech sample and the target speech sample using the trained emotion encoder to obtain a first emotion feature sequence corresponding to the original speech sample and a second emotion feature sequence corresponding to the target speech sample; The similarity calculation subunit is used to calculate the similarity between the first emotion feature sequence and the second emotion feature sequence as the emotion similarity between the original speech sample and the target speech sample.

[0072] Optionally, the frequency disturbance module 31 includes: The resonance peak extraction submodule is used to obtain the spectrum envelope of the original spectrum, determine K local peaks in the spectrum envelope, and determine the K local peaks as K resonance peaks in the original spectrum.

[0073] It should be noted that the information interaction, execution process and other contents between the above modules are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0074] Figure 4 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Figure 4 As shown, the computer device of this embodiment includes: at least one processor ( Figure 4 Only one is shown in the figure), a memory, and a computer program stored in the memory and executable on at least one processor, wherein when the processor executes the computer program, the steps in any of the above-mentioned speech synthesis method embodiments are implemented.

[0075] The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that Figure 4 This is merely an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components than those shown in the figure, or a combination of certain components, or different components. For example, it may also include a network interface, a display screen, and an input device.

[0076] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0077] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory may be the memory of a computer device, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The readable storage medium may be a hard disk of a computer device, and in other embodiments, it may also be an external storage device of the computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Further, the memory may also include both an internal storage unit of the computer device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of a computer program, etc. The memory may also be used to temporarily store data that has been output or is to be output.

[0078] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiment can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0079] The present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by a computer program product. When the computer program product runs on a computer device, the computer device can implement the steps in the above-mentioned method embodiment when executing.

[0080] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0081] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0082] In the embodiments provided by the present invention, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0083] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0084] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A speech synthesis method with adjustable spectrum intensity, characterized in that: The speech synthesis method comprises: Acquire an original spectrum of the original speech, determine K formants in the original spectrum, randomly perturb the frequencies of the K formants to obtain perturbation frequencies of the corresponding formants, and use the perturbation frequency of each formant to replace the frequency of the corresponding formant in the original spectrum to obtain a perturbation spectrum, where K is an integer greater than 0; Performing feature encoding on the disturbance spectrum to obtain disturbance spectrum features, calculating the correlation between the disturbance spectrum features and N preset intensity matrices, and determining the weight of the corresponding intensity matrix according to each correlation, where N is an integer greater than 0; Performing weighted summation on the N intensity matrices according to weights corresponding to the N intensity matrices to obtain an intensity feature, and performing feature fusion on the intensity feature and the disturbance spectrum feature to obtain a spectrum intensity feature; The target text to be synthesized and the target speech features are obtained, speech synthesis is performed on the target text according to the spectrum intensity features, and the synthesized speech is adjusted using the target speech features to obtain the corresponding target speech.

2. The speech synthesis method according to claim 1, characterized in that: The randomly perturbing the frequencies of the K resonance peaks to obtain the perturbation frequencies of the corresponding resonance peaks comprises: For any formant, the original frequency of the formant is determined according to the spectrum of the original speech, an interference frequency is randomly obtained within a preset frequency range, and the sum of the original frequency and the interference frequency is determined as the disturbance frequency of the formant, and the preset frequency range is [0, F], where F is the fundamental frequency of the spectrum; Traverse all the resonance peaks and obtain the perturbation frequencies of all the resonance peaks.

3. The speech synthesis method according to claim 1, characterized in that: The calculating the correlation between the disturbance spectrum feature and the preset N intensity matrices, and determining the weight of the corresponding intensity matrix according to each correlation comprises: Using the trained attention mechanism model, determining the correlation between the perturbation spectrum features and the preset N intensity matrices; The correlation between the disturbance spectrum feature and the corresponding intensity matrix is ​​determined as the weight of the corresponding intensity matrix.

4. The speech synthesis method according to claim 3, characterized in that: The training process of the attention mechanism model includes: Obtaining N preset intensity matrices, a number of original speech samples, M disturbance spectrum feature samples corresponding to each original speech sample, and a target text sample and a target speech feature sample to be synthesized, where M is an integer greater than 0; For any disturbance spectrum feature sample, use the attention mechanism model to determine the correlation between the disturbance spectrum feature sample and the N intensity matrices, and determine the weight sample of the corresponding intensity matrix by taking the correlation between the disturbance spectrum feature sample and the corresponding intensity matrix; Performing weighted summation on the N intensity matrices according to weight samples corresponding to the N intensity matrices to obtain an intensity feature sample; Performing feature fusion on the disturbance spectrum feature sample and the intensity feature sample to obtain a spectrum intensity feature sample, performing speech synthesis on the target text sample according to the spectrum intensity feature sample, and adjusting the synthesized speech using the target speech feature to obtain a corresponding target speech sample; Calculating the emotional similarity between the original speech sample and the target speech sample corresponding to the perturbation spectrum feature sample, and determining the difference between the emotional similarity and a preset value as the corresponding model sub-loss; Traverse all perturbation spectrum feature samples, obtain all model sub-losses, calculate the sum of all model sub-losses, and obtain the model loss; The parameters of the attention mechanism model are modified according to the model loss until the model loss converges to obtain a trained attention mechanism model.

5. The speech synthesis method according to claim 4, characterized in that: Obtaining M disturbance spectrum feature samples corresponding to each original speech sample includes: For any original speech sample, an original spectrum sample of the original speech sample is obtained, K resonance peaks in the original spectrum sample are determined, the frequencies of the K resonance peaks are randomly perturbed to obtain the perturbation frequencies of the corresponding resonance peaks, and the perturbation frequency of each resonance peak is used to replace the frequency of the corresponding resonance peak in the original spectrum sample to obtain a perturbed spectrum sample; Repeat the above steps of randomly perturbing the frequencies of the K resonance peaks M times to obtain the perturbation frequencies of the corresponding resonance peaks, and using the perturbation frequency of each resonance peak to replace the frequency of the corresponding resonance peak in the original spectrum sample to obtain the perturbed spectrum sample, and obtain M perturbation spectrum samples corresponding to the original speech sample; Encode the M disturbance spectrum samples respectively to obtain corresponding M disturbance spectrum feature samples; All original speech samples are traversed to obtain M disturbance spectrum feature samples corresponding to each original speech sample.

6. The speech synthesis method according to claim 4, characterized in that: The calculating the emotional similarity between the original speech sample and the target speech sample corresponding to the disturbance spectrum feature sample comprises: Encoding the original speech sample and the target speech sample using a trained emotion encoder to obtain a first emotion feature sequence corresponding to the original speech sample and a second emotion feature sequence corresponding to the target speech sample; The similarity between the first emotion feature sequence and the second emotion feature sequence is calculated as the emotion similarity between the original speech sample and the target speech sample.

7. The speech synthesis method according to claim 1, characterized in that: Determining the K resonance peaks in the original spectrum comprises: The spectrum envelope of the original spectrum is acquired, K local peaks in the spectrum envelope are determined, and the K local peaks are determined as K resonance peaks in the original spectrum.

8. A speech synthesis device with adjustable spectrum intensity, characterized in that: The speech synthesis device comprises: A frequency perturbation module is used to obtain an original spectrum of the original speech, determine K resonance peaks in the original spectrum, randomly perturb the frequencies of the K resonance peaks to obtain perturbation frequencies of the corresponding resonance peaks, and use the perturbation frequency of each resonance peak to replace the frequency of the corresponding resonance peak in the original spectrum to obtain a perturbation spectrum, where K is an integer greater than 0; A weight determination module is used to perform feature encoding on the disturbance spectrum to obtain disturbance spectrum features, calculate the correlation between the disturbance spectrum features and N preset intensity matrices, and determine the weight of the corresponding intensity matrix according to each correlation, where N is an integer greater than 0; A feature fusion module is used to perform weighted summation on the N intensity matrices according to the weights corresponding to the N intensity matrices to obtain an intensity feature, and perform feature fusion on the intensity feature and the disturbance spectrum feature to obtain a spectrum intensity feature; The speech synthesis module is used to obtain the target text to be synthesized and the target speech features, perform speech synthesis on the target text according to the spectrum intensity features, and adjust the synthesized speech using the target speech features to obtain the corresponding target speech.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the speech synthesis method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the speech synthesis method according to any one of claims 1 to 7 is implemented.

Images