Human-like expression-oriented dialect speech synthesis method, device and equipment and medium
By extracting and marking the training text and speech, aligning the training data sets for model training and cross-attention diffusion, the problem of poor naturalness and robustness of dialect speech synthesis in the prior art is solved, and the synthesis efficiency and quality are improved.
Patent Information
- Application Number
- CN202510095120.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-21
- Publication Date
- 2025-04-29
AI Technical Summary
Existing speech synthesis technologies are difficult to generate dialect speech with high naturalness and robustness, and the synthesis efficiency is low, mainly due to the lack of dialect labeled data sets.
By performing feature extraction and labeling of pre-acquisitioned training text and speech, aligning the training datasets for model training and cross-attention diffusion, a dialect speech model is generated.
It improves the naturalness, robustness and efficiency of dialect speech synthesis, and achieves more accurate and high-quality dialect speech generation.
Smart Images
Figure CN119905084A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of speech synthesis and medical health, and in particular to a dialect speech synthesis method, device, equipment and medium for human-like expression. Background Art
[0002] Intelligent medical consultation is a medical service method based on artificial intelligence technology, combined with natural language processing, knowledge graphs, machine learning and big data analysis. Intelligent medical consultation simulates the doctor's consultation process, communicates with patients, collects relevant information such as symptoms and medical history, and provides preliminary diagnosis of diseases, treatment recommendations, health consultation and other services. With the development of artificial intelligence technology, intelligent medical diagnosis has been increasingly widely used.
[0003] Financial intelligent customer service is a financial service method based on speech recognition, natural language processing and artificial intelligence technology. It can provide customers with account inquiries, service consultations and market analysis services, and has been increasingly widely used.
[0004] In intelligent medical diagnosis and financial customer service consultation, in order to interact with users who use various dialects, it is necessary to use speech synthesis technology to generate dialect voices for corresponding replies. In actual applications, existing speech synthesis technology is mainly limited to generating non-dialect voices, and the naturalness and robustness of the generated voices are poor, and the efficiency of synthesizing dialect voices is low. Summary of the invention
[0005] The present invention provides a dialect speech synthesis method, device, equipment and medium for human-like expression, so as to solve the technical problem in the existing speech synthesis technology that the naturalness and robustness of the generated dialect speech are poor and the efficiency is low due to the lack of dialect labeled data sets.
[0006] In a first aspect, a dialect speech synthesis method for human-like expression is provided, comprising:
[0007] Performing text feature extraction and text feature labeling on the pre-acquired training text to obtain labeled text features;
[0008] Extracting and marking speech features of the training speech acquired in advance to obtain marked speech features;
[0009] Using the marked text features to perform mark alignment on the marked speech features to obtain an alignment training data set;
[0010] Using the aligned training data set to perform model training on a preset speech synthesis model to obtain a primary speech model;
[0011] Using the aligned training data set to perform cross attention diffusion on the primary speech model to obtain a dialect speech model;
[0012] A medical diagnosis reply text is obtained, and a reply audio corresponding to the medical diagnosis reply text is generated using the dialect speech model.
[0013] In a second aspect, a dialect speech synthesis device for human-like expression is provided, comprising:
[0014] A text tagging module is used to extract and tag text features from pre-acquired training text to obtain tagged text features;
[0015] An audio tagging module is used to extract and tag speech features of pre-acquired training speech to obtain tagged speech features;
[0016] A tag alignment module, used to perform tag alignment on the tag speech feature using the tag text feature to obtain an alignment training data set;
[0017] A model training module, used to perform model training on a preset speech synthesis model using the aligned training data set to obtain a primary speech model;
[0018] A model diffusion module, used to perform cross attention diffusion on the primary speech model using the aligned training data set to obtain a dialect speech model;
[0019] The audio synthesis module is used to obtain the medical diagnosis reply text and generate the reply audio corresponding to the medical diagnosis reply text using the dialect speech model.
[0020] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned dialect speech synthesis method for human-like expression when executing the computer program.
[0021] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the above-mentioned dialect speech synthesis method for human-like expression is implemented.
[0022] In the scheme implemented by the above-mentioned dialect speech synthesis method, device, computer equipment and storage medium for human-like expression, communication can be carried out with the server through the client's network. The server can extract text features and mark text features of the pre-acquired training text through the client to obtain marked text features; extract speech features and mark speech features of the pre-acquired training speech to obtain marked speech features; use the marked text features to mark and align the marked speech features to obtain an aligned training data set; use the aligned training data set to perform model training on a preset speech synthesis model to obtain a primary speech model; use the aligned training data set to perform cross-attention diffusion on the primary speech model to obtain a dialect speech model; obtain a medical diagnosis reply text, and use the dialect speech model to generate a reply audio corresponding to the medical diagnosis reply text. In the present invention, , for using the marked text features to mark and align the marked speech features to obtain an aligned training data set, the method includes: splitting the marked text features into precise marked text features and imprecise marked text features; splitting the marked speech features into precise marked speech features and imprecise marked speech features; performing unsupervised sampling alignment on the imprecise marked text features and the imprecise marked speech features to obtain an unsupervised aligned data set; performing text-speech alignment on the precise marked text features and the precise marked speech features to obtain a standard aligned data set; using the standard aligned data set to fine-tune the sampling and update of the unsupervised aligned data set to obtain an aligned training data set, which can realize semi-supervised relationship mapping learning between marked text features and marked speech features, thereby expanding the training set data volume of the dialect speech synthesis model and improving the efficiency of dialect speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0024] Figure 1 It is a schematic diagram of an application environment of a dialect speech synthesis method for human-like expression in one embodiment of the present invention;
[0025] Figure 2 It is a flow chart of a dialect speech synthesis method for human-like expression in one embodiment of the present invention;
[0026] Figure 3 yes Figure 1 A schematic flow chart of a specific implementation of step S20;
[0027] Figure 4 yes Figure 1 A schematic flow chart of a specific implementation of step S30;
[0028] Figure 5 yes Figure 1 A schematic flow chart of a specific implementation of step S40;
[0029] Figure 6 is a schematic structural diagram of a dialect speech synthesis device for human-like expression in one embodiment of the present invention;
[0030] Figure 7 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;
[0031] Figure 8 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION
[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0033] The dialect speech synthesis method for human-like expression provided by the embodiment of the present invention can be applied in the following aspects: Figure 1In an application environment, the client communicates with the server through a network. The server can perform text feature extraction and text feature tagging on a pre-acquired training text through the client to obtain a tagged text feature; perform speech feature extraction and speech feature tagging on a pre-acquired training speech to obtain a tagged speech feature; use the tagged text feature to align the tagged speech features to obtain an aligned training data set; use the aligned training data set to perform model training on a preset speech synthesis model to obtain a primary speech model; use the aligned training data set to perform cross-attention diffusion on the primary speech model to obtain a dialect speech model; obtain a medical diagnosis reply text, and use the dialect speech model to generate a reply audio corresponding to the medical diagnosis reply text. In the present invention, for using the tagged text feature to align the tagged speech features The method comprises the following steps: aligning the marked text features into precise marked text features and imprecise marked text features; aligning the marked speech features into precise marked speech features and imprecise marked speech features; performing unsupervised sampling alignment on the imprecise marked text features and the imprecise marked speech features to obtain an unsupervised aligned data set; performing text-speech alignment on the precise marked text features and the precise marked speech features to obtain a standard aligned data set; and performing fine-tuning sampling and updating on the unsupervised aligned data set using the standard aligned data set to obtain an aligned training data set, which can realize semi-supervised relationship mapping learning between marked text features and marked speech features, thereby expanding the training set data volume of the dialect speech synthesis model and improving the efficiency of dialect speech synthesis. Among them, the client can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented by an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.
[0034] See also Figure 2 As shown, Figure 2 A flow chart of a dialect speech synthesis method for human-like expression provided by an embodiment of the present invention includes the following steps:
[0035] S10: extracting and marking text features from the pre-acquired training text to obtain marked text features.
[0036] In detail, the training text refers to text information related to medical diagnosis that contains large-scale imprecise dialect audio tags and small-scale high-quality dialect audio tags, and the marked text features refer to features of the Mandarin text corresponding to the text.
[0037] In the embodiment of the present invention, the text feature extraction and text feature tagging of the pre-acquired training text to obtain the tagged text features includes:
[0038] Performing text segmentation on the pre-acquired training text to obtain a training text word sequence;
[0039] Performing feature encoding on the training text word sequence to obtain a training word feature sequence;
[0040] Performing phoneme mapping on the training word feature sequence to obtain a training phoneme sequence;
[0041] Extracting phoneme features from the training phoneme sequence to obtain a phoneme feature sequence;
[0042] Generate training text features according to the phoneme feature sequence and the training word feature sequence;
[0043] The training text is used to perform audio tagging on the training text features to obtain the tagged text features.
[0044] Example description: In the field of healthcare, doctors often need to communicate with patients through text data such as patient medical records. In order to improve the patient experience, medical institutions can optimize the training effect of the medical record-to-speech model by extracting and marking text features of medical record text. This method enables the model to more accurately generate voice output that conforms to medical professionalism and natural interaction, facilitating voice communication between doctors and patients, especially in telemedicine scenarios. For example, suppose the original medical record text is: "The patient complained of chest pain, accompanied by night sweats and recent weight loss." Text feature extraction: extract key medical features such as "chest pain", "night sweats", and "weight loss". Text tagging: add tags to each feature to provide semantic hierarchical information. The labeled text is: <symptom: location> chest pain <symptom: location>, <symptom: time period> night sweats <symptom: time period>, <symptom: health status> recent weight loss <symptom: health status>.
[0045] In the financial field, by converting customers' financial text records into voice form, it can be used in intelligent voice assistants or customer service robots to improve customer experience. Text feature extraction and tagging of original text can help the model generate more professional and semantically clear voice output, especially in the processing of complex financial terminology.
[0046] In detail, text segmentation refers to dividing a continuous text into single words or sub-word units. Each training text word in the training text word sequence corresponds to each smallest word unit in the training text. Text segmentation can be performed using bidirectional maximum matching, regular expression segmentation, or word segmentation methods based on word segmentation such as Byte Pair Encoding, WordPiece and SentencePiece.
[0047] Specifically, the feature encoding refers to the process of converting segmented words into word features, that is, representing discrete text data as continuous numerical vectors, and each training word feature in the training word feature sequence corresponds to the numerical vector representation of each training text word in the training text word sequence.
[0048] In detail, feature encoding can be performed using statistics-based word feature encoding methods such as Bag of Words (BoW) and TFIDF, prediction-based word feature encoding methods such as Word2Vec and GloVe, and context-based dynamic word features such as ELMo and Transformerbased models.
[0049] Specifically, the phoneme mapping refers to mapping word features into corresponding phonemes, and phoneme mapping can be performed using rule-based mapping methods, such as dictionary mapping methods such as English pronunciation dictionaries, or methods such as hidden Markov models and conditional random fields based on statistical learning.
[0050] In detail, the phoneme feature extraction refers to mapping each training phoneme in the training phoneme sequence into a phoneme feature expressed in the form of a one-dimensional vector, and the generation of training text features based on the phoneme feature sequence and the training word feature sequence refers to fusing each phoneme feature in the phoneme feature sequence and the corresponding training word features in the training word feature sequence into a training word feature, and generating training text features based on all the training word features.
[0051] Specifically, using the training text to perform audio tagging on the training text features to obtain the marked text features means tagging the training text features using large-scale non-precise dialect audio tags and small-scale high-quality dialect audio tags of the training text, thereby obtaining a correspondence between Mandarin text, text features and dialect audio.
[0052] In the embodiment of the present invention, by performing text feature extraction and text feature labeling on pre-acquired training texts to obtain labeled text features, the training text data can be converted into continuous feature vectors, thereby simplifying the amount of data calculation and improving model training efficiency.
[0053] S20: extracting and labeling speech features of the pre-acquired training speech to obtain labeled speech features.
[0054] In detail, the training speech includes various dialect audio data with large-scale imprecise Mandarin text labels and small-scale high-quality Mandarin text labels.
[0055] In the embodiment of the present invention, refer to Figure 3As shown, the speech feature extraction and speech feature labeling of the pre-acquired training speech to obtain the labeled speech features includes:
[0056] S21, performing speech framing and speech windowing on the pre-acquired training speech to obtain the windowed training speech;
[0057] S22, performing frequency domain conversion on the training speech to obtain a training speech frequency domain signal;
[0058] S23, performing signal spectrum calculation on the training speech frequency domain signal to obtain a windowed power spectrum signal;
[0059] S24, performing Mel-cepstrum filtering on the windowed power spectrum signal to obtain a training speech spectrum feature;
[0060] S25, performing time domain conversion on the training speech spectrum features to obtain time domain speech features;
[0061] S26. Using the training speech, perform text marking on the time-domain speech features to obtain marked speech features.
[0062] Example description: In the field of medical health, in the voice communication between dialect patients and doctors, the speech-to-Mandarin text system needs to accurately identify dialect features and generate accurate Mandarin text. By extracting features and marking the patient's dialect speech in Mandarin, the system's adaptability to dialects can be improved. For example, the Mandarin text corresponding to the Sichuan dialect speech feature marking of "Doctor, my head hurts. It may be because of the wind at night" is "Doctor, my head hurts. It may be because of the cold at night."
[0063] In the financial field, users may interact with intelligent customer service through voice, especially dialect users. The dialect voice needs to be accurately transcribed into Mandarin text to facilitate subsequent business processing. Through feature extraction and Mandarin tagging, the customer service system's recognition ability of dialect users can be improved. For example, the Mandarin text corresponding to the Cantonese dialect "I want to open an account, how do I go about it?" is "I want to open an account, how do I do it?"
[0064] Specifically, speech windowing can be performed using methods such as Hamming Window, Hanning Window or Blackman Window, and frequency domain conversion can be performed using methods such as Fast Fourier Transform or Wavelet Transform.
[0065] In detail, the speech windowing is a key step in audio signal processing, the purpose of which is to divide a long speech signal into short-term signals for analysis. The speech framing refers to dividing a continuous speech signal into speech frames of equal length. Each speech frame is usually 20ms to 30ms long, and there is a certain overlap between the speech frames to ensure continuity.
[0066] Specifically, the Fast Fourier Transform (FFT) is an efficient algorithm for calculating Fourier transform, which is used to convert time domain signals into frequency domain representation. The wavelet transform is a time-frequency analysis tool, which is used to simultaneously analyze the time domain and frequency domain characteristics of signals. Compared with the Fourier transform, the wavelet transform is more suitable for processing non-stationary signals.
[0067] In detail, the signal spectrum calculation refers to calculating the power spectrum of the problem speech frequency domain signal. The power spectrum is a representation method that describes the energy distribution of the signal in the frequency domain, and is used to reveal the energy intensity of the signal at different frequencies. The power spectrum is the square of the spectrum, that is, the square of the amplitude of the signal, which is proportional to the physical power of the signal.
[0068] Specifically, the Mel Frequency Cepstral Coefficients (MFCC) is a speech feature extraction method that simulates the nonlinear frequency perception of the human ear. The Mel Frequency Cepstral Filter maps the spectrum features to a frequency scale that is more consistent with human auditory perception, thereby realizing the extraction of speech features. The time domain conversion refers to converting frequency domain features into time domain features, and the time domain conversion can be performed using the inverse Fourier transform or inverse wavelet transform method.
[0069] In detail, the method of using the training speech to perform text labeling on the time-domain speech features to obtain labeled speech features refers to labeling the time-domain speech features using large-scale imprecise dialect text labels and small-scale high-quality dialect text labels of the training speech, thereby obtaining a correspondence between Mandarin text, dialect speech and dialect speech features.
[0070] In the embodiment of the present invention, by performing speech feature extraction and speech feature labeling on the pre-acquired training speech, the labeled speech features are obtained, which can effectively remove the high-dimensional redundant information in the training speech, and can capture the rhythm, timbre and intonation features in the dialect speech, which is helpful for the accurate modeling of the dialect speech and improves the accuracy of subsequent dialect synthesis.
[0071] S30: Using the marked text features to perform mark alignment on the marked speech features to obtain an alignment training data set.
[0072] In detail, the aligned training data set includes training set input data consisting of aligned text features and training set output data consisting of corresponding aligned speech features, wherein the aligned text features refer to text features in the marked text features that have a corresponding mapping relationship with speech features in the marked speech features, and the aligned speech features refer to speech features in the marked speech features that have a corresponding mapping relationship with text features in the marked text features.
[0073] In the embodiment of the present invention, refer to Figure 4 As shown, the use of the marked text features to mark and align the marked speech features to obtain an aligned training data set includes:
[0074] S31, splitting the marked text features into precise marked text features and imprecise marked text features;
[0075] S32, splitting the marked speech features into accurate marked speech features and inaccurate marked speech features;
[0076] S33, performing unsupervised sampling alignment on the imprecisely marked text features and the imprecisely marked speech features to obtain an unsupervised alignment data set;
[0077] S34, performing text-speech alignment on the precise marked text features and the precise marked speech features to obtain a standard alignment data set;
[0078] S35. Use the standard alignment data set to perform fine-tuning sampling and updating on the unsupervised alignment data set to obtain an alignment training data set.
[0079] In detail, the precisely marked text feature refers to a text feature in the marked text feature that precisely marks the corresponding dialect audio, and the inaccurately marked text feature refers to a text feature in the marked text feature that does not precisely mark the corresponding dialect audio.
[0080] Specifically, the precisely marked speech feature refers to a speech feature of the marked speech feature that precisely marks the corresponding Mandarin text, and the inaccurately marked speech feature refers to a speech feature of the marked speech feature that does not precisely mark the corresponding Mandarin text.
[0081] In detail, the unsupervised sampling alignment of the imprecisely labeled text features and the imprecisely labeled speech features to obtain an unsupervised aligned data set refers to unsupervised modeling of the imprecisely labeled text features and the imprecisely labeled speech features using an automatic speech autoencoder or a sequence-to-sequence Seq2Seq model to obtain a mapping embedding representation between text features and speech features, and the unsupervised aligned data set is a mapping relationship between text features and speech features.
[0082] Specifically, the text-to-speech alignment refers to determining the mapping relationship between dialect speech features and text features based on the mapping relationship between Mandarin text, text features and dialect speech in the precisely marked text features and the mapping relationship between text features, dialect speech and dialect speech features in the precisely marked speech features, and compiling the corresponding mapping relationships into a standard alignment data set.
[0083] In detail, using the standard aligned dataset to perform fine-tuning sampling and updating on the unsupervised aligned dataset to obtain an aligned training dataset means using the mapping relationship in the standard aligned dataset to perform self-supervised correction on the unsupervised aligned dataset, thereby increasing the data volume of the dataset.
[0084] In an embodiment of the present invention, by utilizing the marked text features to perform mark alignment on the marked speech features to obtain an aligned training data set, semi-supervised relationship mapping learning between marked text features and marked speech features can be achieved, thereby expanding the training set data volume of the dialect speech synthesis model and improving the efficiency of dialect speech synthesis.
[0085] S40: Performing model training on a preset speech synthesis model using the aligned training data set to obtain a primary speech model.
[0086] In detail, the speech synthesis model is a neural network model whose input is text features and whose output is corresponding dialect speech features. The speech synthesis model can be a neural network model such as Tacotron, Transformer TTS and FastSpeech.
[0087] In the embodiment of the present invention, refer to Figure 5 As shown, the method of using the aligned training data set to perform model training on a preset speech synthesis model to obtain a primary speech model includes:
[0088] S41, extracting a text feature training data set and a speech feature training data set from the aligned training data set;
[0089] S42, using a preset speech synthesis model to calculate an analysis speech feature data set corresponding to the text feature training data set;
[0090] S43, calculating the feature loss between the speech feature training data set and the analysis speech feature data set;
[0091] S44. Update the speech synthesis model according to the feature loss to obtain a primary speech model.
[0092] In detail, the text feature training data set is a data set consisting of text features in the aligned training data set, and the speech feature training data set is a data set consisting of speech features in the aligned training data set.
[0093] Specifically, the feature loss between the speech feature training data set and the analysis speech feature data set can be calculated using a mean square error algorithm, a cross entropy loss value algorithm or a KL divergence algorithm (Kullback-Leibler Divergence), wherein the mean square error algorithm is used to calculate the average square difference between the predicted value and the true value, the cross entropy loss value algorithm is used to evaluate the difference between the predicted probability distribution and the target distribution, and the KL divergence algorithm is used to measure the similarity between two probability distributions.
[0094] In detail, the speech synthesis model can be updated according to the feature loss using algorithms such as stochastic gradient descent, Adam or RMSProp to obtain a primary speech model, wherein the RMSProp algorithm is an adaptive learning rate optimization algorithm used to alleviate the problem of unstable learning rate adjustment during gradient descent. The Adam algorithm is an improved version of the RMSProp algorithm, which combines first-order momentum and second-order momentum, and has faster convergence speed and stronger robustness.
[0095] In an embodiment of the present invention, a preset speech synthesis model is trained by using the aligned training data set to obtain a primary speech model, and a corresponding model can be established according to the mapping relationship between the marked text features and the marked speech features, thereby realizing preliminary dialect speech generation.
[0096] S50: Using the aligned training data set to perform cross attention diffusion on the primary speech model to obtain a dialect speech model.
[0097] In detail, the method of using the aligned training data set to perform cross attention diffusion on the primary speech model to obtain a dialect speech model includes:
[0098] Performing cross attention modeling on the aligned training data set to obtain cross attention;
[0099] Performing noise diffusion on the aligned training data set to obtain a noisy training data set;
[0100] Using the primary speech model to perform speech analysis on the noisy training data set to obtain an analysis noisy data set;
[0101] Using the cross attention to perform data denoising on the analysis denoised data set to obtain a denoised training data set;
[0102] The primary speech model is updated using the noise reduction training data set to obtain a dialect speech model.
[0103] In detail, the cross-attention modeling refers to using the attention mechanism to calculate the cross-attention of the text feature training dataset and the speech feature training dataset in the aligned training dataset, and the noise diffusion refers to adding noise to the text feature training dataset and the speech feature training dataset in the aligned training dataset to obtain a noisy text feature dataset and a noisy speech feature dataset, and the noisy text feature dataset and the noisy speech feature dataset are aggregated into a noisy training dataset.
[0104] Specifically, the speech analysis refers to using the primary speech model to generate audio features corresponding to the noisy training data set, and the data denoising refers to restoring the noise data in the analyzed noisy data set to the corresponding original features. The data denoising can be performed using the inverse diffusion method of the denoising loss function.
[0105] In detail, the model update refers to re-training the primary speech model using the noise reduction training data set, updating the parameters of the model, and thus obtaining a dialect speech model.
[0106] In an embodiment of the present invention, by utilizing the aligned training data set to perform cross-attention diffusion on the primary speech model, a dialect speech model is obtained, which can diversify and enhance the training data of the model, effectively improve the generalization ability of the dialect speech model when generating dialect speech, achieve the fusion of multiple dialect speech, and improve the efficiency of dialect speech synthesis.
[0107] S60: Obtain a medical diagnosis reply text, and use the dialect speech model to generate a reply audio corresponding to the medical diagnosis reply text.
[0108] In detail, the medical diagnosis reply text is generated by the intelligent medical diagnosis system based on the patient's oral description of the symptoms through voice recognition to generate a symptom case text, and then the conversational generative artificial intelligence model combines the case data to generate an analysis and guidance text about the patient's case.
[0109] Example description: The patient's voice question is a Cantonese voice "Doctor, I have a very sharp cough recently and a fever. Do I have pneumonia?" The corresponding medical diagnosis reply text is "Based on your description, the cough and fever may be caused by an upper respiratory tract infection, but other possibilities such as bronchitis or pneumonia cannot be ruled out. It is recommended that you go to the hospital as soon as possible for a chest X-ray and blood test to see what is going on with your system. At the same time, remember to drink plenty of water, get enough rest, and don't be too tired. If the fever persists or the symptoms worsen, you must see a doctor as soon as possible."
[0110] Specifically, the step of generating a response audio corresponding to the medical diagnosis response text by using the dialect speech model includes:
[0111] Extracting text features from the medical diagnosis reply text to obtain reply text features;
[0112] Generating a reply audio feature corresponding to the reply text feature using the dialect speech model;
[0113] The reply audio feature is voice decoded to obtain the reply audio.
[0114] In detail, the text feature extraction method is consistent with the method in the above step S10, which will not be repeated here. The speech decoding method is the inverse step of the audio feature extraction method in the above step S20, and speech decoding can be performed using a speech decoder or a waveform generator.
[0115] In the embodiment of the present invention, by using the dialect speech model to generate the response audio corresponding to the medical diagnosis response text, it is possible to implement a medical diagnosis voice response for the dialect, thereby improving the user's medical experience and improving the efficiency of dialect speech synthesis.
[0116] It can be seen that in the above scheme, by performing text feature extraction and text feature labeling on the pre-acquired training text to obtain the labeled text features, the training text data can be converted into a continuous feature vector, thereby simplifying the amount of data calculation and improving the model training efficiency. By performing speech feature extraction and speech feature labeling on the pre-acquired training speech to obtain the labeled speech features, the high-dimensional redundant information in the training speech can be effectively removed, and the rhythm, timbre and intonation features in the dialect speech can be captured, which is helpful for the accurate modeling of the dialect speech and improves the accuracy of subsequent dialect synthesis. By using the labeled text features to label and align the labeled speech features to obtain an aligned training data set, semi-supervised relationship mapping learning between labeled text features and labeled speech features can be realized, thereby expanding the training set data volume of the dialect speech synthesis model and improving the efficiency of dialect speech synthesis.
[0117] By using the aligned training data set to train the preset speech synthesis model, a primary speech model is obtained, and a corresponding model can be established according to the mapping relationship between the marked text features and the marked speech features, thereby realizing preliminary dialect speech generation. By using the aligned training data set to perform cross-attention diffusion on the primary speech model, a dialect speech model is obtained, and the training data of the model can be diversified and enhanced, which can effectively improve the generalization ability of the dialect speech model when generating dialect speech, realize the fusion and communication between multiple dialect speech, and improve the efficiency of dialect speech synthesis. By using the dialect speech model to generate the reply audio corresponding to the medical diagnosis reply text, a medical diagnosis voice reply for the dialect can be realized, thereby improving the user's medical experience and improving the efficiency of dialect speech synthesis.
[0118] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.
[0119] In one embodiment, a dialect speech synthesis device for human-like expression is provided, and the dialect speech synthesis device for human-like expression corresponds one-to-one to the dialect speech synthesis method for human-like expression in the above embodiment. Figure 6 As shown, the dialect speech synthesis device for human-like expression includes a text tagging module 101, an audio tagging module 102, a tag alignment module 103, a model training module 104, a model diffusion module 105 and an audio synthesis module 106. The functional modules are described in detail as follows:
[0120] The text marking module 101 is used to extract and mark text features of pre-acquired training text to obtain marked text features;
[0121] The audio tagging module 102 is used to extract and tag speech features of the pre-acquired training speech to obtain the tagged speech features;
[0122] The tag alignment module 103 is used to perform tag alignment on the tag speech feature using the tag text feature to obtain an alignment training data set;
[0123] The model training module 104 is used to perform model training on a preset speech synthesis model using the aligned training data set to obtain a primary speech model;
[0124] The model diffusion module 105 is used to perform cross attention diffusion on the primary speech model using the aligned training data set to obtain a dialect speech model;
[0125] The audio synthesis module 106 is used to obtain the medical diagnosis reply text and generate a reply audio corresponding to the medical diagnosis reply text using the dialect speech model.
[0126] In one embodiment, when the text marking module 101 performs text feature extraction and text feature marking on the pre-acquired training text to obtain the marked text features, it is used to:
[0127] Performing text segmentation on the pre-acquired training text to obtain a training text word sequence;
[0128] Performing feature encoding on the training text word sequence to obtain a training word feature sequence;
[0129] Performing phoneme mapping on the training word feature sequence to obtain a training phoneme sequence;
[0130] Extracting phoneme features from the training phoneme sequence to obtain a phoneme feature sequence;
[0131] Generate training text features according to the phoneme feature sequence and the training word feature sequence;
[0132] The training text is used to perform audio tagging on the training text features to obtain the tagged text features.
[0133] In one embodiment, when the audio tagging module 102 extracts and tags speech features of the pre-acquired training speech to obtain the tagged speech features, it is used to:
[0134] Performing speech framing and speech windowing on the pre-acquired training speech to obtain the windowed training speech;
[0135] Performing frequency domain conversion on the training speech to obtain a training speech frequency domain signal;
[0136] Performing signal spectrum calculation on the training speech frequency domain signal to obtain a windowed power spectrum signal;
[0137] Performing Mel-cepstrum filtering on the windowed power spectrum signal to obtain a training speech spectrum feature;
[0138] Performing time-domain conversion on the training speech spectrum features to obtain time-domain speech features;
[0139] The time-domain speech features are text-labeled using the training speech to obtain labeled speech features.
[0140] In one embodiment, when the tag alignment module 103 performs tag alignment on the tagged speech feature using the tagged text feature to obtain an alignment training data set, it is used to:
[0141] Splitting the marked text features into precise marked text features and imprecise marked text features;
[0142] Splitting the marked speech features into precise marked speech features and imprecise marked speech features;
[0143] Performing unsupervised sampling alignment on the imprecisely labeled text features and the imprecisely labeled speech features to obtain an unsupervised alignment data set;
[0144] Performing text-speech alignment on the precisely marked text features and the precisely marked speech features to obtain a standard alignment data set;
[0145] The unsupervised alignment dataset is fine-tuned and sampled using the standard alignment dataset to obtain an alignment training dataset.
[0146] In one embodiment, when the model training module 104 performs model training on a preset speech synthesis model using the aligned training data set to obtain a primary speech model, it is used to:
[0147] Extracting a text feature training data set and a speech feature training data set from the aligned training data set;
[0148] Calculating the analysis speech feature data set corresponding to the text feature training data set using a preset speech synthesis model;
[0149] Calculating a feature loss between the speech feature training data set and the analysis speech feature data set;
[0150] The speech synthesis model is updated according to the feature loss to obtain a primary speech model.
[0151] In one embodiment, when the model diffusion module 105 performs cross attention diffusion on the primary speech model using the aligned training data set to obtain the dialect speech model, it is used to:
[0152] Performing cross attention modeling on the aligned training data set to obtain cross attention;
[0153] Performing noise diffusion on the aligned training data set to obtain a noisy training data set;
[0154] Using the primary speech model to perform speech analysis on the noisy training data set to obtain an analysis noisy data set;
[0155] Using the cross attention to perform data denoising on the analysis denoised data set to obtain a denoised training data set;
[0156] The primary speech model is updated using the noise reduction training data set to obtain a dialect speech model.
[0157] In one embodiment, when the audio synthesis module 106 generates the reply audio corresponding to the medical diagnosis reply text using the dialect speech model, it is used to:
[0158] Extracting text features from the medical diagnosis reply text to obtain reply text features;
[0159] Generating a reply audio feature corresponding to the reply text feature using the dialect speech model;
[0160] The reply audio feature is voice decoded to obtain the reply audio.
[0161] The present invention provides a dialect speech synthesis device for human-like expression. The device first extracts text features and marks text features of pre-acquired training text to obtain marked text features, and can convert training text data into continuous feature vectors, thereby simplifying the amount of data calculation and improving the efficiency of model training. The device extracts speech features and marks speech features of pre-acquired training speech to obtain marked speech features, which can effectively remove high-dimensional redundant information in the training speech and can capture the rhythm, timbre and tone features in the dialect speech, which is helpful for accurate modeling of the dialect speech and improves the accuracy of subsequent dialect synthesis. The marked text features are used to mark and align the marked speech features to obtain an aligned training data set, which can realize semi-supervised relationship mapping learning between the marked text features and the marked speech features, thereby expanding the training set data volume of the dialect speech synthesis model and improving the efficiency of dialect speech synthesis.
[0162] By using the aligned training data set to train the preset speech synthesis model, a primary speech model is obtained, and a corresponding model can be established according to the mapping relationship between the marked text features and the marked speech features, thereby realizing preliminary dialect speech generation. By using the aligned training data set to perform cross-attention diffusion on the primary speech model, a dialect speech model is obtained, and the training data of the model can be diversified and enhanced, which can effectively improve the generalization ability of the dialect speech model when generating dialect speech, realize the fusion and communication between multiple dialect speech, and improve the efficiency of dialect speech synthesis. By using the dialect speech model to generate the reply audio corresponding to the medical diagnosis reply text, a medical diagnosis voice reply for the dialect can be realized, thereby improving the user's medical experience and improving the efficiency of dialect speech synthesis.
[0163] For the specific limitations of the dialect speech synthesis device for human-like expression, please refer to the limitations of the intelligent question and answer method above, which will not be repeated here. The various modules in the above-mentioned dialect speech synthesis device for human-like expression can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0164] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it realizes the functions or steps of a dialect speech synthesis method for human-like expression on the server side.
[0165] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of the client side of a dialect speech synthesis method for human-like expression.
[0166] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:
[0167] Performing text feature extraction and text feature labeling on the pre-acquired training text to obtain labeled text features;
[0168] Extracting and marking speech features of the training speech acquired in advance to obtain marked speech features;
[0169] Using the marked text features to perform mark alignment on the marked speech features to obtain an alignment training data set;
[0170] Using the aligned training data set to perform model training on a preset speech synthesis model to obtain a primary speech model;
[0171] Using the aligned training data set to perform cross attention diffusion on the primary speech model to obtain a dialect speech model;
[0172] A medical diagnosis reply text is obtained, and a reply audio corresponding to the medical diagnosis reply text is generated using the dialect speech model.
[0173] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
[0174] Performing text feature extraction and text feature labeling on the pre-acquired training text to obtain labeled text features;
[0175] Extracting and marking speech features of the training speech acquired in advance to obtain marked speech features;
[0176] Using the marked text features to perform mark alignment on the marked speech features to obtain an alignment training data set;
[0177] Using the aligned training data set to perform model training on a preset speech synthesis model to obtain a primary speech model;
[0178] Using the aligned training data set to perform cross attention diffusion on the primary speech model to obtain a dialect speech model;
[0179] A medical diagnosis reply text is obtained, and a reply audio corresponding to the medical diagnosis reply text is generated using the dialect speech model.
[0180] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.
[0181] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0182] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0183] The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should be included in the protection scope of the present invention. It should be noted that if software tools or components other than those of the company appear in the embodiments of this application, they are only used for example introduction and do not represent actual use.
Claims
1. A dialect speech synthesis method for human-like expression, characterized in that: include: Performing text feature extraction and text feature labeling on the pre-acquired training text to obtain labeled text features; Extracting and marking speech features of the training speech acquired in advance to obtain marked speech features; Using the marked text features to perform mark alignment on the marked speech features to obtain an alignment training data set; Using the aligned training data set to perform model training on a preset speech synthesis model to obtain a primary speech model; Using the aligned training data set to perform cross attention diffusion on the primary speech model to obtain a dialect speech model; A medical diagnosis reply text is obtained, and a reply audio corresponding to the medical diagnosis reply text is generated using the dialect speech model.
2. The method for synthesizing dialect speech for human-like expression as claimed in claim 1, characterized in that: The step of extracting and marking text features from the pre-acquired training text to obtain marked text features includes: Performing text segmentation on the pre-acquired training text to obtain a training text word sequence; Performing feature encoding on the training text word sequence to obtain a training word feature sequence; Performing phoneme mapping on the training word feature sequence to obtain a training phoneme sequence; Extracting phoneme features from the training phoneme sequence to obtain a phoneme feature sequence; Generate training text features according to the phoneme feature sequence and the training word feature sequence; The training text is used to perform audio tagging on the training text features to obtain the tagged text features.
3. The method for synthesizing dialect speech for human-like expression as claimed in claim 1, characterized in that: The method of extracting and marking speech features of the pre-acquired training speech to obtain the marked speech features includes: Performing speech framing and speech windowing on the pre-acquired training speech to obtain the windowed training speech; Performing frequency domain conversion on the training speech to obtain a training speech frequency domain signal; Performing signal spectrum calculation on the training speech frequency domain signal to obtain a windowed power spectrum signal; Performing Mel-cepstrum filtering on the windowed power spectrum signal to obtain a training speech spectrum feature; Performing time-domain conversion on the training speech spectrum features to obtain time-domain speech features; The time-domain speech features are text-labeled using the training speech to obtain labeled speech features.
4. The method for synthesizing dialect speech for human-like expression according to claim 1, characterized in that: The step of using the marked text features to perform mark alignment on the marked speech features to obtain an alignment training data set includes: Splitting the marked text features into precise marked text features and imprecise marked text features; Splitting the marked speech features into precise marked speech features and imprecise marked speech features; Performing unsupervised sampling alignment on the imprecisely labeled text features and the imprecisely labeled speech features to obtain an unsupervised alignment data set; Performing text-speech alignment on the precisely marked text features and the precisely marked speech features to obtain a standard alignment data set; The unsupervised alignment dataset is fine-tuned and sampled using the standard alignment dataset to obtain an alignment training dataset.
5. The method for synthesizing dialect speech for human-like expression as claimed in claim 1, characterized in that: The method of using the aligned training data set to perform model training on a preset speech synthesis model to obtain a primary speech model includes: Extracting a text feature training data set and a speech feature training data set from the aligned training data set; Calculating the analysis speech feature data set corresponding to the text feature training data set using a preset speech synthesis model; Calculating a feature loss between the speech feature training data set and the analysis speech feature data set; The speech synthesis model is updated according to the feature loss to obtain a primary speech model.
6. The method for synthesizing dialect speech for human-like expression according to claim 1, characterized in that: The step of performing cross attention diffusion on the primary speech model using the aligned training data set to obtain a dialect speech model comprises: Performing cross attention modeling on the aligned training data set to obtain cross attention; Performing noise diffusion on the aligned training data set to obtain a noisy training data set; Using the primary speech model to perform speech analysis on the noisy training data set to obtain an analysis noisy data set; Using the cross attention to perform data denoising on the analysis denoised data set to obtain a denoised training data set; The primary speech model is updated using the noise reduction training data set to obtain a dialect speech model.
7. The method for synthesizing dialect speech for human-like expression as claimed in claim 1, characterized in that: The step of using the dialect speech model to generate a response audio corresponding to the medical diagnosis response text includes: Extracting text features from the medical diagnosis reply text to obtain reply text features; Generating a reply audio feature corresponding to the reply text feature using the dialect speech model; The reply audio feature is voice decoded to obtain the reply audio.
8. A dialect speech synthesis device for human-like expression, characterized in that: include: A text tagging module is used to extract and tag text features from pre-acquired training texts to obtain tagged text features; An audio tagging module is used to extract and tag speech features of pre-acquired training speech to obtain tagged speech features; A tag alignment module, used to perform tag alignment on the tag speech feature using the tag text feature to obtain an alignment training data set; A model training module, used to perform model training on a preset speech synthesis model using the aligned training data set to obtain a primary speech model; A model diffusion module, used to perform cross attention diffusion on the primary speech model using the aligned training data set to obtain a dialect speech model; The audio synthesis module is used to obtain the medical diagnosis reply text and generate the reply audio corresponding to the medical diagnosis reply text using the dialect speech model.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for synthesizing dialect speech for human-like expression as described in any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for dialect speech synthesis for human-like expression as claimed in any one of claims 1 to 7 is implemented.