Speech synthesis method and device based on side language, computer equipment and medium

Abstract

The invention relates to the technical field of speech synthesis, and particularly discloses a speech synthesis method and device based on a side language, computer equipment and a medium. The prosodic features are obtained through prosodic prediction, the text and the reference acoustic information are effectively fused, the sublingual features are obtained according to the sublingual tag sequence, the target speech is synthesized by cooperatively utilizing the text features, the predicted prosodic features and the sublingual features, the prosodic and sublingual behaviors of the synthesized speech are accurately controlled, and the speech quality is improved. And the naturalness and diversity of the synthesized speech are improved. The method is applied to medical or financial business scenes such as intelligent remote rehabilitation guidance and intelligent repayment reminding, and can generate natural voice which conforms to specific business scenes such as postoperative rehabilitation guidance and financial risk prompt with rich side language behaviors according to the to-be-synthesized text, so that the user experience is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech synthesis, and in particular to a speech synthesis method and device based on paralanguage, a computer device and a medium. BACKGROUND

[0002] In recent years, the TTS (Text to Speech) system based on deep learning has made significant progress in the generation of reading style speech, and the naturalness and fluency of the synthesized speech have approached the level of real people. With the deepening of digital transformation, the text-to-speech technology is widely used in interactive services in the medical and financial industries, such as intelligent remote rehabilitation guidance, intelligent repayment reminders, etc. However, most current TTS systems mainly rely on modeling of prosody, pauses and intonation in training data or reference audio. This approach is acceptable when generating regular reading type speech, but when the application scenario changes from standard reading to vertical fields with high interactivity and high emotional communication requirements, although the speech is clear and accurate, it often lacks systematic modeling of the nonlinear language phenomena commonly found in daily conversations, resulting in a lack of naturalness and diversity in the generated speech, leading to a lack of real communication. For example, when guiding patients after surgery, cold and rigid speech cannot convey care and encouragement, which may reduce patient compliance; in mental health applications, speech lacking emotional fluctuations cannot effectively guide and pacify; when explaining complex financial product terms to customers or reminding them of overdue payments, mechanical and monotonous speech cannot establish trust and may even cause users to feel resentful.

[0003] Therefore, how to improve the naturalness and diversity of synthesized speech has become a problem to be solved. SUMMARY

[0004] The present application provides a speech synthesis method and device based on paralanguage, a computer device and a medium to improve the naturalness and diversity of synthesized speech.

[0005] In a first aspect, the present application provides a speech synthesis method based on paralanguage, the method comprising: obtaining a text to be synthesized and a speaker reference audio; encoding the text to be synthesized to obtain text features, and performing prosody prediction based on the speaker reference audio and the text features to obtain predicted prosody features; obtaining a paralanguage label sequence corresponding to the text to be synthesized, and encoding the paralanguage label sequence to obtain paralanguage features; performing speech synthesis based on the text features, the predicted prosody features and the paralanguage features to generate target speech corresponding to the text to be synthesized.

[0006] In a second aspect, the present application also provides a paralanguage-based speech synthesis device, which comprises: a related data obtaining module, configured to obtain to-be-synthesized text and speaker reference audio; a prosody feature obtaining module, configured to encode the to-be-synthesized text, obtain text features, and perform prosody prediction based on the speaker reference audio and the text features, to obtain predicted prosody features; a paralanguage feature obtaining module, configured to obtain a paralanguage label sequence corresponding to the to-be-synthesized text, and encode the paralanguage label sequence to obtain paralanguage features; a target speech generating module, configured to perform speech synthesis based on the text features, the predicted prosody features, and the paralanguage features, to generate target speech corresponding to the to-be-synthesized text.

[0007] In a third aspect, the present application also provides a computer device, which comprises a memory and a processor; the memory is configured to store a computer program; the processor is configured to execute the computer program and implement the paralanguage-based speech synthesis method as described above when executing the computer program.

[0008] In a fourth aspect, the present application also provides a computer-readable storage medium, which stores a computer program; the computer program, when executed by a processor, causes the processor to implement the paralanguage-based speech synthesis method as described above.

[0009] The present application discloses a paralanguage-based speech synthesis method and device, a computer device, and a medium. The to-be-synthesized text and speaker reference audio are obtained. The to-be-synthesized text is encoded to obtain text features, and prosody prediction is performed based on the speaker reference audio and the text features to obtain predicted prosody features. The paralanguage label sequence corresponding to the to-be-synthesized text is obtained, and the paralanguage label sequence is encoded to obtain paralanguage features. Speech synthesis is performed based on the text features, the predicted prosody features, and the paralanguage features to generate target speech corresponding to the to-be-synthesized text. The prosody features are obtained through prosody prediction, the text and reference acoustic information are effectively fused, the paralanguage features are obtained according to the paralanguage label sequence, and then the text features, the predicted prosody features, and the paralanguage features are cooperatively utilized to synthesize target speech, so that the prosody and paralanguage behavior of the synthesized speech are accurately controlled, and the naturalness and diversity of the synthesized speech are improved. BRIEF DESCRIPTION OF DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative effort.

[0011] Figure 1 is a first schematic flow chart of a paralanguage-based speech synthesis method provided by an embodiment of the present application; Figure 2 is a work flow chart of a speech synthesis system provided by an embodiment of the present application; Figure 3 is a second schematic flow chart of a paralanguage-based speech synthesis method provided by an embodiment of the present application; Figure 4 is a third schematic flow chart of a paralanguage-based speech synthesis method provided by an embodiment of the present application; Figure 5 is a schematic block diagram of a paralanguage-based speech synthesis device provided by an embodiment of the present application; Figure 6 is a structural schematic block diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0012] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort are within the protection scope of the present application.

[0013] The flow charts shown in the drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor are they necessarily executed in the described order. For example, some operations / steps can be decomposed, combined or partially merged, so the actual execution order can be changed according to the actual situation.

[0014] It should be understood that the terms used in the present application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the present application specification and the appended claims, unless otherwise clearly indicated by the context, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0015] It should also be understood that the term "and / or" used in the present application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0016] Embodiments of the present application provide a paralinguistic-based speech synthesis method and device, computer equipment and medium. The paralinguistic-based speech synthesis method can be applied to a server, and the prosody features are obtained through prosody prediction, the text and reference acoustic information are effectively fused, the paralinguistic features are obtained according to the paralinguistic label sequence, and then the target speech is synthesized by cooperatively using the text features, the predicted prosody features and the paralinguistic features, so as to accurately control the prosody and paralinguistic behavior of the synthesized speech and improve the naturalness and diversity of the synthesized speech. The server can be an independent server or a server cluster.

[0017] Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The following embodiments and features in the embodiments can be combined with each other without conflict.

[0018] Please refer to Figure 1 , Figure 1 is a schematic flowchart of a paralinguistic-based speech synthesis method provided by an embodiment of the present application. The paralinguistic-based speech synthesis method can be applied to a server, and is used to obtain prosody features through prosody prediction, effectively fuse text and reference acoustic information, obtain paralinguistic features according to a paralinguistic label sequence, and then cooperatively synthesize target speech by using text features, predicted prosody features and paralinguistic features, so as to accurately control the prosody and paralinguistic behavior of the synthesized speech and improve the naturalness and diversity of the synthesized speech.

[0019] As shown in Figure 1 , the paralinguistic-based speech synthesis method specifically includes steps S101 to S104.

[0020] S101, obtaining to-be-synthesized text and speaker reference audio; In an embodiment, the to-be-synthesized text is natural language text (such as a dialogue sentence, a paragraph, etc.) input by a user and desired to be converted into speech, and can be subjected to basic preprocessing such as error word recognition and label symbol standardization.

[0021] The speaker reference audio is a reference audio segment containing target speaker speech features, and is used to extract speaker prosody style (such as speech speed, intonation and emotional tendency).

[0022] For example, when a medical knowledge popularization video in the form of doctor-patient interaction is generated, the communication dialogue text of the doctor and the patient and the reference audio of the speaker need to be input.

[0023] S102, encoding the to-be-synthesized text to obtain text features, and performing prosody prediction based on the speaker reference audio and the text features to obtain predicted prosody features; In one embodiment, the paralinguistic-based speech synthesis method can be applied in a speech synthesis system, as shown in Figure 2 The speech synthesis system includes a text encoder, a prosody feature predictor, a paralinguistic label predictor, a flow-matching-based acoustic model, and a vocoder. The text encoder is used to encode the text to be synthesized, capturing semantic information, syntactic structure, and contextual relationships, to obtain text features. The prosody feature predictor is used to predict prosody features based on text features and acoustic features. The paralinguistic label predictor is used to analyze text features, predict paralinguistic labels, and encode paralinguistic labels into paralinguistic features according to a paralinguistic behavior encoder. The flow-matching-based acoustic model is used to generate target mel-spectra based on input text features, prosody features, and paralinguistic features. The vocoder is used to convert the target mel-spectra into target speech.

[0024] In one embodiment, the text to be synthesized is encoded by the text encoder to capture semantic information, syntactic structure, and contextual relationships, to obtain text features.

[0025] In one embodiment, the basic prosodic features (such as mel-spectra, F0 (fundamental frequency), energy) and phoneme time features of the target speaker are extracted from the speaker reference audio. The basic prosodic features and phoneme time features are uniformly mapped into the same embedding space for splicing to obtain acoustic features.

[0026] The text features and acoustic features are transmitted to the prosody predictor, and the prosody predictor is used to predict prosody, to obtain predicted prosody features.

[0027] S103, obtaining a paralinguistic label sequence corresponding to the text to be synthesized, and encoding the paralinguistic label sequence to obtain paralinguistic features; In one embodiment, the paralinguistic type includes speech behavior features: pauses (unvoiced / voiced), filler words ("um" "uh"), self-corrections, repetitions, etc. (strongly related to language structure), and obvious vocal features: laughter, breathing, coughing, speech rate changes, accent variations, etc. (strongly related to acoustic characteristics).

[0028] In one embodiment, the paralinguistic label can be obtained by explicit label input, i.e. the user can manually specify the paralinguistic label (such as <laughter> <pause: 0.3s>); it can also be automatically predicted, i.e. the paralinguistic label predictor is used to predict paralinguistic labels based on text features, to generate a paralinguistic label sequence that meets the context (such as predicting laughter based on the "happy" semantic).

[0029] In one embodiment, a paralinguistic behavior encoder is used to map each paralinguistic label (such as "laughter" or "pause") to a fixed-dimensional vector through label embedding, thereby converting discrete labels into continuous feature vectors to obtain paralinguistic features for use by the acoustic model.

[0030] S104. Based on the text features, the predicted prosodic features, and the paralinguistic features, speech synthesis is performed to generate the target speech corresponding to the text to be synthesized.

[0031] In one embodiment, text features, predicted prosodic features, and paralinguistic features are concatenated into a joint conditional vector, which is then used as input to the flow matching acoustic model.

[0032] Based on the stream-matching acoustic model, the joint conditional vector is mapped to the Mel spectrum. A vocoder based on a neural network converts the Mel spectrum output by the stream-matching acoustic model into a time-domain waveform signal. The waveform details (such as the spectral jitter of laughter and the energy attenuation of breathing sounds) are dynamically adjusted in combination with paralinguistic features to obtain anthropomorphic speech containing natural rhythm and paralinguistic details (such as laughter and pauses), i.e., the target speech.

[0033] Further, step S104 includes: mapping and concatenating the text features, the predicted prosodic features, and the paralinguistic features to obtain a conditional vector, and performing vector field estimation based on the conditional vector to generate a target Mel spectrum; and converting the target Mel spectrum based on a preset vocoder to generate the target speech.

[0034] In one embodiment, such as Figure 2 As shown, text features, predicted prosodic features, and paralinguistic features are mapped to the same dimension and concatenated to jointly generate a conditional vector, which is then transmitted to the stream matching acoustic model.

[0035] In one embodiment, the flow matching mechanism learns a continuous-time vector field guided by a conditional vector, progressively converting random noise into the target Mel spectrum. Specifically, at t=0, the input is a random noise vector z0 that follows a standard normal distribution. At time step t∈[0,1], the flow matching acoustic model predicts the current noise z0 using a vector field estimation network. t gradient v(z) t (z, t, c), where c is the condition vector. Using the Euler method, the noise is iteratively updated to minimize the L2 distance between the predicted vector field and the true vector field: z t+1 = z t +ε·v(z t ,t,c) (ε is the step size, usually taken as 100-200 steps); when t=1, z1 converges to the target Mel spectrum, which contains the frequency energy distribution of speech.

[0036] In one embodiment, a vocoder is used to perform dynamic range compression and mean-variance normalization on the target Mel spectrum output by the acoustic model to match the training data distribution; the initial phase is estimated from the Mel spectrum using the Griffin-Lim algorithm to assist in waveform generation stability.

[0037] In one embodiment, for paralinguistic features (such as breathing sounds or laughter), the vocoder enhances realism by detecting the location of the paralinguistic tag in the Mel spectrum (such as a sudden increase in energy in the frequency band corresponding to <laughter>) and superimposing low-intensity Gaussian noise (with variance controlled by the confidence level of the paralinguistic feature) at the corresponding time-domain location.

[0038] In another embodiment, a multi-scale discriminator and a multi-period discriminator can be used to supervise the authenticity of the generated waveform from the time domain and frequency domain, respectively.

[0039] In the above embodiments, the end-to-end generation of high-fidelity speech from multimodal features is achieved through a conditional vector-guided stream matching mechanism and a sub-language enhancement vocoder, thereby improving the naturalness of the synthesized speech.

[0040] Please see Figure 3 , Figure 3 This is a schematic flowchart illustrating a paralinguistic-based speech synthesis method provided in an embodiment of this application. This paralinguistic-based speech synthesis method can be applied in a server to generate context-appropriate paralinguistic sequences through manual input of labels or automatic prediction by a label predictor. In subsequent speech synthesis, paralinguistic features are used to precisely control paralinguistic behaviors such as pauses, laughter, and filler words, improving the naturalness and diversity of the synthesized speech. Furthermore, the paralinguistic label acquisition mechanism, employing both explicit labeling and automated label prediction, avoids reliance on manual annotation, thereby improving controllability and flexibility.

[0041] like Figure 3 As shown, the speech synthesis method based on paralinguistics specifically includes steps S201 to S202.

[0042] S201. Based on a preset label predictor, perform sub-language prediction on the text features to obtain the sub-language label sequence; In one embodiment, a label predictor is obtained by using conversational text (such as casual conversation or interview transcripts) with manually annotated sub-language tags as training data and by performing supervised training with the goal of minimizing the cross-entropy loss between the predicted label sequence and the real label sequence.

[0043] In one embodiment, the text features are encoded using a trained label predictor to trigger label prediction, obtain at least one sub-language label, and generate a sub-language label sequence.

[0044] Further, the step of predicting the sub-language based on the text features using a preset label predictor to obtain the sub-language label sequence includes: analyzing the text features based on the label predictor to obtain a sub-sequence of sub-language behavior labels and a sub-sequence of sub-language vocalization labels; identifying conflicts between the sub-sequence of sub-language behavior labels and the sub-sequence of sub-language vocalization labels to determine conflicting sub-languages; matching the conflict resolution strategy corresponding to the conflicting sub-language in a preset conflict resolution rule base and performing conflict processing based on the conflict resolution strategy; and performing time alignment and splicing on the processed sub-sequence of sub-language behavior labels and the sub-sequence of sub-language vocalization labels to generate the predicted sub-language label sequence.

[0045] In one embodiment, the label predictor encodes text features (semantics, sentiment, grammatical structure) and then generates two types of paralinguistic label subsequences through dual-branch decoding: paralinguistic behavior label subsequences and paralinguistic vocalization label subsequences.

[0046] The label predictor triggers paralinguistic behavior label prediction based on text grammatical structure (such as punctuation marks and word spacing) and hesitant expressions (such as "um..." "let me think"), obtaining a subsequence of paralinguistic behavior labels, including speech behavior features such as silent / vocal pauses, filler words, elongated sounds, self-correction, etc. The output format is [(behavior label, start time, duration, confidence), ...], for example: [("silent pause", 1.2s, 0.3s, 0.92), ("filler word:uh", 1.5s, 0.2s, 0.85), ...].

[0047] The label predictor predicts paralinguistic vocalization labels based on text sentiment (e.g., the positive word "happy" triggers laughter) and acoustic context (e.g., breathing patterns in reference audio), obtaining a subsequence of paralinguistic vocalization labels, including obvious vocalization features such as laughter, breathing sounds, throat clearing, and speech rate changes. The output format is [(vocalization label, start time, duration, confidence), ...], for example: [("laughter", 2.0s, 0.8s, 0.78), ("breathing sounds", 3.5s, 0.3s, 0.65),...].

[0048] In one embodiment, conflict identification includes temporal overlap detection and semantic contradiction identification. Conflicts include temporal overlap conflict: the time intervals of two labels overlap (such as predicting "silent pause" and "breathing sound" at the same time interval), and semantic contradiction conflict: the label types are semantically mutually exclusive (such as "rapid speech" and "prolonged hesitant tone" appearing at the same time).

[0049] In one embodiment, the conflict resolution rule base includes priority rules, confidence rules, and context adaptation rules. For example, the priority rules are: vocal feature labels > speech behavior labels (e.g., "laughter" takes precedence over "pause"); emotion-related labels > neutral labels (e.g., "excited laughter" takes precedence over "normal breathing"); confidence rules: retain labels with confidence scores higher than a threshold (e.g., 0.6), and if all are higher than the threshold, take the label with the highest confidence score (e.g., "laughter: 0.85" eliminates "pause: 0.72"); context adaptation rules: correct conflicts based on text semantics, such as retaining the "rising tone" label at the end of an interrogative sentence and eliminating the "falling tone pause"; retaining "laughter" in positive text and eliminating "hesitation filler words".

[0050] In a specific embodiment, the conflicting sub-language is a conflicting label pair, such as (laughter, 0.5-1.0s, confidence 0.85) and (pause, 0.7-1.2s, confidence 0.72). Priority rules (vocal features take precedence) and confidence rules (laughter has higher confidence) are triggered. The conflict resolution result is to retain the "laughter" label and truncate the "pause" label to a non-overlapping portion (pause, 1.0-1.2s, 0.72).

[0051] In one embodiment, the labels of the two types of subsequences are mapped to a unified time scale based on the phoneme time axis output by the text encoder. The time offset of the labels after conflict handling is fine-tuned to avoid overlapping label boundaries (e.g., according to the above conflict handling results, the start time of "pause" needs to be delayed to the end time of "laughter").

[0052] Merge the processed behavior label subsequence and vocalization label subsequence in chronological order to obtain a predicted paralinguistic label sequence aligned with the length of the text sequence, in the format [(label type, start frame, end frame, confidence), ...], for example: [("laughter", 50-100 frames, 0.85), ("pause", 100-120 frames, 0.72), ("silence", 120-150 frames, 1.0), ...].

[0053] Furthermore, obtaining the sub-language tag sequence corresponding to the text to be synthesized further includes: obtaining the sub-language annotation corresponding to the text to be synthesized input by the user; extracting and verifying the sub-language annotation to obtain at least one valid sub-language tag, and generating the sub-language tag sequence.

[0054] In one embodiment, the display label input format may be to embed predefined labels in the text to be synthesized, for example, "Your account balance <pause: 0.3s> is 5000 yuan <emphasis>, please repay in time <warning tone>".

[0055] In one embodiment, for explicit tag input, regular expressions are used to separate tags from the text and parse them into structured data. Specifically, for base tags without attributes, the type is directly extracted, such as <laughter> → {"type": "laughter", "attribute": {}}; for tags with attributes, the type and attribute are separated by a colon, such as <pause:0.5s> which is parsed as {"type": "pause", "attribute": {"duration": "0.5s"}}, and multiple attributes are separated by semicolons (such as <speech rate:slow;volume:high>).

[0056] In one embodiment, the extracted tags are validated to filter out invalid tags (such as <unknown tags>). Validation includes, but is not limited to, attribute validity validation: checking whether the attribute values ​​conform to preset rules (such as the duration must be a non-negative number plus the unit "s", and the speaking speed must be "fast / medium / slow").

[0057] In one embodiment, a default timestamp is predicted based on the text length and phoneme duration, and the tags are sorted in ascending order of their start time to generate an ordered sequence, thus obtaining a sub-language tag sequence.

[0058] S202. Based on a preset behavior encoder, the predicted sub-language label sequence is mapped to obtain the sub-language features.

[0059] In one embodiment, the behavior encoder uses a label embedding approach, mapping each sub-language label to a fixed-dimensional vector representation.

[0060] Specifically, a learnable embedding vector is assigned to each aligned sublinguistic label. The vector space is dynamically adjusted through training to bring semantically similar labels (such as "laughter" and "happy") closer together. A 1D convolutional layer is used to smooth the embedding vector sequence, eliminating feature jumps during label switching (such as transitioning from laughter to normal speech). Finally, a sublinguistic feature sequence aligned with the time steps of text features and prosodic features is generated, containing information on the type, intensity, and duration of sublinguistic behaviors.

[0061] In the above embodiments, paralinguistic sequences that conform to the context are generated by manually inputting display tags or by automatically predicting them through a tag predictor. In the subsequent speech synthesis process, paralinguistic behaviors such as pauses, laughter, and filler words are precisely controlled through paralinguistic features, which improves the naturalness and diversity of synthesized speech. Furthermore, the paralinguistic tag acquisition mechanism, which uses both explicit tags and automatically predicted tags, avoids dependence on manual annotation, thereby improving controllability and flexibility.

[0062] Please see Figure 4 , Figure 4This is a schematic flowchart illustrating a paralinguistic-based speech synthesis method provided in an embodiment of this application. This paralinguistic-based speech synthesis method can be applied in a server to effectively fuse text and reference acoustic information through a prosody predictor, achieving high-quality and efficient prosody transfer, ensuring the natural continuity of the generated speech in terms of pitch, energy, and duration, and improving the naturalness of the synthesized speech.

[0063] like Figure 4 As shown, the speech synthesis method based on paralinguistics specifically includes steps S301 to S302.

[0064] S301. Extract prosodic features and phoneme features from the speaker reference audio to obtain basic prosodic features and phoneme time features, and concatenate the basic prosodic features and the phoneme time features to obtain acoustic features. In one embodiment, such as Figure 2 As shown, basic prosodic features reflecting speech prosodic style are extracted from the speaker's reference audio, including Mel spectrum, F0 (fundamental frequency), and energy. Phoneme-level temporal duration information is obtained through phoneme alignment.

[0065] Specifically, a short-time Fourier transform is used to segment the speaker's reference audio into frames, converting the spectrum to a Mel scale and outputting a Mel spectrum matrix that reflects the frequency energy distribution. The fundamental frequency is extracted frame by frame using the PyWorld toolkit, and a dynamic programming algorithm is used to eliminate harmonic and half-frequency errors, resulting in a smooth F0 curve, which is then converted to a logarithmic scale to conform to human auditory perception. The root mean square energy of each frame of audio signal is calculated, normalized, and used as an energy feature to reflect the intensity variations of speech. Finally, a basic prosodic feature matrix containing the Mel spectrum, F0 (fundamental frequency), and energy is obtained.

[0066] In one embodiment, phoneme-level temporal duration information is obtained through phoneme alignment to achieve accurate mapping between speech and text. Specifically, the MFA (Montreal Forced Aligner) tool is used, with reference audio and its corresponding text annotations (phoneme sequence) as input. A Hidden Markov Model is used to force the alignment of phonemes with audio frames, obtaining the start time, end time, duration (in seconds) of each phoneme, as well as the frame index at the phoneme boundary, in the format [(phoneme1, start frame, end frame, duration), (phoneme2, start frame, end frame, duration), ...].

[0067] In one embodiment, basic prosodic features (Mel spectrum, F0, energy) and phoneme temporal features are uniformly mapped to the same embedding space. The two types of features are aligned in the temporal dimension (ensuring that each frame of audio corresponds to a unique phoneme temporal feature), and then concatenated in the feature dimension to form an acoustic feature matrix.

[0068] S302. Based on a preset prosodic predictor, the acoustic features and the text features are processed to obtain the predicted prosodic features.

[0069] In one embodiment, the prosody predictor is based on a Perceiver IO architecture, which consists of an encoder, a processor, and a decoder. Each module operates with a global query-key-value (QKV) attention operation, followed by a multilayer perceptron. Both the encoder and decoder are connected to two input matrices: the first is used as the key and value, and the second is used as the query. The output features of each module have the same shape as the query. The encoder's query is a learnable query feature, initialized as an all-one vector.

[0070] Prosodic features are predicted by a prosodic predictor, which includes phoneme-by-phone / frame-by-frame prosodic parameters of the audio to be synthesized. These features are used to guide the subsequent acoustic model to generate natural speech rhythm and intonation.

[0071] Further, the process of processing the acoustic features and the text features based on the preset prosodic predictor to obtain the predicted prosodic features includes: performing cross-attention processing on the preset initial query features and the acoustic features based on the encoder of the prosodic predictor to generate initial latent features; performing self-attention processing on the initial latent features based on the processor of the prosodic predictor to generate target latent features; and performing cross-attention processing on the target latent features and the text features based on the decoder of the prosodic predictor to obtain the predicted prosodic features.

[0072] In one embodiment, the initial query feature is a learnable parameter matrix initialized as an all-one vector with shape [N, D], which actively extracts the most critical information from the acoustic features as the Query.

[0073] The encoder performs cross-attention computation on the acoustic features using the initial query features. Specifically, each initial query vector (a row in [N, D]) is compared for similarity with all frames in the acoustic features. A set of attention weights is calculated based on the similarity, indicating which parts of the acoustic features each query vector should focus on. These weights are then used to perform a weighted summation of the acoustic features (Value) to generate a summarized vector, which is the initial latent feature.

[0074] Each initial query vector extracts the most relevant global prosodic information from the entire reference audio. For example, one query vector might be specifically responsible for extracting average speech rate information, while another is responsible for the overall pitch range. Therefore, the initial latent features are a condensed representation or essence summary of the prosodic style of the entire reference audio, compressing the variable-length input into a fixed-size representation, greatly reducing the amount of subsequent computation.

[0075] In one embodiment, initial latent features are passed to a processor consisting of multiple (e.g., eight) processing layers, each of which performs self-attention computation on the input.

[0076] Specifically, the initial latent features input to the processor serve simultaneously as the Query, Key, and Value. Each latent vector interacts with all other latent vectors, calculating attention weights. Through weighted summation, each vector gathers information from other vectors and updates itself, ultimately yielding the target latent features. The target latent vector is a deeply refined and fused latent representation, not merely a summary of the original information, but a high-quality prosodic style representation containing complex relationships between information. In one embodiment, text features and target latent vectors are input into a decoder, with the text features serving as the query and the target latent features serving as the key and value. The decoder then performs cross-attention calculation on the query (text features) and the key and value (target latent features) to obtain the predicted prosodic features. In the above embodiments, the prosody predictor can effectively integrate text and reference acoustic information, achieving high-quality and high-efficiency prosody transfer, ensuring the natural continuity of generated speech in terms of pitch, energy, duration, etc., and improving the naturalness of synthesized speech. Please see Figure 5 , Figure 5 This application provides a schematic block diagram of a paralinguistic speech synthesis device, which is used to execute the aforementioned paralinguistic speech synthesis method. The paralinguistic speech synthesis device can be configured on a server.

[0077] like Figure 5 As shown, the paralinguistic-based speech synthesis device 400 includes: The relevant data acquisition module 401 is used to acquire the text to be synthesized and the speaker's reference audio; The prosodic feature acquisition module 402 is used to encode the text to be synthesized, obtain text features, and perform prosodic prediction based on the speaker reference audio and the text features to obtain predicted prosodic features. The sub-language feature acquisition module 403 is used to acquire the sub-language tag sequence corresponding to the text to be synthesized, and to encode the sub-language tag sequence to obtain sub-language features; The target speech generation module 404 is used to perform speech synthesis based on the text features, the predicted prosodic features and the paralinguistic features to generate the target speech corresponding to the text to be synthesized.

[0078] Furthermore, the sub-language feature acquisition module 403 includes: The sub-language prediction unit is used to perform sub-language prediction on the text features based on a preset label predictor to obtain the sub-language label sequence; The label mapping unit is used to map the predicted sub-language label sequence based on a preset behavior encoder to obtain the sub-language features.

[0079] Furthermore, the sub-language prediction unit includes: The subsequence is obtained by sub-units, which are used to analyze the text features based on the label predictor to obtain subsequences of sub-language behavior labels and subsequences of sub-language vocalization labels; The conflict identification subunit is used to identify conflicts between the sub-language behavior label subsequence and the sub-language occurrence label subsequence, and determine the conflicting sub-language; The conflict handling subunit is used to match the conflict resolution strategy corresponding to the conflict sub-language in a preset conflict resolution rule base, and to perform conflict handling based on the conflict resolution strategy. The sequence splicing subunit is used to perform time alignment and splicing of the processed sub-language behavior label sub-sequence and the sub-language vocalization label sub-sequence to generate the predicted sub-language label sequence.

[0080] Furthermore, the prosodic feature acquisition module 402 includes: The acoustic feature acquisition unit is used to extract prosodic features and phoneme features from the speaker reference audio, obtain basic prosodic features and phoneme time features, and concatenate the basic prosodic features and the phoneme time features to obtain acoustic features. The prosodic feature prediction unit is used to process the acoustic features and the text features based on a preset prosodic predictor to obtain the predicted prosodic features.

[0081] Furthermore, the prosodic feature prediction unit includes: The initial latent feature acquisition subunit is used to perform cross-attention processing on the preset initial query features and the acoustic features based on the encoder of the prosody predictor to generate initial latent features; The target latent feature generation subunit is used to perform self-attention processing on the initial latent features based on the processor of the prosody predictor to generate target latent features; The predictive prosodic feature acquisition subunit is used to perform cross-attention processing on the target latent features and the text features based on the decoder of the prosodic predictor to obtain the predicted prosodic features.

[0082] Furthermore, the target speech generation module 404 includes: The target Mel spectrum generation unit is used to map and concatenate the text features, the predicted prosodic features, and the paralinguistic features to obtain a conditional vector, and to perform vector field estimation based on the conditional vector to generate the target Mel spectrum. The target speech generation unit is used to convert the target Mel spectrum based on a preset vocoder to generate the target speech.

[0083] Furthermore, the sub-language feature acquisition module 403 further includes: The sub-language annotation acquisition unit is used to acquire the sub-language annotations corresponding to the text to be synthesized input by the user; The tag extraction and verification unit is used to extract and verify the sub-language annotations, obtain at least one valid sub-language tag, and generate the sub-language tag sequence.

[0084] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described apparatus and modules can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0085] The aforementioned device can be implemented as a computer program, which can be used in, for example... Figure 6 It runs on the computer device shown.

[0086] Please see Figure 6 , Figure 6 This is a schematic block diagram illustrating the structure of a computer device according to an embodiment of this application. The computer device may be a server.

[0087] See Figure 6 The computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0088] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform any language-based speech synthesis method.

[0089] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0090] Internal memory provides an environment for the execution of computer programs in non-volatile storage media. When the computer program is executed by the processor, it enables the processor to execute any speech synthesis method based on a paralinguistic language.

[0091] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0092] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0093] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps: Obtain the text to be synthesized and the speaker's reference audio; The text to be synthesized is encoded to obtain text features, and prosodic prediction is performed based on the speaker reference audio and the text features to obtain predicted prosodic features; Obtain the sub-language tag sequence corresponding to the text to be synthesized, and encode the sub-language tag sequence to obtain sub-language features; Speech synthesis is performed based on the text features, the predicted prosodic features, and the paralinguistic features to generate the target speech corresponding to the text to be synthesized.

[0094] In one embodiment, when the processor acquires the sub-language tag sequence corresponding to the text to be synthesized and encodes the sub-language tag sequence to obtain sub-language features, it is configured to: Based on a preset label predictor, the text features are used to perform sub-language prediction to obtain the sub-language label sequence; The predicted sub-language label sequence is mapped based on a preset behavior encoder to obtain the sub-language features.

[0095] In one embodiment, when the processor performs sub-language prediction on the text features based on a preset label predictor to obtain the sub-language label sequence, it is configured to: Based on the label predictor, the text features are analyzed to obtain subsequences of paralinguistic behavior labels and subsequences of paralinguistic vocalization labels; Conflict identification is performed on the subsequence of sub-language behavior labels and the subsequence of sub-language occurrence labels to determine the conflicting sub-language; The conflict resolution strategy corresponding to the conflict sub-language is matched with the pre-defined conflict resolution rule base, and conflict processing is performed based on the conflict resolution strategy. The processed subsequence of secondary language behavior labels and the subsequence of secondary language vocalization labels are time-aligned and concatenated to generate the predicted secondary language label sequence.

[0096] In one embodiment, when the processor performs prosodic prediction based on the speaker reference audio and the text features to obtain predicted prosodic features, it is configured to: Prosodic features and phoneme features are extracted from the speaker reference audio to obtain basic prosodic features and phoneme temporal features. The basic prosodic features and phoneme temporal features are then concatenated to obtain acoustic features. The acoustic features and text features are processed based on a preset prosodic predictor to obtain the predicted prosodic features.

[0097] In one embodiment, when the processor processes the acoustic features and the text features based on a preset prosodic predictor to obtain the predicted prosodic features, it is configured to: The encoder based on the prosody predictor performs cross-attention processing on the preset initial query features and the acoustic features to generate initial latent features; The processor based on the prosody predictor performs self-attention processing on the initial latent features to generate target latent features; The decoder based on the prosody predictor performs cross-attention processing on the target latent features and the text features to obtain the predicted prosody features.

[0098] In one embodiment, when the processor performs speech synthesis based on the text features, the predicted prosodic features, and the paralinguistic features to generate the target speech corresponding to the text to be synthesized, it is configured to: The text features, the predicted prosodic features, and the paralinguistic features are mapped and concatenated to obtain a conditional vector, and a vector field estimation is performed based on the conditional vector to generate the target Mel spectrum. Based on a preset vocoder, the target Mel spectrum is converted to generate the target speech.

[0099] In one embodiment, when the processor acquires the sub-language tag sequence corresponding to the text to be synthesized, it is further configured to: Obtain the sub-language annotations corresponding to the text to be synthesized input by the user; The sub-language annotations are extracted and verified to obtain at least one valid sub-language tag, and the sub-language tag sequence is generated.

[0100] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement any of the paralinguistic speech synthesis methods provided in the embodiments of this application.

[0101] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.

[0102] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A speech synthesis method based on paralinguistics, characterized in that, include: Obtain the text to be synthesized and the speaker's reference audio; The text to be synthesized is encoded to obtain text features, and prosodic prediction is performed based on the speaker reference audio and the text features to obtain predicted prosodic features; Obtain the sub-language tag sequence corresponding to the text to be synthesized, and encode the sub-language tag sequence to obtain sub-language features; Speech synthesis is performed based on the text features, the predicted prosodic features, and the paralinguistic features to generate the target speech corresponding to the text to be synthesized.

2. The speech synthesis method based on paralinguistics according to claim 1, characterized in that, The step of obtaining the sub-language tag sequence corresponding to the text to be synthesized, and encoding the sub-language tag sequence to obtain sub-language features, includes: Based on a preset label predictor, the text features are used to perform sub-language prediction to obtain the sub-language label sequence; The predicted sub-language label sequence is mapped based on a preset behavior encoder to obtain the sub-language features.

3. The speech synthesis method based on paralinguistics according to claim 2, characterized in that, The method of performing paralinguistic prediction on the text features based on a preset label predictor to obtain the paralinguistic label sequence includes: Based on the label predictor, the text features are analyzed to obtain subsequences of paralinguistic behavior labels and subsequences of paralinguistic vocalization labels; Conflict identification is performed on the subsequence of sub-language behavior labels and the subsequence of sub-language occurrence labels to determine the conflicting sub-language; The conflict resolution strategy corresponding to the conflict sub-language is matched with the pre-defined conflict resolution rule base, and conflict processing is performed based on the conflict resolution strategy. The processed subsequence of secondary language behavior labels and the subsequence of secondary language vocalization labels are time-aligned and concatenated to generate the predicted secondary language label sequence.

4. The speech synthesis method based on paralinguistics according to claim 1, characterized in that, The step of prosodic prediction based on the speaker reference audio and the text features to obtain predicted prosodic features includes: Prosodic features and phoneme features are extracted from the speaker reference audio to obtain basic prosodic features and phoneme temporal features. The basic prosodic features and phoneme temporal features are then concatenated to obtain acoustic features. The acoustic features and text features are processed based on a preset prosodic predictor to obtain the predicted prosodic features.

5. The speech synthesis method based on paralinguistics according to claim 4, characterized in that, The method of processing the acoustic features and text features based on a preset prosodic predictor to obtain the predicted prosodic features includes: The encoder based on the prosody predictor performs cross-attention processing on the preset initial query features and the acoustic features to generate initial latent features; The processor based on the prosody predictor performs self-attention processing on the initial latent features to generate target latent features; The decoder based on the prosody predictor performs cross-attention processing on the target latent features and the text features to obtain the predicted prosody features.

6. The speech synthesis method based on paralinguistics according to claim 1, characterized in that, The step of generating target speech corresponding to the text to be synthesized based on the text features, the predicted prosodic features, and the paralinguistic features includes: The text features, the predicted prosodic features, and the paralinguistic features are mapped and concatenated to obtain a conditional vector, and a vector field estimation is performed based on the conditional vector to generate the target Mel spectrum. Based on a preset vocoder, the target Mel spectrum is converted to generate the target speech.

7. The speech synthesis method based on paralinguistics according to any one of claims 1 to 6, characterized in that, The step of obtaining the sub-language tag sequence corresponding to the text to be synthesized further includes: Obtain the sub-language annotations corresponding to the text to be synthesized input by the user; The sub-language annotations are extracted and verified to obtain at least one valid sub-language tag, and the sub-language tag sequence is generated.

8. A speech synthesis device based on paralinguistics, characterized in that, include: The relevant data acquisition module is used to acquire the text to be synthesized and the speaker's reference audio; The prosodic feature acquisition module is used to encode the text to be synthesized, obtain text features, and perform prosodic prediction based on the speaker reference audio and the text features to obtain predicted prosodic features. The sub-language feature acquisition module is used to acquire the sub-language tag sequence corresponding to the text to be synthesized, and to encode the sub-language tag sequence to obtain sub-language features; The target speech generation module is used to perform speech synthesis based on the text features, the predicted prosodic features and the paralinguistic features to generate the target speech corresponding to the text to be synthesized.

9. A computer device, characterized in that, The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and, in executing the computer program, implement the speech synthesis method based on a sub-language as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to implement the speech synthesis method based on a secondary language as described in any one of claims 1 to 7.

Images