Speech synthesis method and device, equipment and storage medium

Abstract

The invention belongs to the technical field of artificial intelligence, and relates to a speech synthesis method and device, equipment and a storage medium, and the method comprises the steps: obtaining all predicted paragraph tags and the initial insertion information of each paragraph tag; inputting the to-be-synthesized voice text data and all the auxiliary language tags into an improved voice synthesis model; semantic features, emotion features and dynamic rhythm features are extracted to serve as context features to be input into a sublingual tag insertion self-adaptive adjustment component, and sublingual tag insertion real-time adjustment information is obtained; and dynamically inserting the target secondary language into the real-time synthesis voice sequence based on the label insertion real-time adjustment information and the initial insertion information to complete target voice synthesis. When the method is applied to a consultation reply scene in financial science and technology business or health medical business, prediction and insertion of the side language are combined with more features except text features, so that the synthesized voice has higher quality in naturalness, reliability and scene adaptability.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology and is applied to the scenario of speech synthesis for AI question-answering agents. It relates to a speech synthesis method, device, equipment and storage medium. Background Technology

[0002] Paralinguistic vocalizations, such as nonverbal sounds like laughter and breathing, and lexicalized interjections like "um" and "oh," are widely present in spontaneous speech. These cues, especially interjections, often encode emotions, intentions, and speaker states beyond the literal lexical content through unique rhythms, enhancing expressiveness. Existing speech recognition and speech synthesis systems have significant technical limitations. Especially in communication-heavy scenarios like finance and healthcare consultations, the introduction of paralinguistic synthesis can provide customers with higher-quality speech responses.

[0003] Currently, in financial business scenarios with a high volume of communication and in health and medical consultation services, paralanguage recognition relies on a single audio modality and lacks multimodal context fusion, including text semantics and scene features. This leads to a decrease in recognition accuracy in complex scenarios, especially for paralanguage categories with close semantic connections. Moreover, the introduction of paralanguage and its connection to the prosodic context appear abrupt, lacking a real-time adaptive adjustment mechanism and making it difficult to match dynamic conversational scenarios. These shortcomings result in low quality of synthesized speech in terms of naturalness, reliability, and scene adaptability. Summary of the Invention

[0004] The purpose of this application is to provide a speech synthesis method, apparatus, device, and storage medium to solve the technical problem that the quality of existing synthesized speech is low in terms of naturalness, reliability, and scene adaptability.

[0005] In a first aspect, embodiments of this application provide a speech synthesis method, which employs the following technical solution: A speech synthesis method includes the following steps: Acquire multimodal material data for speech synthesis, wherein the multimodal material data includes speech-text data to be synthesized; The multimodal material data is subjected to feature encoding processing, and input data in the target format for sub-language insertion prediction is generated; The input data is fed into a pre-trained sub-language insertion prediction model to obtain all sub-language labels output by the prediction model and the initial insertion information for each sub-language label. The speech text data to be synthesized and all the sub-language tags are input into the improved speech synthesis model, wherein the improved speech synthesis model introduces a sub-language tag insertion adaptive adjustment component, which is used to adjust the insertion information of sub-language tags based on real-time synthesis context information; Extract the semantic and sentiment features of the speech-text data to be synthesized; The improved speech synthesis model outputs pre-synthesized speech in real time, and the dynamic prosodic features of the pre-synthesized speech are extracted. The semantic features, the emotional features, and the dynamic prosodic features are input as contextual features into the sub-language label insertion adaptive adjustment component containing a multilayer perceptron to obtain the real-time label insertion adjustment information output by the sub-language label insertion adaptive adjustment component; Based on the real-time adjustment information of the tag insertion and the initial insertion information of each sub-language tag, the target sub-language is dynamically inserted into the real-time synthesized speech sequence to complete the target speech synthesis.

[0006] Secondly, embodiments of this application also provide a speech synthesis device, which adopts the technical solution described below: A speech synthesis device, comprising: The synthesis material data acquisition module is used to acquire multimodal material data to be synthesized, wherein the multimodal material data includes the speech-text data to be synthesized; The prediction input data generation module is used to perform feature encoding processing on the multimodal material data and generate input data in the target format for sub-language insertion prediction. The initial insertion information acquisition module is used to input the input data into a pre-trained sub-language insertion prediction model and acquire all sub-language labels output by the prediction model and the initial insertion information of each sub-language label. The speech synthesis input module is used to input the speech text data to be synthesized and all the sub-language tags into the improved speech synthesis model. The improved speech synthesis model introduces a sub-language tag insertion adaptive adjustment component, which is used to adjust the insertion information of sub-language tags based on real-time synthesis context information. The text data feature extraction module is used to extract the semantic and emotional features of the speech-text data to be synthesized; The dynamic prosodic feature extraction module is used to acquire the pre-synthesized speech output by the improved speech synthesis model in real time and extract the dynamic prosodic features of the pre-synthesized speech. The real-time adjustment information acquisition module is used to input the semantic features, the emotional features and the dynamic prosodic features as context features into the sub-language label insertion adaptive adjustment component containing a multilayer perceptron, and obtain the label insertion real-time adjustment information output by the sub-language label insertion adaptive adjustment component; The target speech synthesis module is used to dynamically insert the target sub-language into the real-time synthesized speech sequence based on the real-time adjustment information of the tag insertion and the initial insertion information of each sub-language tag, thereby completing the target speech synthesis.

[0007] Thirdly, embodiments of this application also provide a computer device that adopts the technical solution described below: A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the speech synthesis method described above.

[0008] Fourthly, embodiments of this application also provide a computer-readable storage medium, which adopts the technical solutions described below: A computer-readable storage medium storing computer-readable instructions that, when executed by a processor, implement the steps of the speech synthesis method described above.

[0009] Compared with the prior art, the embodiments of this application have the following main advantages: The speech synthesis method described in this application involves: acquiring multimodal material data for speech synthesis; performing feature encoding processing to generate paralinguistic insertion prediction input data; performing prediction to obtain all predicted paralinguistic tags and initial insertion information for each paralinguistic tag; inputting the speech text data to be synthesized and all paralinguistic tags into an improved speech synthesis model; extracting semantic and emotional features from the speech text data to be synthesized; acquiring the preceding synthesized speech output by the improved speech synthesis model in real time and extracting the dynamic prosodic features of the preceding synthesized speech; inputting the semantic features, emotional features, and dynamic prosodic features as contextual features into a paralinguistic tag insertion adaptive adjustment component containing a multilayer perceptron to obtain the tag insertion real-time adjustment information output by the paralinguistic tag insertion adaptive adjustment component; and dynamically inserting the target paralinguistic tag into the real-time synthesized speech sequence based on the tag insertion real-time adjustment information and the initial insertion information for each paralinguistic tag to complete the target speech synthesis. When applied to consultation and response scenarios in fintech or healthcare businesses, the method combines more features beyond text features in both paralinguistic prediction and insertion, resulting in higher quality synthesized speech in terms of naturalness, reliability, and scene adaptability. Attached Figure Description

[0010] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is an exemplary system architecture diagram to which this application can be applied; Figure 2 This is a flowchart of an embodiment of a speech synthesis method according to this application; Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 202 shown; Figure 4 This is a flowchart of a specific embodiment of the pre-training of the paralinguistic insertion prediction model in the speech synthesis method described in this application; Figure 5 This is a flowchart of a specific embodiment of the speech synthesis method described in this application, which involves annotating multimodal data used for training. Figure 6 yes Figure 4 A flowchart of a specific embodiment of step 403 shown; Figure 7 This is a schematic diagram of one embodiment of a speech synthesis device according to this application; Figure 8 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0012] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0013] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0014] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0015] like Figure 1 As shown, system architecture 100 may include terminal device 101, network 102, and server 103. Terminal device 101 may be a laptop 1011, tablet 1012, or mobile phone 1013. Network 102 is used as a medium to provide a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables.

[0016] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0017] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to laptops 1011, tablets 1012, or mobile phones 1013, terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, and a desktop computer, etc.

[0018] Server 103 can be a server that provides various services, such as a backend server that provides support for the pages displayed on terminal device 101.

[0019] It should be noted that the speech synthesis method provided in this application is generally executed by a server, and correspondingly, a speech synthesis device is generally located in the server.

[0020] It should be understood that Figure 1The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0021] Continue to refer to Figure 2 The diagram shows a flowchart of an embodiment of a speech synthesis method according to this application. The speech synthesis method includes the following steps: Step 201: Obtain multimodal material data for speech synthesis.

[0022] The multimodal material data includes the speech-text data to be synthesized. Specifically, the speech-text data to be synthesized refers to text data that has been directly converted into speech by the target TTS model.

[0023] In this embodiment, in order to make the speech synthesis results have a richer and more realistic contextual effect, the multimodal material data also includes at least one of the following: guiding audio data, visual data, and dialogue scene description data.

[0024] Specifically, the acquisition of multimodal material data to be synthesized can be achieved by obtaining the multimodal material data uploaded by the target data provider through a preset multimodal material data receiving interface.

[0025] By acquiring multimodal material data for speech synthesis, it is possible to generate target synthesized speech with more realistic output and better quality based on the multimodal material data.

[0026] Step 202: Perform feature encoding processing on the multimodal material data and generate input data in the target format for paralinguistic insertion prediction.

[0027] The secondary language includes speech that cannot be directly converted into natural language.

[0028] In this embodiment, the paralanguage, such as laughter, surprise sounds, coughs, etc. during voice dialogue, specifically refers to a series of sounds that cannot be directly converted into natural language; natural language refers to the languages ​​used by humans in daily communication, such as English, Chinese, French, etc.; the paralanguage can also be understood as sounds that cannot be directly represented by corresponding language text.

[0029] Specifically, the step of performing feature encoding processing on the multimodal material data and generating input data in a target format for paralinguistic insertion prediction serves to generate input data in a uniform format before the prediction model processes the data, thus facilitating subsequent processing by the paralinguistic insertion prediction model.

[0030] Step 203: Input the input data into a pre-trained paralinguistic insertion prediction model to obtain all paralinguistic labels output by the prediction model and the initial insertion information of each paralinguistic label.

[0031] Among them, the initial insertion information includes an initial insertion position and an initial pronunciation intensity.

[0032] Specifically, the initial insertion position in the initial insertion information is not a fixed position sequence value, but a position representation. For example, if the current paralinguistic is often inserted after "le", the initial insertion position can be represented as "le", indicating that the current paralinguistic is often inserted after the text character "le"; and the initial pronunciation intensity, which represents pitch, length, loudness, etc., can be represented by a numerical value. For example, the pronunciation intensity is 0.6, and this intensity value can be set according to a single specific index or comprehensively set according to multiple indexes, which will not be elaborated here; More specifically, assume that the paralinguistic label is laugh, representing laughter. Due to differences in pitch, length, loudness, etc., there are big laughs, crazy laughs, gentle laughs, smiles, etc. For example, in a happy situation, "I won the lottery", corresponding to a big laugh, the insertion position is "le", and the pronunciation intensity is 0.6; when a service staff makes a polite greeting, "Hello", corresponding to a smile, the insertion position is "hao", and the pronunciation intensity is 0.2; in the situation of turning from extreme sadness to joy, usually crying heartbreakingly and only able to utter "ah" without being able to speak, suddenly, finding that there is a misunderstanding or the event is wrong, corresponding to a crazy laugh, the pronunciation position is "ah", and the pronunciation intensity is 0.9; at this time, the initial insertion information of the paralinguistic label laugh includes {("le": 0.6), ("hao": 0.2), ("ah": 0.9)}. Therefore, the initial insertion information of each paralinguistic label can be understood as a JSON-like set data corresponding to each paralinguistic label, and the initial insertion information is a JSON-like set data pre-learned, trained, and sorted. Other paralinguistic labels are also JSON-like set data, and the specific content of the JSON-like set data may overlap with each other or may not overlap with each other. These differences are related to specific speech pronunciation scenarios and will not be elaborated here.

[0033] Step 204: Input the speech text data to be synthesized and all the paralinguistic labels into an improved speech synthesis model.

[0034] Among them, a paralinguistic label insertion adaptive adjustment component is introduced in the improved speech synthesis model, and the paralinguistic label insertion adaptive adjustment component is used to adjust the insertion information of the paralinguistic label based on real-time synthesis context information.

[0035] Specifically, by inputting the speech-text data to be synthesized and all the sub-language tags into the improved speech synthesis model, after recognizing the speech-text data to be synthesized, the model can obtain the initial insertion information combination of the speech-text data to be synthesized and the sub-language tags according to different text characters, so as to facilitate subsequent sub-language insertion adjustments on the initial insertion information combination of the speech-text data to be synthesized and the sub-language tags.

[0036] In this embodiment, the improved speech synthesis model refers to the introduction of a sub-language tag insertion adaptive adjustment component into a conventional TTS text-to-speech model.

[0037] Step 205: Extract the semantic and emotional features of the speech-text data to be synthesized.

[0038] Specifically, the semantic and emotional features of the speech text data to be synthesized can be extracted directly through the text feature encoding layer of the improved speech synthesis model.

[0039] By extracting the semantic and emotional features of the speech-text data to be synthesized, it is possible to subsequently perform paralinguistic insertion and adjustment by combining the textual semantic and emotional features of the speech-text data itself.

[0040] Step 206: Real-time acquisition of the preceding synthesized speech output by the improved speech synthesis model and extraction of the dynamic prosodic features of the preceding synthesized speech.

[0041] Specifically, when the improved speech synthesis model performs encoding and decoding to synthesize speech, it is a sequential encoding and decoding dynamic process with a continuous state of preceding synthesized speech, data being encoded and decoded, and subsequent unencoded data. Here, by acquiring the preceding synthesized speech output by the improved speech synthesis model in real time and extracting the dynamic prosodic features of the preceding synthesized speech, the dynamic prosodic features of the preceding synthesized speech can be applied to the subsequent paralinguistic insertion and adjustment, thus paying more attention to the real-time speech context effect.

[0042] Step 207: Input the semantic features, the emotional features, and the dynamic prosodic features as contextual features into the sub-language label insertion adaptive adjustment component containing a multilayer perceptron, and obtain the real-time label insertion adjustment information output by the sub-language label insertion adaptive adjustment component.

[0043] Specifically, the semantic features, the emotional features, and the dynamic prosodic features are input as contextual features into the sub-language label insertion adaptive adjustment component containing a multilayer perceptron, thereby realizing sub-language insertion adjustment through feature fusion.

[0044] In this embodiment, the semantic features, the emotional features, and the dynamic prosodic features are input as contextual features into the sub-language label insertion adaptive adjustment component containing a multilayer perceptron, according to a preset formula: To implement this in detail, among which... This represents the contextual features. This represents a multilayer perceptron. For sigmoid activation function, This represents the preset constant coefficient. This indicates the real-time adjustment value of the output.

[0045] Specifically, the multilayer perceptron in the sub-language tag insertion adaptive adjustment component integrates the semantic features, the emotional features, and the dynamic prosodic features, and after final activation, obtains the real-time tag insertion adjustment information output by the sub-language tag insertion adaptive adjustment component, thus enabling sub-language insertion adjustment based on the real-time speech synthesis situation.

[0046] Step 208: Based on the real-time adjustment information of the tag insertion and the initial insertion information of each sub-language tag, the target sub-language is dynamically inserted into the real-time synthesized speech sequence to complete the target speech synthesis.

[0047] Specifically, the step of dynamically inserting the target sub-language into the real-time synthesized speech sequence refers to inserting the final selected sub-language currently used for real-time speech synthesis into the real-time synthesized speech sequence based on the adjusted insertion information.

[0048] In this embodiment, the speech synthesis method is applied to consultation and response scenarios in fintech or healthcare businesses. It can assist intelligent AI customer service in speech synthesis, replacing human responses and making the process more intelligent. Furthermore, during speech synthesis, the insertion of paralinguistics can be dynamically adjusted based on real-time contextual changes. For example, in financial consultation, if a customer service representative is having a conversation with a customer and the customer experiences emotional fluctuations during the conversation, these real-time emotional fluctuations can be used as a basis for contextual changes. When the customer service representative is synthesizing the response speech in real time, this dynamic contextual change basis is combined to update, predict, and adjust the insertion of paralinguistics, thereby synthesizing a more realistic and higher-quality real-time speech.

[0049] In this embodiment, the process involves: acquiring multimodal material data for speech synthesis; performing feature encoding to generate paralinguistic insertion prediction input data; performing prediction to obtain all predicted paralinguistic tags and initial insertion information for each tag; inputting the speech text data to be synthesized and all paralinguistic tags into an improved speech synthesis model; extracting semantic and emotional features from the speech text data; acquiring the preceding synthesized speech output by the improved speech synthesis model in real time and extracting the dynamic prosodic features of the preceding synthesized speech; inputting the semantic, emotional, and dynamic prosodic features as contextual features into a paralinguistic tag insertion adaptive adjustment component containing a multilayer perceptron to obtain the real-time tag insertion adjustment information output by the component; and dynamically inserting the target paralinguistic tag into the real-time synthesized speech sequence based on the real-time tag insertion adjustment information and the initial insertion information for each paralinguistic tag to complete the target speech synthesis. When this method is applied to consultation and response scenarios in fintech or healthcare businesses, the prediction and insertion of paralinguistic tags incorporate more features beyond textual features, resulting in synthesized speech with higher quality in terms of naturalness, reliability, and scene adaptability.

[0050] Continue to refer to Figure 3 , Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 202 shown includes: Step 301: Perform log-Mel feature encoding on the audio material data to obtain the first prediction element; Specifically, the log-Mel feature encoding can be used for feature representation in audio processing tasks such as speech recognition and voiceprint recognition. It extracts effective information from audio by simulating the auditory perception characteristics of the human ear, while suppressing irrelevant noise and highlighting key speech features. First, the audio material data is preprocessed, including resampling and framing, to meet the short-time stationarity assumption. Then, a Fast Fourier Transform (FFT) is applied to each frame to transform it to the frequency domain. Next, the spectrum is weighted using a Mel filter bank to simulate the nonlinear perception of frequency by the human ear. Then, the energy of each filter is logarithmically taken to compress the dynamic range. Finally, normalization is performed to obtain the log-Mel features. The acquisition of log-Mel features, based on the Mel spectrogram, further applies a logarithmic transformation, which helps to enhance the discriminative ability of speech features and focuses more on relative energy changes rather than absolute values.

[0051] Step 302: Perform word segmentation and semantic encoding on the text-based material data to obtain the second prediction element; Specifically, a pre-trained BERT model can be used to perform word segmentation and semantic encoding on text-based material data.

[0052] Step 303: Perform one-hot encoding on the composite scene material data to obtain the third prediction element; Specifically, since the synthesis scenes are divided into actual speech synthesis scenarios, the number of scene categories is generally limited and the number of types is not large. Therefore, one-hot encoding is directly performed, which reduces the difficulty of encoding and processing the source data for the synthesis scene categories.

[0053] Step 304: Extract facial expression feature vectors from the visual material data to obtain the fourth prediction element; Specifically, a convolutional neural network can be used to extract facial expression feature vectors from visual material data to obtain the fourth prediction element.

[0054] Step 305: The first prediction element, the second prediction element, the third prediction element, and the fourth prediction element are sorted into sets according to a preset sorting format to obtain the target set as the input data.

[0055] Specifically, the input data is formatted as a set of elements, and the elements within the set are ordered as follows: first predicted element, second predicted element, third predicted element, and fourth predicted element. For example: The elements in the set X are, in order, the first prediction element, the second prediction element, the third prediction element, and the fourth prediction element, which represent the feature encoding vectors obtained from audio material data, text material data, synthetic scene material data, and visual material data, respectively.

[0056] Continue to refer to Figure 4 In some embodiments, prior to step 203, the speech synthesis method further includes a step of pre-training a paralinguistic insertion prediction model. Figure 4 This is a flowchart of a specific embodiment of the pre-training of the paralinguistic insertion prediction model in the speech synthesis method described in this application, including: Step 401: Obtain the multimodal training dataset labeled with secondary language tags; Step 402: Input the multimodal training dataset into the sub-language insertion prediction model to be trained; Step 403: Based on the annotation information of the sub-language labels in the multimodal training dataset, learn the insertion information of the sub-languages ​​with different labels in different speech scenarios, wherein the insertion information includes the insertion position and pronunciation intensity of the sub-languages; Specifically, the step of learning the insertion information of different sub-languages ​​in different speech scenarios based on the annotation information of the sub-language labels in the multimodal training dataset can be achieved by using a comprehensive analysis method. This involves analyzing the annotation information of the sub-language labels in the multimodal training dataset, and then organizing the results based on a large amount of analysis to learn the insertion information of different sub-languages ​​in different speech scenarios.

[0057] Step 404: The learned paralinguistic insertion information of different labels in different speech scenarios is added as prior knowledge to the prediction model to obtain a pre-trained paralinguistic insertion prediction model.

[0058] In this embodiment, a pre-trained paralinguistic insertion prediction model is obtained by acquiring a multimodal training dataset labeled with paralinguistic tags. This allows the pre-trained model to output all paralinguistic tags used in the text data to be synthesized, as well as the initial insertion information for each tag, before actual speech synthesis. This facilitates the subsequent insertion of paralinguistic tags into the text-speech sequence, improving the quality of the synthesized speech and making it more closely resemble real speech.

[0059] Continue to refer to Figure 5 In some embodiments, prior to step 401, the speech synthesis method further includes a step of labeling the multimodal data used for training. Figure 5 This is a flowchart of a specific embodiment of the speech synthesis method described in this application, which involves annotating multimodal data used for training, including: Step 501: Obtain unlabeled multimodal data from multiple data sources; Step 502: Divide the unlabeled multimodal data into manually labeled datasets and automatically labeled datasets according to a preset ratio; Specifically, when there is a large amount of unlabeled multimodal data, the partitioning ratio can be dynamically adjusted so that the amount of labeled data in the manually labeled dataset is much smaller than the amount of labeled data in the automatically labeled dataset.

[0060] Step 503: Obtain the sub-language label annotation results of the manually annotated dataset; Specifically, when annotating manually labeled datasets, NVSpeech can be used for auxiliary annotation. NVSpeech is a multilingual dataset for text-to-speech (TTS) containing approximately 170,000 audio-text pairs and currently has 18 sub-language labels. In this embodiment, in order to enrich the insertion prediction of sub-languages, an outer layer of labels is added based on the 18 sub-language labels, namely scene-sub-language labels. This label represents the speech scene used by each sub-language.

[0061] Step 504: The manually annotated dataset and the sub-language label annotation results of the manually annotated dataset are used as reference knowledge and input into the preset sub-language perception ASR model; Specifically, the preset sub-language-aware ASR model is, for example, a sub-language-aware ASR model with wav2vec 2.0 as the backbone network.

[0062] Step 505: Input the automated annotation dataset into the sub-language awareness ASR model, and combine the reference knowledge in the sub-language awareness ASR model to automatically annotate each data in the automated annotation dataset with sub-language labels.

[0063] Specifically, after inputting the automated annotation dataset into the sub-language perception ASR model, the sub-language perception ASR is used to extract multi-layer audio features from the automated annotation dataset; a learnable weighted summation method is used to fuse the extracted multi-layer audio features to obtain audio fusion features; a preset Transformer encoder is used to encode the audio fusion features to obtain an encoded output result; the encoded output result is decoded using a manually annotated dataset to obtain a decoded output result; for the undecoded encoded results, a difference matrix is ​​obtained and activated by a preset activation function to obtain the sub-language perception result; based on the sub-language perception, all sub-languages ​​are identified and automatically annotated.

[0064] In this embodiment, by using a combination of manual and automated annotation methods to annotate the multimodal training data, the learning and training efficiency of the para-language insertion prediction model is improved to some extent.

[0065] Continue to refer to Figure 6 , Figure 6 yes Figure 4 A flowchart of a specific embodiment of step 403 shown includes: Step 601: By parsing the annotation information of the sub-language tags, the speech text sequence in which different sub-languages ​​are located, the speech scene information corresponding to different sub-languages, and the audio signals corresponding to different sub-languages ​​are identified. Step 602: Perform feature encoding processing on the speech text sequence, speech scene information and audio signal corresponding to the same language to obtain the text semantic features, scene features and audio features corresponding to the same language; Step 603: The text semantic features, scene features and audio features corresponding to the same sub-language are introduced into a preset attention fusion layer to perform multimodal feature weighted fusion to obtain the multimodal fusion features corresponding to each sub-language. Step 604: Based on the multimodal fusion features corresponding to each sub-language, extract the insertion information of the sub-language with different labels in different speech scenarios.

[0066] Specifically, by decoding and activating the multimodal fusion features corresponding to each sub-language, the insertion information of sub-languages ​​with different labels in different speech scenarios is extracted.

[0067] In this embodiment, by introducing an attention fusion layer during the learning and training of the sub-language insertion prediction model, multi-modal feature weighted fusion is performed to obtain multi-modal fusion features corresponding to each sub-language. This allows the subsequent actual speech synthesis to combine other modal features besides the synthesized text features for sub-language insertion prediction, ensuring that the inserted sub-language is closer to the real context.

[0068] In this embodiment, the step of learning the insertion information of different sub-language tags in different speech scenarios based on the annotation information of the sub-language tags in the multimodal training dataset further includes: statistically analyzing the annotation information of all sub-language tags to obtain the total number of annotations corresponding to different sub-language tags; filtering out target sub-language tags with high annotation frequency based on the total number of annotations corresponding to different sub-language tags, a preset ratio algorithm, and a preset ratio threshold; and training the target sub-language tags separately using a transfer learning strategy to obtain the learning and training results for the target sub-language tags. The step of training the target sub-language tags separately using a transfer learning strategy includes, when the multimodal training dataset is shared, transferring the feature extraction parameters of the target sub-language tags to a new feature recognition and extraction component.

[0069] Specifically, through comprehensive analysis, high-frequency target sub-language tags are selected. For these high-frequency target sub-language tags, a transfer learning strategy is used to train them separately. This avoids training high-frequency target sub-language tags together with other low-frequency sub-language tags, which would consume too many training resources. By adopting a separate training approach, the final training time of the sub-language insertion prediction model can be reduced, allowing it to be put into practical speech synthesis applications earlier.

[0070] In this embodiment, after performing the step of obtaining all sub-language tags output by the prediction model and the initial insertion information of each sub-language tag, the method further includes: marking the output basis for all sub-language tags output by the prediction model according to a preset interpretability tracing component, wherein the output basis includes the text data, audio data, visual data and scene data on which the sub-language tags are based when they are output, as well as the output contribution of the different types of data on which they are based.

[0071] Specifically, in conjunction with the preset interpretability tracing component, for all sub-language tags output by the prediction model, their contribution to the output based on their source and different modal data is marked, thereby improving the interpretability of sub-language insertion during speech synthesis.

[0072] In this embodiment, after performing the step of dynamically inserting the target sub-language into the real-time synthesized speech sequence based on the real-time adjustment information of the tag insertion and the initial insertion information of each sub-language tag to complete the target speech synthesis, the method further includes: collecting user quality feedback on the target synthesized speech according to a preset user feedback mechanism; dynamically optimizing the fusion parameters of the preset attention fusion layer and the adjustment parameters of the sub-language tag insertion adaptive adjustment component based on the quality feedback; and continuously performing target speech synthesis using the optimized attention fusion layer and the sub-language tag insertion adaptive adjustment component.

[0073] Specifically, by combining user quality feedback, the fusion parameters of the preset attention fusion layer and the adjustment parameters of the sub-language tag insertion adaptive adjustment component are dynamically optimized, so that the speech synthesis method can gradually synthesize better and higher quality speech, thereby improving the user experience.

[0074] In this embodiment, the process involves: acquiring multimodal material data for speech synthesis; performing feature encoding to generate paralinguistic insertion prediction input data; performing prediction to obtain all predicted paralinguistic tags and initial insertion information for each tag; inputting the speech text data to be synthesized and all paralinguistic tags into an improved speech synthesis model; extracting semantic and emotional features from the speech text data; acquiring the preceding synthesized speech output by the improved speech synthesis model in real time and extracting the dynamic prosodic features of the preceding synthesized speech; inputting the semantic, emotional, and dynamic prosodic features as contextual features into a paralinguistic tag insertion adaptive adjustment component containing a multilayer perceptron to obtain the real-time tag insertion adjustment information output by the component; and dynamically inserting the target paralinguistic tag into the real-time synthesized speech sequence based on the real-time tag insertion adjustment information and the initial insertion information for each paralinguistic tag to complete the target speech synthesis. When this method is applied to consultation and response scenarios in fintech or healthcare businesses, the prediction and insertion of paralinguistic tags incorporate more features beyond textual features, resulting in synthesized speech with higher quality in terms of naturalness, reliability, and scene adaptability.

[0075] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0076] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0077] In this embodiment, the process involves: acquiring multimodal material data for speech synthesis; performing feature encoding to generate paralinguistic insertion prediction input data; performing prediction to obtain all predicted paralinguistic tags and initial insertion information for each tag; inputting the speech text data to be synthesized and all paralinguistic tags into an improved speech synthesis model; extracting semantic and emotional features from the speech text data; acquiring the preceding synthesized speech output by the improved speech synthesis model in real time and extracting the dynamic prosodic features of the preceding synthesized speech; inputting the semantic, emotional, and dynamic prosodic features as contextual features into a paralinguistic tag insertion adaptive adjustment component containing a multilayer perceptron to obtain the real-time tag insertion adjustment information output by the component; and dynamically inserting the target paralinguistic tag into the real-time synthesized speech sequence based on the real-time tag insertion adjustment information and the initial insertion information for each paralinguistic tag to complete the target speech synthesis. When this method is applied to consultation and response scenarios in fintech or healthcare businesses, the prediction and insertion of paralinguistic tags incorporate more features beyond textual features, resulting in synthesized speech with higher quality in terms of naturalness, reliability, and scene adaptability.

[0078] Further reference Figure 7 As a response to the above Figure 2 To implement the method shown, this application provides an embodiment of a speech synthesis device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0079] like Figure 7As shown, the speech synthesis device 700 described in this embodiment includes: a synthesis material data acquisition module 701, a prediction input data generation module 702, an initial insertion information acquisition module 703, a speech synthesis input module 704, a text data feature extraction module 705, a dynamic prosodic feature extraction module 706, a real-time adjustment information acquisition module 707, and a target speech synthesis module 708. Wherein: The synthesis material data acquisition module 701 is used to acquire multimodal material data to be synthesized, wherein the multimodal material data includes speech-text data to be synthesized; The prediction input data generation module 702 is used to perform feature encoding processing on the multimodal material data and generate input data in the target format for sub-language insertion prediction. The initial insertion information acquisition module 703 is used to input the input data into a pre-trained sub-language insertion prediction model and acquire all sub-language labels output by the prediction model and the initial insertion information of each sub-language label. The speech synthesis input module 704 is used to input the speech text data to be synthesized and all the sub-language tags into the improved speech synthesis model. The improved speech synthesis model introduces a sub-language tag insertion adaptive adjustment component, which is used to adjust the insertion information of sub-language tags based on real-time synthesis context information. The text data feature extraction module 705 is used to extract the semantic features and sentiment features of the speech-text data to be synthesized. The dynamic prosodic feature extraction module 706 is used to acquire the preceding synthesized speech output by the improved speech synthesis model in real time and extract the dynamic prosodic features of the preceding synthesized speech. The real-time adjustment information acquisition module 707 is used to input the semantic features, the emotional features and the dynamic prosodic features as context features into the sub-language label insertion adaptive adjustment component containing a multilayer perceptron, and obtain the label insertion real-time adjustment information output by the sub-language label insertion adaptive adjustment component; The target speech synthesis module 708 is used to dynamically insert the target sub-language into the real-time synthesized speech sequence based on the real-time adjustment information of the tag insertion and the initial insertion information of each sub-language tag, thereby completing the target speech synthesis.

[0080] This application acquires multimodal source data for speech synthesis; performs feature encoding processing to generate paralinguistic insertion prediction input data; performs prediction to obtain all predicted paralinguistic tags and initial insertion information for each paralinguistic tag; inputs the speech text data to be synthesized and all paralinguistic tags into an improved speech synthesis model; extracts semantic and emotional features from the speech text data to be synthesized; acquires the preceding synthesized speech output by the improved speech synthesis model in real time and extracts the dynamic prosodic features of the preceding synthesized speech; inputs the semantic, emotional, and dynamic prosodic features as contextual features into a paralinguistic tag insertion adaptive adjustment component containing a multilayer perceptron to obtain the real-time tag insertion adjustment information output by the paralinguistic tag insertion adaptive adjustment component; based on the real-time tag insertion adjustment information and the initial insertion information of each paralinguistic tag, the target paralinguistic tag is dynamically inserted into the real-time synthesized speech sequence to complete the target speech synthesis. Applying this method to consultation and response scenarios in fintech or healthcare businesses, because the prediction and insertion of paralinguistic tags incorporate more features beyond text features, the synthesized speech exhibits higher quality in terms of naturalness, reliability, and scene adaptability.

[0081] In this embodiment, the speech synthesis device 700 further includes a training dataset acquisition module, a training input learning module, a paralinguistic insertion knowledge learning module, and a paralinguistic insertion prediction model acquisition module. Wherein: The training dataset acquisition module is used to acquire a multimodal training dataset that has been labeled with secondary language tags; The learning and training input module is used to input the multimodal training dataset into the sub-language insertion prediction model to be learned and trained; The sublanguage insertion knowledge learning module is used to learn the insertion information of sublanguages ​​with different labels in different speech scenarios based on the annotation information of sublanguage labels in the multimodal training dataset. The insertion information includes the insertion position and pronunciation intensity of the sublanguage. The sublanguage insertion prediction model acquisition module is used to add the learned sublanguage insertion information of different labels in different speech scenarios as prior knowledge to the prediction model to obtain a pre-trained sublanguage insertion prediction model.

[0082] In this embodiment, the paralinguistic insertion knowledge learning module includes a labeled information parsing unit, a multimodal coding unit, a multimodal feature fusion unit, and a paralinguistic insertion information extraction unit. Wherein: The annotation information parsing unit is used to identify the speech text sequence in which different sub-languages ​​are located, the speech scene information corresponding to different sub-languages, and the audio signals corresponding to different sub-languages ​​by parsing the annotation information of the sub-language tags. The multimodal coding unit is used to perform feature encoding processing on the speech text sequence, speech scene information and audio signal corresponding to the same language, respectively, to obtain the text semantic features, scene features and audio features corresponding to the same language; The multimodal feature fusion unit is used to introduce the text semantic features, scene features and audio features corresponding to the same sub-language into the preset attention fusion layer, and perform multimodal feature weighted fusion to obtain the multimodal fusion features corresponding to each sub-language. The sub-language insertion information extraction unit is used to extract the insertion information of sub-languages ​​with different labels in different speech scenarios based on the multimodal fusion features corresponding to each sub-language.

[0083] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0084] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0085] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 8 , Figure 8 This is a basic structural block diagram of the computer device in this embodiment.

[0086] The computer device 8 includes a memory 8a, a processor 8b, and a network interface 8c, which are interconnected via a system bus. It should be noted that... Figure 8Only a computer device 8 with component memory 8a, processor 8b, and network interface 8c is shown. However, it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0087] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0088] The memory 8a includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 8a may be an internal storage unit of the computer device 8, such as the hard disk or memory of the computer device 8. In other embodiments, the memory 8a may also be an external storage device of the computer device 8, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Of course, the memory 8a may include both internal storage units and external storage devices of the computer device 8. In this embodiment, the memory 8a is typically used to store the operating system and various application software installed on the computer device 8, such as computer-readable instructions for a speech synthesis method. In addition, the memory 8a can also be used to temporarily store various types of data that have been output or will be output.

[0089] In some embodiments, the processor 8b may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 8b is typically used to control the overall operation of the computer device 8. In this embodiment, the processor 8b is used to execute computer-readable instructions stored in the memory 8a or to process data, such as executing computer-readable instructions for the speech synthesis method described above.

[0090] The network interface 8c may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 8 and other electronic devices.

[0091] The computer device proposed in this embodiment belongs to the field of artificial intelligence technology and is applied to the scenario of speech synthesis for AI question-answering agents. This application acquires multimodal material data to be synthesized; performs feature encoding processing to generate paralinguistic insertion prediction input data; performs prediction to obtain all predicted paralinguistic tags and the initial insertion information of each paralinguistic tag; inputs the speech text data to be synthesized and all paralinguistic tags into an improved speech synthesis model; extracts semantic and emotional features from the speech text data to be synthesized; acquires the preceding synthesized speech output by the improved speech synthesis model in real time and extracts the dynamic prosodic features of the preceding synthesized speech; inputs the semantic features, emotional features, and dynamic prosodic features as contextual features into a paralinguistic tag insertion adaptive adjustment component containing a multilayer perceptron to obtain the real-time tag insertion adjustment information output by the paralinguistic tag insertion adaptive adjustment component; based on the real-time tag insertion adjustment information and the initial insertion information of each paralinguistic tag, dynamically inserts the target paralinguistic tag into the real-time synthesized speech sequence to complete the target speech synthesis. When the method is applied to consultation and response scenarios in fintech or healthcare businesses, the synthesized speech achieves higher quality in terms of naturalness, reliability, and scene adaptability because the prediction and insertion of paralinguistics incorporate more features beyond textual features.

[0092] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by a processor to cause the processor to perform the steps of the speech synthesis method described above.

[0093] The computer-readable storage medium proposed in this embodiment belongs to the field of artificial intelligence technology and is applied to the speech synthesis scenario of AI question-answering agents. This application acquires multimodal material data to be synthesized; performs feature encoding processing to generate paralinguistic insertion prediction input data; performs prediction to obtain all predicted paralinguistic tags and the initial insertion information of each paralinguistic tag; inputs the speech text data to be synthesized and all paralinguistic tags into an improved speech synthesis model; extracts semantic and emotional features from the speech text data to be synthesized; acquires the preceding synthesized speech output by the improved speech synthesis model in real time and extracts the dynamic prosodic features of the preceding synthesized speech; inputs the semantic features, emotional features, and dynamic prosodic features as contextual features into a paralinguistic tag insertion adaptive adjustment component containing a multilayer perceptron to obtain the real-time tag insertion adjustment information output by the paralinguistic tag insertion adaptive adjustment component; based on the real-time tag insertion adjustment information and the initial insertion information of each paralinguistic tag, dynamically inserts the target paralinguistic tag into the real-time synthesized speech sequence to complete the target speech synthesis. When the method is applied to consultation and response scenarios in fintech or healthcare businesses, the synthesized speech achieves higher quality in terms of naturalness, reliability, and scene adaptability because the prediction and insertion of paralinguistics incorporate more features beyond textual features.

[0094] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0095] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to make the disclosure of this application more thorough and comprehensive. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application. Software tools or components not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

Claims

1. A speech synthesis method, characterized in that, Includes the following steps: Acquire multimodal material data for speech synthesis, wherein the multimodal material data includes speech-text data to be synthesized; The multimodal material data is subjected to feature encoding processing, and input data in the target format for sub-language insertion prediction is generated; The input data is fed into a pre-trained sub-language insertion prediction model to obtain all sub-language labels output by the prediction model and the initial insertion information for each sub-language label. The speech text data to be synthesized and all the sub-language tags are input into the improved speech synthesis model, wherein the improved speech synthesis model introduces a sub-language tag insertion adaptive adjustment component, which is used to adjust the insertion information of sub-language tags based on real-time synthesis context information; Extract the semantic and sentiment features of the speech-text data to be synthesized; The improved speech synthesis model outputs pre-synthesized speech in real time, and the dynamic prosodic features of the pre-synthesized speech are extracted. The semantic features, the emotional features, and the dynamic prosodic features are input as contextual features into the sub-language label insertion adaptive adjustment component containing a multilayer perceptron to obtain the real-time label insertion adjustment information output by the sub-language label insertion adaptive adjustment component; Based on the real-time adjustment information of the tag insertion and the initial insertion information of each sub-language tag, the target sub-language is dynamically inserted into the real-time synthesized speech sequence to complete the target speech synthesis.

2. The speech synthesis method according to claim 1, characterized in that, The step of performing feature encoding processing on the multimodal material data and generating input data in the target format for para-language insertion prediction includes: The audio material data is processed by log-Mel feature encoding to obtain the first prediction element; The text-based material data is first segmented into words and then semantically encoded to obtain the second prediction element; One-hot encoding is performed on the composite scene material data to obtain the third prediction element; Extract facial expression feature vectors from visual material data to obtain the fourth prediction element; The first, second, third, and fourth prediction elements are aggregated and organized according to a preset format to obtain the target set as the input data.

3. The speech synthesis method according to claim 1, characterized in that, Before performing the step of inputting the input data into a pre-trained sub-language insertion prediction model and obtaining all sub-language labels output by the prediction model and the initial insertion information for each sub-language label, the method further includes: Obtain a multimodal training dataset that has been labeled with secondary language tags; The multimodal training dataset is input into the sub-language insertion prediction model to be trained. Based on the annotation information of the sub-language labels in the multimodal training dataset, the insertion information of the sub-language with different labels in different speech scenarios is learned, wherein the insertion information includes the insertion position and pronunciation intensity of the sub-language; The learned paralinguistic insertion information of different labels in different speech scenarios is added as prior knowledge to the prediction model to obtain a pre-trained paralinguistic insertion prediction model.

4. The speech synthesis method according to claim 3, characterized in that, Before performing the step of obtaining the multimodal training dataset labeled with secondary language tags, the method further includes: Obtain unlabeled multimodal data from multiple data sources; The unlabeled multimodal data is divided into manually labeled datasets and automatically labeled datasets according to a preset ratio; Obtain the sub-language label annotation results of the manually annotated dataset; The manually annotated dataset and the sub-language label annotation results of the manually annotated dataset are used as reference knowledge and input into the preset sub-language perception ASR model; The automated annotation dataset is input into the sub-language-aware ASR model, and the reference knowledge in the sub-language-aware ASR model is used to automatically annotate each data point in the automated annotation dataset with sub-language labels.

5. The speech synthesis method according to claim 3, characterized in that, The step of learning the insertion information of different labeled sub-languages ​​in different speech scenarios based on the annotation information of the sub-language labels in the multimodal training dataset includes: By analyzing the annotation information of the sub-language tags, we can identify the speech text sequence in which different sub-languages ​​are located, the speech scene information corresponding to different sub-languages, and the audio signals corresponding to different sub-languages. Feature encoding is performed on the speech text sequence, speech scene information and audio signal corresponding to the same language to obtain the text semantic features, scene features and audio features corresponding to the same language; The text semantic features, scene features, and audio features corresponding to the same sub-language are introduced into a pre-defined attention fusion layer to perform multimodal feature weighted fusion, thereby obtaining the multimodal fusion features corresponding to each sub-language. Based on the multimodal fusion features corresponding to each sub-language, the insertion information of sub-languages ​​with different labels in different speech scenarios is extracted.

6. The speech synthesis method according to claim 3, characterized in that, The step of learning the insertion information of different labeled sub-languages ​​in different speech scenarios based on the annotation information of the sub-language labels in the multimodal training dataset includes: Statistical analysis was performed on the annotation information of all sub-language tags to obtain the total number of annotations corresponding to each sub-language tag; Based on the total number of annotations corresponding to different sub-language tags, the preset ratio algorithm, and the preset ratio threshold, target sub-language tags with high annotation frequency are selected. For the target sub-language label, a transfer learning strategy is used to train it separately to obtain the training result for the target sub-language label. The separate training using the transfer learning strategy includes transferring the feature extraction parameters of the target sub-language label to a new feature recognition extraction component when the multimodal training dataset is shared.

7. The speech synthesis method according to claim 5, characterized in that, After executing the step of dynamically inserting the target sub-language into the real-time synthesized speech sequence based on the real-time adjustment information inserted by the tags and the initial insertion information of each sub-language tag, and completing the target speech synthesis, the method further includes: Based on a pre-set user feedback mechanism, collect user feedback on the quality of the target synthesized speech; Based on the quality feedback, the fusion parameters of the preset attention fusion layer and the adjustment parameters of the sub-language tag insertion adaptive adjustment component are dynamically optimized. The target speech synthesis is continuously performed by using an optimized attention fusion layer and a sub-language label insertion adaptive adjustment component.

8. A speech synthesis device, characterized in that, include: The synthesis material data acquisition module is used to acquire multimodal material data to be synthesized, wherein the multimodal material data includes the speech-text data to be synthesized; The prediction input data generation module is used to perform feature encoding processing on the multimodal material data and generate input data in the target format for sub-language insertion prediction. The initial insertion information acquisition module is used to input the input data into a pre-trained sub-language insertion prediction model and acquire all sub-language labels output by the prediction model and the initial insertion information of each sub-language label. The speech synthesis input module is used to input the speech text data to be synthesized and all the sub-language tags into the improved speech synthesis model. The improved speech synthesis model introduces a sub-language tag insertion adaptive adjustment component, which is used to adjust the insertion information of sub-language tags based on real-time synthesis context information. The text data feature extraction module is used to extract the semantic and emotional features of the speech-text data to be synthesized; The dynamic prosodic feature extraction module is used to acquire the pre-synthesized speech output by the improved speech synthesis model in real time and extract the dynamic prosodic features of the pre-synthesized speech. The real-time adjustment information acquisition module is used to input the semantic features, the emotional features and the dynamic prosodic features as context features into the sub-language label insertion adaptive adjustment component containing a multilayer perceptron, and obtain the label insertion real-time adjustment information output by the sub-language label insertion adaptive adjustment component; The target speech synthesis module is used to dynamically insert the target sub-language into the real-time synthesized speech sequence based on the real-time adjustment information of the tag insertion and the initial insertion information of each sub-language tag, thereby completing the target speech synthesis.

9. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the speech synthesis method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the speech synthesis method as described in any one of claims 1 to 7.

Images