Voice synthesis method, device and equipment based on AI large model in multi-language scene

Abstract

The invention provides a speech synthesis method, device and equipment based on an AI large model in a multi-language scene, and the method comprises the steps: carrying out the text feature extraction of an input multi-language to-be-synthesized text, and obtaining a to-be-synthesized text feature; performing speech synthesis parameter generation on the to-be-synthesized text features based on the fusion feature model; wherein the pre-trained fusion feature model has a mapping relationship between speech synthesis parameters and a text, and the speech synthesis parameters comprise a fundamental frequency parameter, a duration parameter and a frequency spectrum parameter; inputting the speech synthesis parameters into a vocoder which is trained in advance, processing a plurality of parameters in the speech synthesis parameters into intermediate feature vectors based on the vocoder, converting the intermediate feature vectors into discrete speech sample points, converting the discrete speech sample points into continuous speech waveforms, and outputting the continuous speech waveforms. And obtaining synthetic speech corresponding to the multi-language to-be-synthesized text. According to the invention, the coherence and naturalness of the generated voice waveform are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of speech synthesis, and in particular to a speech synthesis method, device and equipment based on an AI large model in a multilingual scenario. Background Art

[0002] In the field of multilingual communication, such as international business meetings, cross-border distance education, multilingual customer service systems, etc. In these scenarios, speech synthesis technology plays the role of converting text information into natural and fluent speech, providing convenience for users with different language backgrounds.

[0003] Traditional speech synthesis technologies are mainly designed and optimized for a single language. However, these technologies face many challenges in a multilingual environment. First, different languages ​​have their own unique grammatical structures, vocabulary, pronunciation rules, and intonation. For example, Chinese is a tonal language, and its fundamental frequency changes are crucial for semantic expression, while some languages ​​distinguish semantics by stress and duration (such as English). At the same time, the spectral characteristics of different languages ​​are also very different, depending on the differences in pronunciation and voice timbre. The huge differences between the above languages ​​make it difficult for a single language model to accurately synthesize multilingual text.

[0004] In addition, in the process of converting speech synthesis parameters into speech waveforms, traditional vocoders also have the problem of being unable to effectively integrate the characteristics of different language parameters when processing multi-language synthesis parameters, resulting in defects in the coherence and naturalness of the generated speech waveforms. Summary of the invention

[0005] The main purpose of the present invention is to provide a speech synthesis method, device and equipment based on an AI large model in a multilingual scenario, aiming to overcome the defect that current speech synthesis methods are difficult to accurately synthesize multilingual texts.

[0006] To achieve the above object, the present invention provides a speech synthesis method based on an AI large model in a multilingual scenario, comprising the following steps: Extracting text features from the input multilingual text to be synthesized to obtain features of the text to be synthesized; Based on the fusion feature model, speech synthesis parameters are generated for the features of the text to be synthesized to obtain speech synthesis parameters; wherein the pre-trained fusion feature model has a mapping relationship between speech synthesis parameters and text, and the speech synthesis parameters include fundamental frequency parameters, duration parameters, and spectrum parameters; The speech synthesis parameters are input into a pre-trained vocoder, multiple parameters of the speech synthesis parameters are processed into intermediate feature vectors based on the vocoder, the intermediate feature vectors are converted into discrete speech sample points, and the discrete speech sample points are converted into continuous speech waveforms to obtain synthesized speech corresponding to the multi-language text to be synthesized.

[0007] Further, based on the vocoder, processing multiple parameters of the speech synthesis parameters into intermediate feature vectors includes: Based on the first layer of neurons of the vocoder, the fundamental frequency parameter is weighted to obtain a weighted fundamental frequency parameter; the duration parameter is normalized to make the duration parameter fall within a standard range; Performing feature enhancement processing on the spectrum parameters based on the second layer neurons of the vocoder to obtain enhanced spectrum parameters; The normalized duration parameter, weighted fundamental frequency parameter and enhanced spectrum parameter are transmitted to the third layer of neurons of the vocoder to perform feature fusion processing to obtain the intermediate feature vector.

[0008] Further, converting the intermediate feature vector into discrete speech sample points, and converting the discrete speech sample points into a continuous speech waveform, comprises: Based on the fourth layer neurons of the vocoder, the intermediate feature vector is subjected to dimensionality reduction processing by a principal component analysis algorithm to obtain a reduced-dimensional feature vector; Converting the dimension-reduced feature vector into discrete speech sample points based on the fifth layer of neurons of the vocoder; According to the time series relationship of the discrete speech sample points, the discrete speech sample points are converted into a continuous speech waveform through an interpolation algorithm.

[0009] Furthermore, the step of extracting text features from the input multilingual text to be synthesized to obtain features of the text to be synthesized includes: Performing lexical analysis, syntactic analysis and semantic analysis on the multilingual text to be synthesized to obtain lexical features, grammatical structure features and semantic information features in the text; The vocabulary features, grammatical structure features and semantic information features are respectively converted into low-dimensional vector representations; and each low-dimensional vector representation is standardized; The standardized low-dimensional vector representations are fused to obtain the text features to be synthesized.

[0010] Furthermore, the generating of speech synthesis parameters for the features of the text to be synthesized based on the fusion feature model to obtain the speech synthesis parameters includes: Inputting the text features to be synthesized into the input layer of the fusion feature model, and the input layer preprocesses the data format thereof; Performing deep feature mapping on the preprocessed text features by fusing multiple hidden layers in the feature model to obtain the speech synthesis parameters; In the process of deep feature mapping, the intonation information features, language type features and semantic sentiment features in the text are calculated, and the fundamental frequency parameter estimation value is generated through neuron connection and weight adjustment; According to the number of words in the text, the complexity of the grammatical structure, and the speaking speed characteristics of different languages, the long short-term memory unit in the hidden layer is used to calculate the duration parameters of each syllable or word; The corresponding spectrum parameters are calculated by the spectrum generation subunit by combining the timbre characteristics of the text, the language pronunciation rule characteristics and the contextual semantic association characteristics.

[0011] Furthermore, after obtaining the synthesized speech corresponding to the multi-language text to be synthesized, the method includes: Acquire multiple language types in the multilingual text to be synthesized, and acquire multi-dimensional text features of the multilingual text to be synthesized; the multi-dimensional text features include text length, text data volume, and the number of characters in the text; Generate a key character array based on the multiple language types and multi-dimensional text features; Acquiring speech features of the synthesized speech; generating a communication key based on the key character array and the speech characteristics of the synthesized speech; The synthesized speech is encrypted based on the communication key and then stored.

[0012] Furthermore, the speech features include speech duration, speech data volume, and tone simulation curve; Generating a communication key based on the key character array and the voice feature of the synthesized voice, comprising: The speech duration and the amount of speech data are standardized and combined into a combination of digital characters; Discretely distribute the characters in the digital character combination into the key character array to obtain a modified key character array; Adding characters in the modified key character array to the tone simulation curve one by one in sequence, and generating a rectangular frame in the tone simulation curve according to the speech duration; On the tone simulation curve, all characters in the rectangular frame are obtained and combined to obtain the communication key.

[0013] The present invention also provides a speech synthesis device based on an AI large model in a multi-language scenario, comprising: An extraction unit, used for extracting text features from the input multi-language text to be synthesized, to obtain features of the text to be synthesized; A generating unit, configured to generate speech synthesis parameters for the features of the text to be synthesized based on a fusion feature model to obtain speech synthesis parameters; wherein the pre-trained fusion feature model has a mapping relationship between speech synthesis parameters and text, and the speech synthesis parameters include fundamental frequency parameters, duration parameters, and spectrum parameters; The synthesis unit is used to input the speech synthesis parameters into a pre-trained vocoder, process multiple parameters of the speech synthesis parameters into intermediate feature vectors based on the vocoder, convert the intermediate feature vectors into discrete speech sample points, and convert the discrete speech sample points into continuous speech waveforms to obtain synthesized speech corresponding to the multi-language text to be synthesized.

[0014] The present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0015] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods are implemented.

[0016] The speech synthesis method, device and equipment based on AI large model in multilingual scenarios provided by the present invention include: extracting text features of input multilingual text to be synthesized to obtain features of the text to be synthesized; generating speech synthesis parameters for the features of the text to be synthesized based on a fusion feature model to obtain speech synthesis parameters; wherein the pre-trained fusion feature model has a mapping relationship between speech synthesis parameters and text, and the speech synthesis parameters include fundamental frequency parameters, duration parameters, and spectrum parameters; inputting the speech synthesis parameters into a pre-trained vocoder, processing multiple parameters in the speech synthesis parameters into intermediate feature vectors based on the vocoder, converting the intermediate feature vectors into discrete speech sample points, and converting the discrete speech sample points into continuous speech waveforms to obtain synthesized speech corresponding to the multilingual text to be synthesized. In the present invention, by generating fundamental frequency parameters, duration parameters, and spectrum parameters for the language text to be synthesized, the characteristics of various speech can be highlighted; at the same time, multiple parameters in the speech synthesis parameters are processed into intermediate feature vectors, the intermediate feature vectors are converted into discrete speech sample points, and the discrete speech sample points are converted into continuous speech waveforms to obtain synthesized speech corresponding to the multi-language text to be synthesized, which can effectively integrate the characteristics of different language parameters and improve the coherence and naturalness of the generated speech waveform. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flowchart of a speech synthesis method based on an AI large model in a multi-language scenario in one embodiment of the present invention; Figure 2 It is a schematic diagram of the steps of a speech synthesis method based on an AI large model in a multi-language scenario in one embodiment of the present invention; Figure 3 It is a structural block diagram of a speech synthesis device based on an AI large model in a multi-language scenario in one embodiment of the present invention; Figure 4 It is a schematic block diagram of the structure of a computer device according to an embodiment of the present invention.

[0018] The implementation, functional features and advantages of the present invention will be further described in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0020] Reference Figure 1 as well as Figure 2 In one embodiment of the present invention, a speech synthesis method based on an AI large model in a multilingual scenario is provided, comprising the following steps: Step S1, extracting text features from the input multilingual text to be synthesized to obtain features of the text to be synthesized; Step S2, generating speech synthesis parameters for the features of the text to be synthesized based on the fusion feature model to obtain speech synthesis parameters; wherein the pre-trained fusion feature model has a mapping relationship between speech synthesis parameters and text, and the speech synthesis parameters include fundamental frequency parameters, duration parameters, and spectrum parameters; Step S3, inputting the speech synthesis parameters into a pre-trained vocoder, processing multiple parameters of the speech synthesis parameters into intermediate feature vectors based on the vocoder, converting the intermediate feature vectors into discrete speech sample points, converting the discrete speech sample points into continuous speech waveforms, and obtaining synthesized speech corresponding to the multi-language text to be synthesized.

[0021] In this embodiment, the speech synthesis method based on the AI ​​large model in a multilingual scenario proposed by the present invention mainly includes: text feature extraction, speech synthesis parameter generation and vocoder processing, which aims to solve the problem of speech synthesis in a multilingual environment and improve the quality and efficiency of synthesized speech.

[0022] As described in step S1 above, it is crucial to extract features from the input multilingual text to be synthesized. Texts in different languages ​​have their own unique features. By extracting these features, it is possible to provide a basis for the subsequent accurate generation of speech synthesis parameters. The above features can capture information about the text at multiple levels such as vocabulary, grammar, and semantics, and are the key link in connecting text and speech. A variety of natural language processing technologies and tools can be used. For example, first, the text is lexically analyzed to identify words, parts of speech, and other information in different languages; then a syntactic analysis is performed to determine the structure of the sentence, such as the subject-predicate-object relationship; at the same time, a semantic analysis is also required to understand the meaning of the text. Through these analyses, rich text features can be obtained, including lexical features (such as the frequency of words, the appearance of specific words, etc.), grammatical structure features (such as sentence types, nesting of clauses, etc.), and semantic information features (such as themes, emotional tendencies, etc.). Then, appropriate feature embedding techniques can be used to convert these features into low-dimensional vector representations for subsequent processing.

[0023] As described in step S2 above, generating speech synthesis parameters based on the fusion feature model is one of the core steps of this method. Since different languages ​​have huge differences in speech performance, such as fundamental frequency, duration, spectrum and other parameters have different rules in different languages, a model that can integrate the characteristics of multiple languages ​​is needed to accurately generate these parameters. In this embodiment, the above-mentioned fusion feature model is a pre-trained AI large model. The mapping relationship between speech synthesis parameters and text established in the above-mentioned fusion feature model can fully consider the complex correspondence between multilingual text features and speech parameters, and ensure that the generated parameters conform to the speech rules of the corresponding language.

[0024] After the fusion feature model inputs the features of the text to be synthesized, it processes them through its internal structure and parameters. For the fundamental frequency parameters, the model calculates them based on the information related to intonation in the text. For example, when processing tonal languages, the change pattern of the fundamental frequency is determined based on factors such as the tone markers and semantic emotions of the vocabulary; for non-tonal languages, the tone of the sentence (such as declarative sentences, interrogative sentences, etc.) and the stress position are considered to adjust the fundamental frequency. The generation of duration parameters is related to the number of words in the text, the complexity of the grammar, and the language-specific speaking habits. For example, the complex clause structure in some languages ​​will lead to an increase in the speech duration of the corresponding part. The above model can accurately calculate the duration of each syllable or word based on the complexity of the grammatical structure. The generation of spectral parameters combines the timbre characteristics of the text, the pronunciation rules of the language, and the contextual semantics. Different pronunciation methods and speech timbre have different performances on the spectrum. The above model accurately generates spectral parameters by learning the pronunciation characteristics and semantic associations of different languages.

[0025] As described in step S3, the vocoder plays a key role in converting speech synthesis parameters into actual speech waveforms in the entire speech synthesis process. It can effectively process the fundamental frequency, duration, spectrum and other parameters obtained from the fusion feature model, convert these abstract parameters into audible speech, complete the conversion from digital information to analog speech, and ensure the quality and naturalness of the synthesized speech.

[0026] The above-mentioned vocoder first processes the input speech synthesis parameters into intermediate feature vectors. This process is an integration and conversion of the parameters so that the parameters can better adapt to the subsequent speech generation steps. Next, the intermediate feature vectors are converted into discrete speech sample points. This step is to further discretize the feature vectors in order to accurately control the generation of speech. Finally, the discrete speech sample points are converted into continuous speech waveforms to obtain the synthesized speech corresponding to the multi-language text to be synthesized. In this process, the vocoder needs to have the ability to process the fusion of multiple language parameters to ensure the coherence and naturalness of the synthesized speech when switching and mixing different languages.

[0027] In this embodiment, by generating fundamental frequency parameters, duration parameters, and spectrum parameters for the language text to be synthesized, the characteristics of various speech can be highlighted; at the same time, multiple parameters in the speech synthesis parameters are processed into intermediate feature vectors, the intermediate feature vectors are converted into discrete speech sample points, and the discrete speech sample points are converted into continuous speech waveforms to obtain synthesized speech corresponding to the multi-language text to be synthesized, which can effectively integrate the characteristics of different language parameters and improve the coherence and naturalness of the generated speech waveform.

[0028] The multilingual speech synthesis method in the embodiment of the present invention has significant advantages over the traditional method. It avoids the high cost and low efficiency of establishing a speech synthesis system for each language separately, and overcomes the defect of poor speech quality caused by simply piecing together multilingual models. By integrating the collaborative work of the feature model and the vocoder, it is possible to accurately and efficiently process multilingual texts, generate high-quality, natural and fluent synthesized speech, and meet the needs of multilingual interaction scenarios.

[0029] In one embodiment, processing multiple parameters of the speech synthesis parameters into intermediate feature vectors based on the vocoder includes: Based on the first layer of neurons of the vocoder, the fundamental frequency parameter is weighted to obtain a weighted fundamental frequency parameter; the duration parameter is normalized to make the duration parameter fall within a standard range; Performing feature enhancement processing on the spectrum parameters based on the second layer neurons of the vocoder to obtain enhanced spectrum parameters; The normalized duration parameter, weighted fundamental frequency parameter and enhanced spectrum parameter are transmitted to the third layer of neurons of the vocoder to perform feature fusion processing to obtain the intermediate feature vector.

[0030] In this embodiment, the fundamental frequency parameter plays an important role in speech, which determines the pitch of the speech. The importance and expression of the fundamental frequency are different in different languages ​​and contexts. Through weighted processing, the influence of the fundamental frequency parameter can be adjusted according to specific needs to make it more in line with the characteristics of the current multilingual text to be synthesized. The first layer of neurons of the vocoder can multiply the fundamental frequency parameters according to a pre-set weight matrix. The above-mentioned weight matrix can be obtained by learning a large amount of multilingual speech data in advance, which reflects the relative importance of the fundamental frequency parameters in different situations. For example, in some languages, the change of the fundamental frequency is very critical to the distinction of semantics, and the corresponding weight can be set higher at this time; while in other languages, the role of the fundamental frequency is relatively small, and the weight can be reduced accordingly.

[0031] The duration parameter indicates the duration of each syllable or word in speech. Different languages ​​have different speaking speeds and pronunciation habits, resulting in large differences in the range of values ​​of the duration parameter. Normalization can unify the duration parameters into a standard range, which is convenient for subsequent processing and fusion. This can avoid the naturalness and coherence of the synthesized speech being affected by the large differences in duration between different languages. Common normalization methods can be used, such as minimum-maximum normalization. First, determine the range of the duration parameter, and then map it to a standard interval, such as [0,1], through linear transformation. This allows the duration parameters of different languages ​​to be compared and processed on the same scale. In the normalization process, factors such as language type and complexity of text content can also be considered, and the normalization method can be appropriately adjusted to better adapt to multilingual scenarios.

[0032] Spectral parameters reflect the characteristics of speech in the frequency domain, including the intensity distribution of different frequency components. In multilingual scenarios, the spectral characteristics of different languages ​​are significantly different. Feature enhancement processing can highlight important information in spectral parameters and improve the quality and recognizability of synthesized speech. For example, for phonemes with specific frequency characteristics in certain languages, enhancement processing can make them more obvious in synthesized speech.

[0033] The second layer of neurons can process the spectrum parameters using methods such as nonlinear transformation and convolution operations. For example, a convolutional neural network layer can be used to perform convolution operations on the spectrum parameters to extract more representative features and enhance the response of specific frequency areas. It is also possible to design specific filter groups to filter the spectrum according to the spectrum characteristics of different languages ​​to highlight important frequency components.

[0034] After the previous processing steps, the fundamental frequency, duration and spectrum parameters are optimized respectively. These parameters are passed to the third layer of neurons for feature fusion, which can comprehensively consider the information of each parameter and generate an intermediate feature vector that can fully reflect the characteristics of the speech. This intermediate feature vector will serve as the basis for the subsequent speech generation steps to ensure that the synthesized speech can accurately reflect the characteristics of the multilingual text to be synthesized in many aspects.

[0035] The third layer of neurons can be used for feature fusion by using methods such as fully connected layers and attention mechanisms. For example, a fully connected layer is used to linearly combine the normalized duration parameter, weighted base frequency parameter, and enhanced spectrum parameter, and then a nonlinear activation function is used to obtain the intermediate feature vector. Alternatively, the attention mechanism can be used to dynamically adjust the fusion weight according to the importance of different parameters to achieve more effective feature fusion.

[0036] In one embodiment, converting the intermediate feature vector into discrete speech sample points, and converting the discrete speech sample points into a continuous speech waveform includes: Based on the fourth layer neurons of the vocoder, the intermediate feature vector is subjected to dimensionality reduction processing by a principal component analysis algorithm to obtain a reduced-dimensional feature vector; Converting the dimension-reduced feature vector into discrete speech sample points based on the fifth layer of neurons of the vocoder; According to the time series relationship of the discrete speech sample points, the discrete speech sample points are converted into a continuous speech waveform through an interpolation algorithm.

[0037] In this embodiment, in speech synthesis based on the AI ​​large model in a multilingual scenario, converting intermediate feature vectors into discrete speech sample points and then further converting them into continuous speech waveforms is a key step in achieving high-quality speech synthesis.

[0038] The above intermediate feature vectors usually have high dimensions, which will increase the computational complexity, storage requirements, and may introduce noise and redundant information. By reducing the dimension of the intermediate feature vectors through the principal component analysis algorithm, the main feature components can be extracted, the dimension of the data can be reduced, and key information can be retained. This can improve the efficiency of subsequent processing, reduce computational costs, and help improve the quality and stability of synthesized speech.

[0039] The fourth layer of neurons in the vocoder above is responsible for performing dimensionality reduction. The principal component analysis algorithm finds the main component directions in the data, i.e., the principal components, by calculating the covariance matrix of the eigenvectors. Then, the intermediate eigenvectors are projected onto these principal components to obtain the eigenvectors after dimensionality reduction. In practical applications, the number of principal components to be retained can be selected as needed to balance the degree of data compression and information retention.

[0040] Converting the reduced-dimensional feature vector to discrete speech sample points is an important step in converting the abstract feature representation into a specific speech signal. Discrete speech sample points can more intuitively reflect the time series characteristics of speech, providing a basis for subsequent conversion to a continuous speech waveform. The fifth layer of neurons in the above-mentioned vocoder performs this conversion process. The specific conversion method can be determined according to the design and training method of the vocoder. For example, a neural network model can be used to map the reduced-dimensional feature vector to a discrete speech sample point space, or it can be converted through specific mathematical formulas and algorithms. During the conversion process, it is necessary to consider the amplitude, frequency and other characteristics of the speech to ensure that the generated speech sample points can accurately reflect the characteristics of the synthesized speech.

[0041] Discrete speech sample points are discrete points in time, while the actual speech signal is a continuous waveform. By converting discrete speech sample points into a continuous speech waveform through an interpolation algorithm, the synthesized speech can be more natural, smooth, and close to the real speech signal. The interpolation algorithm can estimate the intermediate continuous value based on the relationship between discrete sample points, thereby restoring the complete speech waveform.

[0042] According to the time series relationship of discrete speech sample points, an interpolation algorithm is used for conversion. Common interpolation algorithms include linear interpolation and cubic spline interpolation. The above algorithm calculates the value of the middle point based on the values ​​of adjacent discrete sample points to obtain a continuous speech waveform. When selecting an interpolation algorithm, it is necessary to consider the complexity of the algorithm, the computational efficiency, and the quality requirements of the synthesized speech. For example, cubic spline interpolation can usually provide a smoother continuous waveform, but the computational complexity is relatively high.

[0043] In this embodiment, efficient conversion from intermediate feature vectors to discrete speech sample points and then to continuous speech waveforms is achieved. The above method can effectively handle speech synthesis tasks in multilingual scenarios and improve the quality and naturalness of synthesized speech. At the same time, through the application of dimensionality reduction processing and interpolation algorithms, the computational cost is reduced and the processing efficiency is improved.

[0044] In one embodiment, the step of extracting text features from the input multilingual text to be synthesized to obtain features of the text to be synthesized includes: Performing lexical analysis, syntactic analysis and semantic analysis on the multilingual text to be synthesized to obtain lexical features, grammatical structure features and semantic information features in the text; The vocabulary features, grammatical structure features and semantic information features are respectively converted into low-dimensional vector representations; and each low-dimensional vector representation is standardized; The standardized low-dimensional vector representations are fused to obtain the text features to be synthesized.

[0045] In this embodiment, different languages ​​have unique vocabulary, grammar and semantic rules. Through analysis, we can deeply understand all aspects of the text and provide a basis for subsequent accurate feature extraction.

[0046] Lexical analysis can identify lexical features such as words and parts of speech in a text, which helps to understand the basic components of the text. For example, in different languages, the distribution and usage of parts of speech such as nouns, verbs, and adjectives vary. Lexical analysis can accurately grasp these characteristics.

[0047] Syntactic analysis determines the structure of a sentence, such as the subject-verb-object relationship, clause nesting and other grammatical structural features. This is crucial for understanding the logical relationship and expression of a text. The syntactic structure of different languages ​​varies greatly. For example, the subject-verb-object structure in English is more obvious, while some inflected languages ​​may have more complex grammatical changes.

[0048] Semantic analysis mines the meaning and semantic information features of the text, including themes, sentiment, etc. This can enable the synthesized speech to better convey the true intention of the text and improve the expressiveness and comprehensibility of the speech.

[0049] In this embodiment, natural language processing tools and techniques can be used to perform lexical analysis, syntactic analysis, and semantic analysis. For example, a part-of-speech tagging tool is used to perform lexical analysis to determine the part of speech of each word; a syntactic analyzer is used to parse the sentence structure to identify components such as subject, predicate, and object; and a semantic analysis model is used to extract the theme and sentiment information of the text.

[0050] The purpose of converting vocabulary features, grammatical structure features, and semantic information features into low-dimensional vector representations is to facilitate subsequent processing and fusion. High-dimensional text features are difficult to calculate and analyze directly, while low-dimensional vector representations can compress complex feature information into a processable space while retaining key information.

[0051] For lexical features, low-dimensional vectors can represent the semantics and contextual relationships of words. For example, by converting words into low-dimensional vectors through word vector models (such as Word2Vec, GloVe, etc.), similar words are close in the vector space, so that the semantic associations between words can be captured. The low-dimensional vector representation of grammatical structure features can reflect the structural pattern of sentences. For example, by encoding different syntactic structures and converting them into vector form, it is convenient to subsequently analyze and compare the grammatical structures of different sentences. The low-dimensional vector of semantic information features can reflect the theme and emotional tendency of the text. Converting semantic information into vectors through sentiment analysis models or topic models helps synthetic speech better convey the sentiment and theme of the text.

[0052] In this embodiment, various feature embedding techniques can be used to convert text features into low-dimensional vector representations. For example, for vocabulary features, a pre-trained word vector model can be used; for grammatical structure features and semantic information features, automatic encoding or feature extraction can be performed through a neural network model to convert them into low-dimensional vectors.

[0053] The purpose of standardizing each low-dimensional vector representation is to make different features comparable in the same feature space. Different types of features may have different numerical ranges and distributions. Standardization can eliminate these differences, making subsequent fusion and analysis more accurate and effective. For example, for vocabulary feature vectors and grammatical structure feature vectors, if standardization is not performed, some features will dominate the fusion process due to differences in numerical ranges, affecting the final feature representation. Through standardization, different features can be fused on the same scale to improve the quality and stability of the features. Common standardization methods include Z-score standardization, minimum-maximum standardization, etc. These methods can adjust the values ​​of the vectors to a specific range or a standard distribution with a mean of 0 and a variance of 1, thereby achieving the purpose of standardization.

[0054] The above fusion of the standardized low-dimensional vector representation to obtain the features of the text to be synthesized is to comprehensively consider the multiple aspects of the text and form a comprehensive feature representation. The fused features can better reflect the overall information of the text and provide richer input for the subsequent generation of speech synthesis parameters. For example, the fusion of vocabulary features, grammatical structure features and semantic information features can make the synthesized speech more accurate and natural in vocabulary selection, grammatical correctness and semantic expression. Different features complement each other and improve the quality and expressiveness of the synthesized speech.

[0055] In this embodiment, a variety of methods can be used for feature fusion, such as splicing, weighted summation, neural network fusion, etc. For example, the standardized vocabulary feature vector, grammatical structure feature vector and semantic information feature vector are spliced ​​together to form a longer feature vector; or by setting different weight coefficients, each feature vector is weighted summed to obtain a fused feature vector. Neural network fusion can use deep neural networks to automatically learn the fusion method between different features, thereby improving the fusion effect and adaptability.

[0056] In one embodiment, the generating speech synthesis parameters based on the features of the text to be synthesized based on the fusion feature model to obtain the speech synthesis parameters includes: Inputting the text features to be synthesized into the input layer of the fusion feature model, and the input layer preprocesses the data format thereof; Performing deep feature mapping on the preprocessed text features by fusing multiple hidden layers in the feature model to obtain the speech synthesis parameters; In the process of deep feature mapping, the intonation information features, language type features and semantic sentiment features in the text are calculated, and the fundamental frequency parameter estimation value is generated through neuron connection and weight adjustment; According to the number of words in the text, the complexity of the grammatical structure, and the speaking speed characteristics of different languages, the long short-term memory unit in the hidden layer is used to calculate the duration parameters of each syllable or word; The corresponding spectrum parameters are calculated by the spectrum generation subunit by combining the timbre characteristics of the text, the language pronunciation rule characteristics and the contextual semantic association characteristics.

[0057] In this embodiment, the above preprocessing may include operations such as data cleaning, normalization, and standardization, the purpose of which is to remove noise, unify data scale, make input data more standardized and stable, and improve the training efficiency and performance of the model.

[0058] Data cleaning can remove abnormal characters, erroneous marks, etc. in the text. For example, for text obtained from a web page, there may be HTML tags, special characters, etc., which need to be cleaned.

[0059] Normalization can map the data value to a specific range, such as [0, 1] or [-1, 1]. Standardization can make the data have zero mean and unit variance. Common methods include Z-score standardization. These operations can improve the convergence speed and stability of the model.

[0060] The above hidden layers play a key role in the fusion feature model. By performing deep feature mapping on the preprocessed text features, the complex relationship between text features and speech synthesis parameters can be mined. Different hidden layers can learn different levels of feature representation, from low-level vocabulary features to high-level semantic and contextual features, and gradually extract more representative and abstract features. Deep feature mapping can improve the expressiveness and generalization ability of the model, enabling the model to better adapt to speech synthesis tasks of different languages ​​and text types.

[0061] The above fusion feature model can adopt a variety of neural network structures, such as convolutional neural network (CNN), recurrent neural network (RNN), long short-term memory network (LSTM), etc. The above network structures can be selected and combined according to different task requirements and data characteristics. For example, CNN can be used to extract local features of text, such as n-gram features of vocabulary; RNN and LSTM can process sequence data to capture contextual information and long-term dependencies of text. Multiple hidden layers can gradually learn more complex feature representations through different neuron connections and activation functions.

[0062] The fundamental frequency parameter determines the high and low pitch of the sound in speech, and is crucial for speech synthesis in different languages. By calculating based on the intonation information features, language type features, and semantic sentiment features in the text, it is possible to accurately generate an estimate of the fundamental frequency parameters that conform to the text content and language characteristics. The intonation information features can reflect the tone, emphasis, and emotional expression of the text. Different intonations will lead to changes in the fundamental frequency. The language type features consider factors such as the tone system and stress rules of different languages, such as the tonal language characteristics of Chinese and the stress language characteristics of English. The semantic sentiment features can adjust the fundamental frequency according to the emotional tendency of the text. For example, the fundamental frequency may increase when expressing strong emotions such as excitement and anger.

[0063] In this embodiment, in the hidden layer of the fusion feature model, the generation of the fundamental frequency parameters can be achieved through preset neuron connections and weight adjustment. For example, a special fundamental frequency generation subnetwork is set, which receives text features as input and outputs fundamental frequency parameter estimates through neuron calculation and activation function. The supervised learning method can be used to train the model through a large amount of labeled data, adjust the weights of the neurons, and enable the model to learn the relationship between different features and fundamental frequency parameters.

[0064] The above duration parameters determine the duration of each syllable or word in the speech, and have an important impact on the naturalness and fluency of the speech. According to the number of words in the text, the complexity of the grammatical structure, and the speaking speed characteristics of different languages, the duration parameters that conform to the language habits and text content can be generated.

[0065] Texts with a large number of words may take longer to pronounce, and complex grammatical structures may cause some parts of the pronunciation to be prolonged or shortened. Different languages ​​also have different speaking habits, for example, some languages ​​speak faster while others speak slower. Considering these factors can make the synthesized speech more natural and idiomatic.

[0066] The long short-term memory (LSTM) unit in the hidden layer can effectively process sequence data, capture the contextual information and long-term dependencies of the text, and calculate the duration parameters of each syllable or word. LSTM can predict the duration of each syllable or word based on the input text features and historical information.

[0067] During the training process, the model can be supervised by using labeled data, so that the model can learn the relationship between different text features and duration parameters. At the same time, the duration parameters can be adjusted and optimized by combining language models and phonetics knowledge.

[0068] The above-mentioned spectrum parameters reflect the timbre and frequency distribution characteristics of speech, and play an important role in the quality and recognizability of synthesized speech. By combining the timbre characteristics of the text, the language pronunciation rule characteristics, and the context semantic association characteristics, accurate spectrum parameters can be generated to make the synthesized speech more natural and realistic. The timbre characteristics can adjust the timbre of the speech according to the theme and emotion of the text. For example, a bright timbre can be used to express cheerful emotions. The language pronunciation rule characteristics consider the pronunciation methods, vowel and consonant characteristics of different languages, such as the pronunciation of vowels in English and the pronunciation of initials and finals in Chinese. The context semantic association features can adjust the spectrum parameters according to the context information of the text, making the synthesized speech more semantically coherent.

[0069] In this embodiment, the spectrum generation subunit can be specifically responsible for calculating the spectrum parameters. This subunit can receive text features as input and generate spectrum parameters through a series of calculations and transformations. It can use neural network models, filter groups and other technologies to realize the calculation of spectrum parameters. For example, a convolutional neural network (CNN) is used to process text features and extract features related to the spectrum; or a filter group is used to analyze the speech signal to obtain spectrum parameters. During the training process, the model can also be supervised by using labeled data to adjust the parameters of the model so that the model can learn the relationship between different features and spectrum parameters.

[0070] In one embodiment, after obtaining the synthesized speech corresponding to the multi-language text to be synthesized, the method includes: Acquire multiple language types in the multilingual text to be synthesized, and acquire multi-dimensional text features of the multilingual text to be synthesized; the multi-dimensional text features include text length, text data volume, and the number of characters in the text; Generate a key character array based on the multiple language types and multi-dimensional text features; Acquiring speech features of the synthesized speech; generating a communication key based on the key character array and the speech characteristics of the synthesized speech; The synthesized speech is encrypted based on the communication key and then stored.

[0071] In the present embodiment, determining the language type in the multilingual text to be synthesized is an important basis for the subsequent generation of the key character array. Different languages ​​have different characteristics and encodings, and understanding the language type helps to carry out encryption processing in a targeted manner. The above-mentioned multidimensional text features include the number of characters in the text length, text data volume and the text, and these features can provide scale and complexity information about the text, which plays a supporting role in the encryption process. For example, longer text may require more complicated encryption methods, and larger data volumes may require higher encryption strength.

[0072] The language type in the text can be identified through natural language processing technology. Common methods include judging based on character encoding, vocabulary features, grammatical structure, etc. For example, the language type is determined based on the character set range of a specific language, the frequency of occurrence of a specific vocabulary, or a specific grammatical rule. For the length of the text, the number of characters or bytes of the text can be directly identified. The amount of text data can be determined by calculating the size of the text file. The number of characters is the total number of characters in the text, which can be counted by traversing the text.

[0073] The above key character array is the basis for generating communication keys. Combining language types and multi-dimensional text features can generate targeted and complex key character arrays to improve the security of encryption. Different language types and text features can produce different key character combinations, increasing the difficulty of cracking.

[0074] The language type and multi-dimensional text features can be used as inputs to generate a key character array through calculation and conversion. For example, a specific character set or encoding method can be selected according to the language type, and then hash operations or other mathematical transformations can be performed in combination with features such as text length, data volume, and number of characters to generate a series of key characters. These key characters can be arranged in the form of an array for subsequent use. Alternatively, the characters corresponding to the language type and multi-dimensional text features can be directly added to a blank array of multiple rows and columns to form the above-mentioned key character array.

[0075] The speech features of the synthesized speech can provide information about the speech signal, which can be used to generate a more secure and personalized communication key. Speech features can include frequency features, amplitude features, duration features, etc. These features can reflect the unique nature of speech and increase the complexity of encryption. The features of synthesized speech can be extracted using speech signal processing technology. For example, the spectral features of speech can be analyzed by methods such as Fourier transform to obtain frequency distribution information; the amplitude features can be obtained by measuring the amplitude changes of speech signals; and the duration features can be obtained by calculating the duration of speech. These features can be represented in numerical vectors or other forms for subsequent processing.

[0076] Generating the communication key is the core step of the entire encryption process. Combining the key character array and the voice characteristics of the synthesized speech can generate a key with high security and uniqueness, ensuring the confidentiality of the synthesized speech during storage and transmission. The key generation process should be complex enough to prevent it from being cracked.

[0077] In this embodiment, a variety of methods can be used to generate a communication key. One possible method is to merge the key character array and the voice feature. For example, the voice feature can be input as a parameter into an encryption algorithm, and the algorithm is initialized using the key character array. The encryption algorithm can be a symmetric encryption algorithm (such as AES) or an asymmetric encryption algorithm (such as RSA), which generates a communication key by performing an encryption operation on the input data. Another method is to use a hash function to perform a hash operation on the key character array and the voice feature to obtain a key value of a fixed length.

[0078] Encrypting synthesized speech can protect the confidentiality of speech content and prevent unauthorized access and use. Encrypted speech is more secure during storage, and even if the storage device is stolen or attacked, it is difficult to obtain the original speech content. Use the generated communication key to encrypt the synthesized speech. You can choose a suitable encryption algorithm, such as symmetric encryption algorithm or asymmetric encryption algorithm. For symmetric encryption algorithms, the same key is used for encryption and decryption; for asymmetric encryption algorithms, the public key is used for encryption and the private key is used for decryption. The encrypted speech can be stored in a local storage device, a cloud server, or other storage media. When storing, additional security measures such as data backup and access control can also be taken to further improve the security of the data.

[0079] In one embodiment, the speech features include speech duration, speech data volume, and tone simulation curve; Generating a communication key based on the key character array and the voice feature of the synthesized voice, comprising: The speech duration and the amount of speech data are standardized and combined into a combination of digital characters; Discretely distribute the characters in the digital character combination into the key character array to obtain a modified key character array; Adding characters in the modified key character array to the tone simulation curve one by one in sequence, and generating a rectangular frame in the tone simulation curve according to the speech duration; On the tone simulation curve, all characters in the rectangular frame are obtained and combined to obtain the communication key.

[0080] In this embodiment, the above-mentioned speech duration reflects the duration of the synthesized speech, and can provide information of the time dimension for the encryption process. The above-mentioned speech data volume indicates the size of the storage space occupied by the synthesized speech, and can also be used as one of the factors considered for encryption. The above-mentioned tone simulation curve reflects the tone changes of the speech at different time points, which has the uniqueness of the speech and can increase the complexity and security of encryption.

[0081] Standardizing speech duration and speech data volume can convert values ​​of different ranges into a unified format for subsequent processing. This can avoid the encryption process being unstable or unreliable due to large differences in the original data. Combining it into a digital character combination can fuse the two features together into a single representation, which is convenient for manipulation in subsequent steps.

[0082] Specifically, for the speech duration, a normalization method can be used to map it to a specific interval, such as [0, 1]. Normalization can be achieved by dividing the speech duration by a reference duration (such as the longest possible speech duration). For the amount of speech data, a similar method can be used to perform grading or normalization according to the size of the data. The standardized speech duration and speech data volume are then converted into digital character representations, such as converting the numerical value into a specific character code and then combining two digital characters together.

[0083] The above-mentioned discrete distribution of the characters in the digital character combination into the key character array can make the key character array more complex and random. It can increase the difficulty of cracking and improve the security of encryption. By combining the voice feature with the key character array, the generated communication key is more dependent on the specific synthesized voice, which increases the uniqueness of the key. Specifically, the characters in the digital character combination can be inserted into different positions in the key character array one by one using random or preset rules. For example, the insertion position can be determined according to the value of the digital character, or a random number generator can be used to determine the insertion position.

[0084] By adding characters in the modified key character array to the tone simulation curve, the voice features can be further integrated with the key characters. The dynamic change characteristics of the tone simulation curve make the encryption process more complicated and unpredictable. Specifically, by adding characters in the modified key character array to the tone simulation curve one by one in a certain order (e.g., from left to right or from top to bottom), characters can be marked at specific positions on the curve.

[0085] Generating a rectangular frame can delimit a specific area on the tone simulation curve for extracting the final communication key, thereby increasing the pertinence and security of encryption. Specifically, a rectangular frame is generated in the tone simulation curve according to the speech duration, and the length or width of the rectangular frame can be determined according to the speech duration, and it is placed in a suitable area position on the curve. In this embodiment, the rectangular frame delimits a specific range on the tone simulation curve, and this range will be used to extract characters closely related to the synthesized speech, thereby generating a communication key. In this way, it is ensured that the generated key is highly associated with the specific synthesized speech features, thereby increasing the pertinence and security of encryption. The speech duration, as a dynamic feature, reflects the actual length of the synthesized speech. Combining it with the static tone simulation curve makes the generated rectangular frame have the characteristic of dynamic change. Speech of different lengths will produce rectangular frames of different sizes or positions, increasing the complexity and unpredictability of the encryption process and increasing the difficulty of cracking.

[0086] Finally, characters are obtained from the rectangular box on the tone simulation curve and combined into a communication key, and the key used to encrypt the synthesized speech is finally determined. This key combines the characteristics of the multilingual text to be synthesized, the characteristics of the synthesized speech, and a series of processing steps. It is highly complex and unique and can effectively protect the security of the synthesized speech. Specifically, by traversing all positions in the rectangular box on the tone simulation curve, the characters are extracted. Then these characters are combined in order to form a communication key. At the same time, the communication key can also be further processed using a preset encoding method or encryption algorithm to improve its security.

[0087] In a specific embodiment, generating a rectangular frame in the tone simulation curve according to the speech duration includes: Length determination: The length of the rectangular box is determined based on the speech duration and a preset ratio coefficient. For example, the speech duration is multiplied by a specific value to obtain the length of the rectangular box on the time axis of the tone simulation curve. In this way, a longer speech duration will generate a longer rectangular box, covering more of the tone simulation curve area.

[0088] Width determination: A fixed width value can be preset, or the width of the rectangular box can be dynamically determined based on other parameters related to speech characteristics (such as the amount of speech data, language type, etc.). The width should be selected to accommodate enough information on the tone simulation curve while not being too wide to reduce the security of encryption.

[0089] Place the rectangular box: Determine the starting position of the rectangular box on the tone simulation curve. One possible way is to determine one end point of the rectangular box according to the corresponding position of the speech start time point on the tone simulation curve, and then determine the other end point according to the length of the rectangular box. In this way, the rectangular box can correspond to the time axis of the synthesized speech and better reflect the characteristics of the speech.

[0090] The direction of the rectangular frame can be determined according to a preset rule, such as from left to right, from top to bottom or other specific directions. The direction can also be associated with the order in which characters are added to the tone simulation curve, so that the characters in the rectangular frame can be more conveniently extracted in subsequent steps.

[0091] In the above manner, the rectangular box generated in the tone simulation curve according to the speech duration can effectively integrate the speech features and the key generation process, playing a key role in generating secure and reliable communication keys.

[0092] Reference Figure 3 In another embodiment of the present invention, a speech synthesis device based on an AI large model in a multi-language scenario is provided, comprising: An extraction unit, used for extracting text features from the input multi-language text to be synthesized, to obtain features of the text to be synthesized; A generating unit, configured to generate speech synthesis parameters for the features of the text to be synthesized based on a fusion feature model to obtain speech synthesis parameters; wherein the pre-trained fusion feature model has a mapping relationship between speech synthesis parameters and text, and the speech synthesis parameters include fundamental frequency parameters, duration parameters, and spectrum parameters; The synthesis unit is used to input the speech synthesis parameters into a pre-trained vocoder, process multiple parameters of the speech synthesis parameters into intermediate feature vectors based on the vocoder, convert the intermediate feature vectors into discrete speech sample points, and convert the discrete speech sample points into continuous speech waveforms to obtain synthesized speech corresponding to the multi-language text to be synthesized.

[0093] In this embodiment, for the specific implementation of each unit in the above device embodiment, please refer to the above method embodiment, which will not be repeated here.

[0094] Reference Figure 4 In an embodiment of the present invention, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 4As shown. The computer device includes a processor, a memory, a display screen, an input device, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above method is implemented.

[0095] Those skilled in the art will understand that Figure 4 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied.

[0096] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0097] In summary, the speech synthesis method, apparatus and device based on the AI ​​large model in a multilingual scenario provided in the embodiments of the present invention include: extracting text features of the input multilingual text to be synthesized to obtain features of the text to be synthesized; generating speech synthesis parameters for the features of the text to be synthesized based on a fusion feature model to obtain speech synthesis parameters; wherein the pre-trained fusion feature model has a mapping relationship between speech synthesis parameters and text, and the speech synthesis parameters include fundamental frequency parameters, duration parameters, and spectrum parameters; inputting the speech synthesis parameters into a pre-trained vocoder, processing multiple parameters in the speech synthesis parameters into intermediate feature vectors based on the vocoder, converting the intermediate feature vectors into discrete speech sample points, and converting the discrete speech sample points into continuous speech waveforms to obtain synthesized speech corresponding to the multilingual text to be synthesized. In the present invention, by generating fundamental frequency parameters, duration parameters, and spectrum parameters for the language text to be synthesized, the characteristics of various speech can be highlighted; at the same time, multiple parameters in the speech synthesis parameters are processed into intermediate feature vectors, the intermediate feature vectors are converted into discrete speech sample points, and the discrete speech sample points are converted into continuous speech waveforms to obtain synthesized speech corresponding to the multi-language text to be synthesized, which can effectively integrate the characteristics of different language parameters and improve the coherence and naturalness of the generated speech waveform.

[0098] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided by the present invention and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM.

[0099] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0100] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A speech synthesis method based on an AI large model in a multilingual scenario, characterized in that: The following steps are involved: Extracting text features from the input multilingual text to be synthesized to obtain features of the text to be synthesized; Based on the fusion feature model, speech synthesis parameters are generated for the features of the text to be synthesized to obtain speech synthesis parameters; wherein the pre-trained fusion feature model has a mapping relationship between speech synthesis parameters and text, and the speech synthesis parameters include fundamental frequency parameters, duration parameters, and spectrum parameters; The speech synthesis parameters are input into a pre-trained vocoder, multiple parameters of the speech synthesis parameters are processed into intermediate feature vectors based on the vocoder, the intermediate feature vectors are converted into discrete speech sample points, and the discrete speech sample points are converted into continuous speech waveforms to obtain synthesized speech corresponding to the multi-language text to be synthesized.

2. The speech synthesis method based on AI large model in a multilingual scenario according to claim 1 is characterized in that: Processing multiple parameters of the speech synthesis parameters into intermediate feature vectors based on the vocoder includes: Based on the first layer of neurons of the vocoder, weighted processing is performed on the fundamental frequency parameter to obtain a weighted fundamental frequency parameter; and normalization processing is performed on the duration parameter to make the duration parameter fall within a standard range; Performing feature enhancement processing on the spectrum parameters based on the second layer neurons of the vocoder to obtain enhanced spectrum parameters; The normalized duration parameter, weighted fundamental frequency parameter and enhanced spectrum parameter are transmitted to the third layer of neurons of the vocoder to perform feature fusion processing to obtain the intermediate feature vector.

3. The speech synthesis method based on AI large model in a multilingual scenario according to claim 1 is characterized in that: Converting the intermediate feature vector into discrete speech sample points, and converting the discrete speech sample points into a continuous speech waveform, comprising: Based on the fourth layer neurons of the vocoder, the intermediate feature vector is subjected to dimensionality reduction processing by a principal component analysis algorithm to obtain a reduced-dimensional feature vector; Converting the dimension-reduced feature vector into discrete speech sample points based on the fifth layer of neurons of the vocoder; According to the time series relationship of the discrete speech sample points, the discrete speech sample points are converted into a continuous speech waveform through an interpolation algorithm.

4. The speech synthesis method based on AI large model in a multilingual scenario according to claim 1 is characterized in that: The step of extracting text features from the input multilingual text to be synthesized to obtain features of the text to be synthesized includes: Performing lexical analysis, syntactic analysis and semantic analysis on the multilingual text to be synthesized to obtain lexical features, grammatical structure features and semantic information features in the text; The vocabulary features, grammatical structure features and semantic information features are respectively converted into low-dimensional vector representations; and each low-dimensional vector representation is standardized; The standardized low-dimensional vector representations are fused to obtain the text features to be synthesized.

5. The speech synthesis method based on AI large model in a multilingual scenario according to claim 1 is characterized in that: The generating speech synthesis parameters based on the features of the text to be synthesized based on the fusion feature model to obtain the speech synthesis parameters includes: Inputting the text features to be synthesized into the input layer of the fusion feature model, and the input layer preprocesses the data format thereof; Performing deep feature mapping on the preprocessed text features by fusing multiple hidden layers in the feature model to obtain the speech synthesis parameters; In the process of deep feature mapping, the intonation information features, language type features and semantic sentiment features in the text are calculated, and the fundamental frequency parameter estimation value is generated through neuron connection and weight adjustment; According to the number of words in the text, the complexity of the grammatical structure, and the speaking speed characteristics of different languages, the long short-term memory unit in the hidden layer is used to calculate the duration parameters of each syllable or word; The corresponding spectrum parameters are calculated by the spectrum generation subunit by combining the timbre characteristics of the text, the language pronunciation rule characteristics and the contextual semantic association characteristics.

6. The speech synthesis method based on AI large model in a multilingual scenario according to claim 1 is characterized in that: After obtaining the synthesized speech corresponding to the multi-language text to be synthesized, the method includes: Acquire multiple language types in the multilingual text to be synthesized, and acquire multi-dimensional text features of the multilingual text to be synthesized; the multi-dimensional text features include text length, text data volume, and the number of characters in the text; Generate a key character array based on the multiple language types and multi-dimensional text features; Acquiring speech features of the synthesized speech; generating a communication key based on the key character array and the speech characteristics of the synthesized speech; The synthesized speech is encrypted based on the communication key and then stored.

7. The speech synthesis method based on AI large model in a multilingual scenario according to claim 6 is characterized in that: The speech features include speech duration, speech data volume and tone simulation curve; Generating a communication key based on the key character array and the voice feature of the synthesized voice, comprising: The speech duration and the amount of speech data are standardized and combined into a combination of digital characters; Discretely distribute the characters in the digital character combination into the key character array to obtain a modified key character array; Adding characters in the modified key character array to the tone simulation curve one by one in sequence, and generating a rectangular frame in the tone simulation curve according to the speech duration; On the tone simulation curve, all characters in the rectangular frame are obtained and combined to obtain the communication key.

8. A speech synthesis device based on an AI large model in a multi-language scenario, characterized in that: include: An extraction unit, used for extracting text features from the input multi-language text to be synthesized, to obtain features of the text to be synthesized; A generating unit, configured to generate speech synthesis parameters for the features of the text to be synthesized based on a fusion feature model to obtain speech synthesis parameters; wherein the pre-trained fusion feature model has a mapping relationship between speech synthesis parameters and text, and the speech synthesis parameters include fundamental frequency parameters, duration parameters, and spectrum parameters; The synthesis unit is used to input the speech synthesis parameters into a pre-trained vocoder, process multiple parameters of the speech synthesis parameters into intermediate feature vectors based on the vocoder, convert the intermediate feature vectors into discrete speech sample points, and convert the discrete speech sample points into continuous speech waveforms to obtain synthesized speech corresponding to the multi-language text to be synthesized.

9. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images