Pronunciation duration prediction method and related equipment based on duration prediction model

Abstract

The present invention relates to the field of artificial intelligence, and discloses a method for predicting pronunciation duration based on a duration prediction model and related equipment. The method is mainly used in intelligent diagnosis and treatment systems, including: obtaining medical text data and corresponding audio sequences generated by the intelligent diagnosis and treatment system, converting the medical text data into a phoneme pronunciation sequence, and inputting it into a preset duration prediction model, vectorizing each phoneme in the phoneme pronunciation sequence, and calculating the implicit feature vector and multidimensional Gaussian distribution corresponding to each phoneme vector; aligning each audio frame in the audio sequence and each phoneme in the multidimensional Gaussian distribution to determine the pronunciation duration of each phoneme. The present invention predicts pronunciation duration by combining context information with a bidirectional long short-term memory neural network in a preset duration prediction model, thereby improving the accuracy of the prediction. In addition, the present invention also relates to the field of blockchain, and medical text data and audio sequences can be stored in the blockchain.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a pronunciation duration prediction method based on a duration prediction model and related equipment. Background Art

[0002] With the development of the Internet, intelligent voice technology has also been developed and gradually applied to scenarios such as navigation, voice broadcasting, and voice interaction. It can also be applied to digital medical scenarios such as intelligent diagnosis and treatment and online consultation, bringing great convenience to users' lives. Speech synthesis is a more important processing process in intelligent voice technology, but in related technologies, the effect of speech synthesis is poor, and it cannot achieve accurate synthesis of sound, which leads to the synthesized speech not being realistic enough. Accurate phoneme duration prediction has always been a crucial step in speech synthesis. The phoneme duration model can depict the intonation and importance of speech, thereby improving the expressiveness and naturalness of synthesized speech.

[0003] The existing prediction of phoneme pronunciation duration is mainly based on models, but it cannot fully consider contextual information, and its duration prediction result only considers the mean of the multi-dimensional Gaussian distribution, resulting in low accuracy of pronunciation duration prediction. Summary of the invention

[0004] The main purpose of the present invention is to solve the technical problem of low accuracy in pronunciation duration prediction in the prior art.

[0005] The first aspect of the present invention provides a pronunciation duration prediction method based on a duration prediction model, and the pronunciation duration prediction method based on the duration prediction model includes: obtaining medical text data and a corresponding audio sequence generated by an intelligent diagnosis and treatment system, and converting the medical text data into a phoneme pronunciation sequence, wherein the phoneme pronunciation sequence includes multiple phonemes; inputting the phoneme pronunciation sequence into a preset duration prediction model, vectorizing each phoneme in the phoneme pronunciation sequence to obtain a phoneme vector sequence; calculating the implicit feature vector corresponding to each phoneme vector in the phoneme vector sequence, and calculating the multidimensional Gaussian distribution of each phoneme based on the implicit feature vector; aligning each audio frame in the audio sequence with each phoneme in the multidimensional Gaussian distribution to obtain an alignment relationship, and determining the pronunciation duration of each phoneme based on the alignment relationship.

[0006] Optionally, in a first implementation method of the first aspect of the present invention, the phoneme pronunciation sequence is input into a preset duration prediction model, and each phoneme in the phoneme pronunciation sequence is vectorized to obtain a phoneme vector sequence, including: converting the phoneme pronunciation sequence into a feature dimension sequence of fixed dimension according to a preset window length and step length; inputting the feature dimension sequence into the phoneme vector layer of a preset duration prediction model, and extracting the feature dimension and sequence position number of each phoneme in the feature dimension sequence; encoding each phoneme in the feature dimension sequence according to the feature dimension and sequence position number to obtain multiple encoding values; vectorizing each of the encoding values ​​to obtain multiple phoneme vectors, and generating a phoneme vector sequence based on each of the phoneme vectors.

[0007] Optionally, in a second implementation method of the first aspect of the present invention, the calculation of the implicit feature vector corresponding to each phoneme vector in the phoneme vector sequence, and the calculation of the multi-dimensional Gaussian distribution of each phoneme based on the implicit feature vector includes: calculating the implicit feature vector corresponding to each phoneme vector in the phoneme vector sequence according to the multi-head attention mechanism preset in the duration prediction model, wherein the multi-head attention mechanism is an attention mechanism for selecting multiple specific input information; performing Gaussian processing on the implicit feature vector to obtain a Gaussian distribution; calculating the mean and variance of the Gaussian distribution, and generating a mean vector and a variance diagonal matrix based on the mean and variance; constructing a functional relationship between the mean vector and the variance diagonal matrix to generate a multi-dimensional Gaussian function; and calculating the multi-dimensional Gaussian distribution of each phoneme according to the preset bidirectional long short-term memory network and multi-dimensional Gaussian function.

[0008] Optionally, in a third implementation method of the first aspect of the present invention, the alignment processing of each audio frame in the audio sequence and each phoneme in the multidimensional Gaussian distribution to obtain an alignment relationship, and determining the pronunciation duration of each phoneme based on the alignment relationship includes: calculating the alignment probability of each audio frame in the audio sequence and each phoneme in the multidimensional Gaussian distribution; aligning each audio frame in the audio sequence and each phoneme in the multidimensional Gaussian distribution according to the alignment probability to obtain an alignment relationship; searching for the start position and end position of each phoneme in the audio sequence based on the alignment relationship; and determining the pronunciation duration of each phoneme based on the start position and end position corresponding to each phoneme.

[0009] Optionally, in a fourth implementation method of the first aspect of the present invention, the aligning each audio frame in the audio sequence with each phoneme in the multidimensional Gaussian distribution to obtain an alignment relationship, and determining the pronunciation duration of each phoneme based on the alignment relationship includes: calculating the amplitude spectrum of the audio sequence according to a preset short-time Fourier transform rule; inputting the amplitude spectrum into a preset Mel filter group, and converting the amplitude spectrum into a Mel spectrum frame sequence; extracting each Mel spectrum frame in the Mel spectrum frame sequence; aligning each Mel spectrum frame with each phoneme in the multidimensional Gaussian distribution to obtain an alignment relationship, and determining the pronunciation duration of each phoneme based on the alignment relationship.

[0010] Optionally, in the fifth implementation method of the first aspect of the present invention, before the phoneme pronunciation sequence is input into the preset duration prediction model and each phoneme in the phoneme pronunciation sequence is vectorized to obtain a phoneme vector sequence, it also includes: aligning a preset phoneme training sequence and a corresponding audio training sequence to obtain the pronunciation duration of each phoneme; vectorizing the phoneme training sequence to obtain a phoneme vector; inputting the phoneme vector into a preset machine learning model to obtain a predicted duration of each phoneme; determining a loss function based on the pronunciation duration and predicted duration of each phoneme; and using the loss function to adjust parameters of the machine learning model to obtain a duration prediction model.

[0011] The second aspect of the present invention proposes a pronunciation duration prediction device, which includes: an acquisition module, used to acquire medical text data and a corresponding audio sequence generated by an intelligent diagnosis and treatment system, and convert the medical text data into a phoneme pronunciation sequence; a processing module, used to input the phoneme pronunciation sequence into a preset duration prediction model, and vectorize each phoneme in the phoneme pronunciation sequence to obtain a phoneme vector sequence; a calculation module, used to calculate the implicit feature vector corresponding to each phoneme vector in the phoneme vector sequence, and calculate the multidimensional Gaussian distribution of each phoneme based on the implicit feature vector; a determination module, used to align each audio frame in the audio sequence with each phoneme in the multidimensional Gaussian distribution to obtain an alignment relationship, and determine the pronunciation duration of each phoneme based on the alignment relationship.

[0012] Optionally, in a first implementation method of the second aspect of the present invention, the processing module includes: a conversion unit, used to convert the phoneme pronunciation sequence into a feature dimension sequence of fixed dimension according to a preset window length and step length; an extraction unit, used to input the feature dimension sequence into the phoneme vector layer of a preset duration prediction model, and extract the feature dimension and sequence position number of each phoneme in the feature dimension sequence; an encoding unit, used to encode each phoneme in the feature dimension sequence according to the feature dimension and sequence position number to obtain multiple encoding values; a generation unit, used to vectorize each of the encoding values ​​to obtain multiple phoneme vectors, and generate a phoneme vector sequence based on each of the phoneme vectors.

[0013] Optionally, in a second implementation method of the second aspect of the present invention, the calculation module includes: a first calculation unit, used to calculate the implicit feature vector corresponding to each phoneme vector in the phoneme vector sequence according to the multi-head attention mechanism preset in the duration prediction model, wherein the multi-head attention mechanism is an attention mechanism that selects multiple specific input information; a Gaussian processing unit, used to perform Gaussian processing on the implicit feature vector to obtain a Gaussian distribution; a second calculation unit, used to calculate the mean and variance of the Gaussian distribution, and generate a mean vector and a variance diagonal matrix based on the mean and variance; a construction unit, used to construct a functional relationship between the mean vector and the variance diagonal matrix to generate a multidimensional Gaussian function; a third calculation unit, used to calculate the multidimensional Gaussian distribution of each of the phonemes according to a preset bidirectional long short-term memory network and a multidimensional Gaussian function.

[0014] Optionally, in a third implementation method of the second aspect of the present invention, the determination module is specifically used to: calculate the alignment probability of each audio frame in the audio sequence and each phoneme in the multidimensional Gaussian distribution; align each audio frame in the audio sequence and each phoneme in the multidimensional Gaussian distribution according to the alignment probability to obtain an alignment relationship; according to the alignment relationship, search for the starting position and ending position of each phoneme in the audio sequence; determine the pronunciation duration of each phoneme according to the starting position and ending position corresponding to each phoneme.

[0015] Optionally, in a fourth implementation manner of the second aspect of the present invention, the determination module is further specifically used to: calculate the amplitude spectrum of the audio sequence according to a preset short-time Fourier transform rule; input the amplitude spectrum into a preset Mel filter group, and convert the amplitude spectrum into a Mel spectrum frame sequence; extract each Mel spectrum frame in the Mel spectrum frame sequence; align each of the Mel spectrum frames with each phoneme in the multidimensional Gaussian distribution to obtain an alignment relationship, and determine the pronunciation duration of each of the phonemes based on the alignment relationship.

[0016] Optionally, in a fifth implementation of the second aspect of the present invention, the pronunciation duration prediction device also includes a training module, and the training module includes: a sequence alignment unit, used to align a preset phoneme training sequence and a corresponding audio training sequence to obtain the pronunciation duration of each phoneme; a vectorization unit, used to vectorize the phoneme training sequence to obtain a phoneme vector; a duration prediction unit, used to input the phoneme vector into a preset machine learning model to obtain the predicted duration of each phoneme; a function determination unit, used to determine a loss function based on the pronunciation duration and predicted duration of each phoneme; and an adjustment unit, used to use the loss function to adjust the parameters of the machine learning model to obtain a duration prediction model.

[0017] The third aspect of the present invention provides a pronunciation duration prediction device, which includes: a memory and at least one processor, wherein instructions are stored in the memory, and the memory and the at least one processor are interconnected through lines; the at least one processor calls the instructions in the memory so that the pronunciation duration prediction device executes the steps of the above-mentioned pronunciation duration prediction method based on the duration prediction model.

[0018] A fourth aspect of the present invention provides a computer-readable storage medium having instructions stored thereon, which, when executed on a computer, enables the computer to execute the steps of the above-mentioned pronunciation duration prediction method based on the duration prediction model.

[0019] In the technical solution provided by the present invention, the medical text data and the corresponding audio sequence generated by the intelligent diagnosis and treatment system are obtained, and the medical text data is converted into a phoneme pronunciation sequence; the phoneme pronunciation sequence is input into a preset duration prediction model, and each phoneme in the phoneme pronunciation sequence is vectorized to obtain a phoneme vector sequence; the implicit feature vector corresponding to each phoneme vector in the phoneme vector sequence and the multidimensional Gaussian distribution of each phoneme are calculated; each audio frame in the audio sequence and each phoneme in the multidimensional Gaussian distribution are aligned to determine the pronunciation duration of each phoneme. The technical solution of the present invention improves the accuracy of the prediction by combining the bidirectional long short-term memory neural network in the preset duration prediction model with context information, and the mean vector and variance diagonal matrix of the multidimensional Gaussian distribution to predict the pronunciation duration. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Schematic diagram of a first embodiment of a method for predicting pronunciation duration based on a duration prediction model in an embodiment of the present invention;

[0021] Figure 2 Schematic diagram of a second embodiment of a method for predicting pronunciation duration based on a duration prediction model in an embodiment of the present invention;

[0022] Figure 3 Schematic diagram of a third embodiment of a method for predicting pronunciation duration based on a duration prediction model in an embodiment of the present invention;

[0023] Figure 4 Schematic diagram of a fourth embodiment of a method for predicting pronunciation duration based on a duration prediction model in an embodiment of the present invention;

[0024] Figure 5 A schematic diagram of an embodiment of a pronunciation duration prediction device in an embodiment of the present invention;

[0025] Figure 6 It is a schematic diagram of another embodiment of the pronunciation duration prediction device in the embodiment of the present invention;

[0026] Figure 7 The figure is a schematic diagram of an embodiment of a pronunciation duration prediction device in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The embodiment of the present invention provides a method and related equipment for predicting pronunciation duration based on a duration prediction model, which obtains medical text data and corresponding audio sequences generated by an intelligent diagnosis and treatment system, and converts the medical text data into a phoneme pronunciation sequence; inputs the phoneme pronunciation sequence into a preset duration prediction model, vectorizes each phoneme in the phoneme pronunciation sequence, and obtains a phoneme vector sequence; calculates the implicit feature vector corresponding to each phoneme vector in the phoneme vector sequence, and the multidimensional Gaussian distribution of each phoneme; aligns each audio frame in the audio sequence and each phoneme in the multidimensional Gaussian distribution to determine the pronunciation duration of each phoneme. The technical solution of the present invention improves the accuracy of prediction by combining a bidirectional long short-term memory neural network in a preset duration prediction model with context information, and a mean vector and a variance diagonal matrix of a multidimensional Gaussian distribution to predict pronunciation duration.

[0028] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] For ease of understanding, the specific contents of the embodiments of the present invention are described below. Figure 1, the first embodiment of the pronunciation duration prediction method based on the duration prediction model in the embodiments of the present invention includes:

[0030] 101. Obtain the medical text data generated by the intelligent diagnosis and treatment system and the corresponding audio sequence, and convert the medical text data into a phoneme pronunciation sequence;

[0031] The server obtains the medical text data generated by the intelligent diagnosis and treatment system and the audio data corresponding to the medical text data, that is, the audio sequence; the server converts the text data into a phoneme pronunciation sequence according to a preset phoneme table. Among them, the medical text data refers to the data presented in text form and containing medical electronic record information such as personal health records, prescriptions, inspection reports, etc. The phoneme pronunciation sequence refers to a column of arranged phoneme elements. The medical text data can be text data of English words, and each English word in the text data corresponds to a phoneme pronunciation sequence; the medical text data can also be Chinese words, and each word in the text data corresponds to a phoneme pronunciation sequence; the audio sequence contains the voice features corresponding to the medical text data.

[0032] In the embodiments of the present invention, the user sends text data containing medical electronic record information such as personal health records, prescriptions, inspection reports, etc. to the server. After obtaining the text data, the server will convert each word in the text data into its corresponding phoneme pronunciation sequence by querying the phoneme table, which is convenient for subsequent processing of at least one phoneme pronunciation sequence corresponding to the text data.

[0033] A phoneme is the smallest speech unit divided according to the natural attributes of speech. Analyzing according to the pronunciation actions in a syllable, one action constitutes one phoneme. For example, a single word is decomposed into multiple syllables, and then each syllable is decomposed into corresponding multiple phonemes. Further, for example, for the word sequence "Ping An", it has two corresponding syllables, namely "ping" and "an"; the syllable "ping" can be further decomposed into the phonemes "p" and "ing", and the phoneme "an" can be further decomposed into the phonemes "a" and "n". In Chinese, one Chinese character corresponds to one syllable; in English, one word corresponds to one syllable, and the same is true for other languages.

[0034] 102. Input the phoneme pronunciation sequence into a preset duration prediction model, perform vectorization processing on each phoneme in the phoneme pronunciation sequence, and obtain a phoneme vector sequence;

[0035] In this embodiment, the preset duration prediction model is composed of a phoneme vector layer, a fast Fourier transform layer, and a bidirectional long short-term memory neural network, and is used to model and predict the pronunciation durations of each initial consonant and final consonant in the phoneme pronunciation sequence, that is, the pronunciation durations of each phoneme.

[0036] The server inputs the phoneme pronunciation sequence into the phoneme vector layer of the preset duration prediction model, vectorizes each phoneme in the phoneme pronunciation sequence, performs one-hot encoding (One-Hot encoding) on ​​each phoneme in the phoneme pronunciation sequence, obtains the encoding value corresponding to each phoneme, converts the encoding value into a binary vector, generates the phoneme vector corresponding to each phoneme, sorts each phoneme vector according to the sorting position of each phoneme in the corresponding phoneme pronunciation sequence, and obtains a phoneme vector sequence.

[0037] 103, calculating the implicit feature vector corresponding to each phoneme vector in the phoneme vector sequence, and calculating the multi-dimensional Gaussian distribution of each phoneme according to the implicit feature vector;

[0038] In the fast Fourier transform layer of the duration prediction model, the implicit feature vector corresponding to each phoneme vector in the phoneme vector sequence is calculated, and the bidirectional long short-term memory neural network calculates the multi-dimensional Gaussian distribution of each phoneme in the phoneme vector sequence according to the implicit feature vector and the context information of each phoneme in the phoneme vector sequence. In this embodiment, the fast Fourier transform layer is composed of a multi-head attention mechanism and two one-dimensional convolutional layers, in which residual connection, layer normalization and random inactivation (dropout) algorithm are used.

[0039] 104 , aligning each audio frame in the audio sequence with each phoneme in the multi-dimensional Gaussian distribution to obtain an alignment relationship, and determining the pronunciation duration of each phoneme according to the alignment relationship.

[0040] The audio frames in the audio sequence and the phonemes in the multidimensional Gaussian distribution are aligned using the forced alignment technology through the hidden Markov model algorithm to obtain the alignment relationship between each phoneme in the phoneme pronunciation sequence and each audio frame in the corresponding audio sequence. According to the alignment relationship, the start and end positions of each phoneme in the audio sequence are found to obtain the duration of each phoneme, that is, the pronunciation duration. The audio is spliced ​​according to the predicted pronunciation duration to synthesize medical speech, which can be used in intelligent diagnosis and treatment and remote consultation.

[0041] In this embodiment, a duration prediction model is obtained by performing deep learning training on a machine learning model, the pronunciation duration of a phoneme is predicted according to the duration prediction model, and the phoneme is expanded according to the prediction result so that the pronunciation duration of each phoneme is equal to the corresponding predicted pronunciation duration. The machine learning model includes a multi-layer bidirectional long short-term memory network (Long Short Term Memory, referred to as: LSTM) model, a multi-layer self-attention network (Self-Attention) model or a multi-layer convolutional neural network (Convolutional Neural Network, referred to as: CNN) model. In this embodiment, a bidirectional long short-term memory network model is selected for deep learning training. Specifically, a preset phoneme training sequence and an audio training sequence corresponding to the phoneme training sequence are aligned to obtain an alignment relationship between each phoneme in the phoneme training sequence and each audio frame in the corresponding audio training sequence; based on the alignment relationship, the start position and end position of each phoneme in the audio training sequence are found to obtain the pronunciation duration of each phoneme; the phoneme training sequence is vectorized to obtain a phoneme vector; the phoneme vector is input into a preset machine learning model for learning and training to obtain a predicted duration of each phoneme; a loss function is determined based on the pronunciation duration and predicted duration of each phoneme, that is, the loss function is dynamically adjusted; when the loss function converges, the Viterbi algorithm is used to extract the time alignment relationship between the phoneme and the corresponding audio to obtain a duration prediction model.

[0042] The embodiments of the present invention can acquire and process medical text data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0043] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0044] In an embodiment of the present invention, by acquiring the medical text data and the corresponding audio sequence generated by the intelligent diagnosis and treatment system, the medical text data is converted into a phoneme pronunciation sequence, and input into a preset duration prediction model, each phoneme in the phoneme pronunciation sequence is vectorized, and the implicit feature vector and multidimensional Gaussian distribution corresponding to each phoneme vector are calculated; each audio frame in the audio sequence and each phoneme in the multidimensional Gaussian distribution are aligned to determine the pronunciation duration of each phoneme. The present invention improves the accuracy of the prediction by combining the bidirectional long short-term memory neural network in the preset duration prediction model with contextual information to predict the pronunciation duration.

[0045] See also Figure 2 A second embodiment of the pronunciation duration prediction method based on the duration prediction model in the embodiment of the present invention includes:

[0046] 201, obtaining medical text data and corresponding audio sequences generated by the intelligent diagnosis and treatment system, and converting the medical text data into a phoneme pronunciation sequence;

[0047] The server obtains the medical text data generated by the intelligent diagnosis and treatment system and the audio data corresponding to the medical text data, that is, the audio sequence; the server converts the text data into a phoneme pronunciation sequence according to a preset phoneme table. Among them, medical text data refers to data presented in text form and containing medical electronic record information such as personal health records, prescriptions, and examination reports. The phoneme pronunciation sequence refers to phoneme elements arranged in a column. The medical text data can be the text data of English words, and each English word in the text data corresponds to a phoneme pronunciation sequence; the medical text data can also be Chinese words, and each word in the text data corresponds to a phoneme pronunciation sequence; the audio sequence contains the sound features corresponding to the medical text data.

[0048] In an embodiment of the present invention, a user sends text data containing medical electronic record information such as personal health records, prescriptions, and examination reports to a server. After obtaining the text data, the server converts each word in the text data into its corresponding phoneme pronunciation sequence by querying a phoneme table, so as to facilitate subsequent processing of at least one phoneme pronunciation sequence corresponding to the text data.

[0049] 202, converting the phoneme pronunciation sequence into a feature dimension sequence of fixed dimension according to a preset window length and step length;

[0050] The server converts the phoneme pronunciation sequence into a feature dimension sequence of fixed dimension according to the preset window length and the preset step length. The preset window length, the preset step length and the fixed dimension can be set according to the implementation requirements and / or system performance during the specific implementation. The present embodiment does not limit the size of the window length, the step length and the fixed dimension.

[0051] 203, inputting the feature dimension sequence into the phoneme vector layer of the preset duration prediction model, and extracting the feature dimension and sequence position number of each phoneme in the feature dimension sequence;

[0052] The feature dimension sequence is input into the phoneme vector layer of the preset duration prediction model. The phoneme vector layer extracts the feature dimension and sequence position number of each phoneme in the feature dimension sequence, that is, the feature dimension sequence is passed through the position encoder preset in the duration prediction model to calculate the sequence position information. The position encoder is used to introduce the sequence position information (PE) into the duration prediction model, as shown in formula (1) and formula (2):

[0053] PE(pos,2i)=sin(pos / 10000 2i / d ) (1)

[0054] PE(pos,2i+1)=cos(pos / 10000 2i / d ) (2)

[0055] Among them, pos is the position number, i is the feature dimension sequence, and d is the feature dimension.

[0056] 204, encoding each phoneme in the feature dimension sequence according to the feature dimension and the sequence position number to obtain multiple encoding values;

[0057] The server performs one-bit effective encoding on each phoneme in the feature dimension sequence according to the feature dimension and sequence position number corresponding to each phoneme, thereby obtaining multiple encoding values. In this embodiment, one-bit effective encoding (One-Hot encoding) mainly uses an N-bit state register to encode N states, each state has an independent register bit, and only one bit is valid at any time.

[0058] 205, performing vectorization processing on each code value to obtain a plurality of phoneme vectors, and generating a phoneme vector sequence according to each phoneme vector;

[0059] Each coding value is vectorized, that is, each coding value is converted into a binary vector to generate a phoneme vector corresponding to each phoneme, and each phoneme vector is sorted according to the sorting position of each phoneme in the corresponding phoneme pronunciation sequence to obtain a phoneme vector sequence.

[0060] 206, calculating the implicit feature vector corresponding to each phoneme vector in the phoneme vector sequence, and calculating the multi-dimensional Gaussian distribution of each phoneme according to the implicit feature vector;

[0061] In the fast Fourier transform layer of the duration prediction model, the implicit feature vector corresponding to each phoneme vector in the phoneme vector sequence is calculated, and the bidirectional long short-term memory neural network calculates the multi-dimensional Gaussian distribution of each phoneme in the phoneme vector sequence according to the implicit feature vector and the context information of each phoneme in the phoneme vector sequence. In this embodiment, the fast Fourier transform layer is composed of a multi-head attention mechanism and two layers of one-dimensional convolutional layers, in which residual connections, layer normalization and random inactivation (dropout) algorithms are used. Among them, the multi-head attention mechanism is an attention mechanism that selects multiple specific input information.

[0062] 207 , aligning each audio frame in the audio sequence with each phoneme in the multi-dimensional Gaussian distribution to obtain an alignment relationship, and determining the pronunciation duration of each phoneme according to the alignment relationship.

[0063] Each audio frame in the audio sequence and each phoneme in the multidimensional Gaussian distribution are aligned using the forced alignment technology through the hidden Markov model algorithm to obtain the alignment relationship between each phoneme in the phoneme pronunciation sequence and each audio frame in the corresponding audio sequence. According to the alignment relationship, the starting position and the ending position of each phoneme in the audio sequence are found to obtain the duration of each phoneme, that is, the pronunciation duration.

[0064] In an embodiment of the present invention, a phoneme pronunciation sequence is converted into a feature dimension sequence, which is input into the phoneme vector layer of the duration prediction model. Each phoneme is encoded and vectorized according to the feature dimension and sequence position number of each phoneme in the feature dimension sequence, and then the pronunciation duration is predicted based on the phoneme, thereby improving the accuracy and reliability of the pronunciation duration prediction.

[0065] See also Figure 3 A third embodiment of the pronunciation duration prediction method based on the duration prediction model in the embodiment of the present invention includes:

[0066] 301, obtaining medical text data and corresponding audio sequences generated by the intelligent diagnosis and treatment system, and converting the medical text data into a phoneme pronunciation sequence;

[0067] 302, inputting the phoneme pronunciation sequence into a preset duration prediction model, performing vectorization processing on each phoneme in the phoneme pronunciation sequence, and obtaining a phoneme vector sequence;

[0068] 303, calculating the implicit feature vector corresponding to each phoneme vector in the phoneme vector sequence according to the multi-head attention mechanism preset in the duration prediction model;

[0069] The latent feature vector corresponding to each phoneme vector in the phoneme vector sequence is calculated in the fast Fourier transform layer of the duration prediction model. In this embodiment, the fast Fourier transform layer is composed of a multi-head attention mechanism and two one-dimensional convolution layers, in which residual connection, layer normalization and random inactivation (dropout) algorithm are used. Specifically, the server performs fast Fourier transform processing on the phoneme vector according to the multi-head attention mechanism, extracts the information that needs attention, and obtains the latent feature vector corresponding to each phoneme vector.

[0070] 304, performing Gaussian processing on the latent feature vector to obtain a Gaussian distribution;

[0071] 305, calculating the mean and variance of the Gaussian distribution, and generating a mean vector and a variance diagonal matrix according to the mean and variance;

[0072] 306, constructing a functional relationship between the mean vector and the variance diagonal matrix to generate a multidimensional Gaussian function;

[0073] The server performs Gaussian processing on the latent feature vector to obtain a one-dimensional Gaussian distribution, calculates the mean vector and variance of the one-dimensional Gaussian distribution, and generates a diagonal matrix of the mean vector and variance of the one-dimensional Gaussian distribution according to the mean vector and variance of the one-dimensional Gaussian distribution.

[0074] The server constructs a functional relationship between the obtained mean vector and variance diagonal matrix to generate a multidimensional Gaussian function:

[0075] Z j =(μ j ,∑ j ).

[0076] 307, calculating the multidimensional Gaussian distribution of each phoneme according to the preset bidirectional long short-term memory network and the multidimensional Gaussian function;

[0077] The server uses a bidirectional long short-term memory neural network to calculate the multidimensional Gaussian distribution of each phoneme in the phoneme vector sequence through a multidimensional Gaussian function based on the implicit feature vector and the context information of each phoneme in the phoneme vector sequence. Among them, the multidimensional Gaussian distribution of each phoneme is:

[0078] Z=(Z 1 , Z 2 , ..., Z m ).

[0079] 308 , aligning each audio frame in the audio sequence with each phoneme in the multi-dimensional Gaussian distribution to obtain an alignment relationship, and determining the pronunciation duration of each phoneme according to the alignment relationship.

[0080] The probability of aligning each audio frame in the Mel spectrum frame sequence with each phoneme in the multidimensional Gaussian distribution is calculated according to the multidimensional Gaussian function. For example, the probability (P(y i |z j )):

[0081] P(y i |z j )=N(y i |μ j ,∑ j )

[0082] The server uses the forced alignment technology to align each audio frame in the Mel spectrum frame sequence and each phoneme in the multidimensional Gaussian distribution according to the corresponding alignment probability through the hidden Markov model algorithm, and uses the Viterbi algorithm to extract the alignment relationship between each audio frame and each phoneme. According to the alignment relationship, the start and end positions of each phoneme in the audio sequence are found to obtain the duration of each phoneme, that is, the pronunciation duration.

[0083] In this embodiment, the Viterbi algorithm is an optimal selection problem with multiple steps and multiple choices in each step. All the choices in each step save the minimum total cost (or maximum value) of all previous steps to the current choice of the current step and the choice of the previous step under the current cost. After calculating all steps in sequence, the optimal selection path is found by backtracking.

[0084] In the embodiment of the present invention, steps 301-302 are consistent with steps 101-102 in the first embodiment of the pronunciation duration prediction method based on the duration prediction model, and are not described in detail here.

[0085] In an embodiment of the present invention, the implicit feature vector of the phoneme vector sequence is calculated, and the mean vector and variance diagonal matrix of the implicit feature vector and the multidimensional Gaussian distribution are calculated, and then the audio sequence and the multidimensional Gaussian distribution are aligned to predict the pronunciation duration of the phoneme. The embodiment of the present invention combines the mean and variance to construct the mean vector and the variance diagonal matrix to predict the pronunciation duration of the phoneme, thereby improving the flexibility and accuracy of the pronunciation duration prediction.

[0086] See also Figure 4 A fourth embodiment of the pronunciation duration prediction method based on the duration prediction model in the embodiment of the present invention includes:

[0087] 401, obtaining medical text data and corresponding audio sequences generated by the intelligent diagnosis and treatment system, and converting the medical text data into a phoneme pronunciation sequence;

[0088] 402, inputting the phoneme pronunciation sequence into a preset duration prediction model, performing vectorization processing on each phoneme in the phoneme pronunciation sequence, and obtaining a phoneme vector sequence;

[0089] 403, calculating the implicit feature vector corresponding to each phoneme vector in the phoneme vector sequence, and calculating the multi-dimensional Gaussian distribution of each phoneme according to the implicit feature vector;

[0090] In the fast Fourier transform layer of the duration prediction model, the implicit feature vector corresponding to each phoneme vector in the phoneme vector sequence is calculated, and the bidirectional long short-term memory neural network calculates the multi-dimensional Gaussian distribution of each phoneme in the phoneme vector sequence according to the implicit feature vector and the context information of each phoneme in the phoneme vector sequence. In this embodiment, the fast Fourier transform layer is composed of a multi-head attention mechanism and two layers of one-dimensional convolutional layers, in which residual connections, layer normalization and random inactivation (dropout) algorithms are used. Among them, the multi-dimensional Gaussian distribution of each phoneme is:

[0091] Z=(Z 1 , Z 2 , ..., Z m ).

[0092] 404, calculating the amplitude spectrum of the audio sequence according to a preset short-time Fourier transform rule;

[0093] 405, inputting the amplitude spectrum into a preset Mel filter bank to convert the amplitude spectrum into a Mel spectrum frame sequence;

[0094] The server uses a preset short-time Fourier transform (STFT) rule to calculate the amplitude spectrum of the audio sequence, that is, the audio sequence is converted into a corresponding amplitude spectrum through the short-time Fourier transform rule; the amplitude spectrum is input into a preset Mel filter group for filtering, and the amplitude spectrum is converted through an 80-dimensional Mel filter group with a 50ms frame length, a 12.5ms frame shift and a Hanning window function in the Mel filter group to calculate a Mel spectrum frame sequence, wherein the Mel spectrum frame sequence includes multiple Mel spectrum frames.

[0095] 406, extracting each mel spectrum frame in the mel spectrum frame sequence;

[0096] 407 , aligning each mel spectrum frame with each phoneme in the multidimensional Gaussian distribution to obtain an alignment relationship, and determining the pronunciation duration of each phoneme according to the alignment relationship.

[0097] The server extracts each mel-spectrogram frame in the mel-spectrogram frame sequence, and aligns each mel-spectrogram frame in the mel-spectrogram frame sequence and each phoneme in the multidimensional Gaussian distribution using a forced alignment technique through a hidden Markov model algorithm, and uses a Viterbi algorithm to extract the alignment relationship between each mel-spectrogram frame and each phoneme. Based on the alignment relationship, the starting position and the ending position of each phoneme in the mel-spectrogram frame sequence are found to obtain the duration of each phoneme, that is, the pronunciation duration.

[0098] In the embodiment of the present invention, steps 401-402 are consistent with steps 101-102 in the first embodiment of the pronunciation duration prediction method based on the duration prediction model, and are not described in detail here.

[0099] In an embodiment of the present invention, the amplitude spectrum of the audio sequence is calculated and converted into a Mel-spectrogram frame sequence, each Mel-spectrogram frame in the Mel-spectrogram sequence is aligned with each phoneme in the multidimensional Gaussian distribution, and the pronunciation duration of each phoneme is determined, thereby improving the accuracy of the prediction of the pronunciation duration of each phoneme.

[0100] The above describes the pronunciation duration prediction method based on the duration prediction model in the embodiment of the present invention. The following describes the pronunciation duration prediction device in the embodiment of the present invention. Please refer to Figure 5 , an embodiment of the pronunciation duration prediction device in the embodiment of the present invention includes:

[0101] An acquisition module 501 is used to acquire medical text data and corresponding audio sequences generated by the intelligent diagnosis and treatment system, and convert the medical text data into a phoneme pronunciation sequence;

[0102] A processing module 502 is used to input the phoneme pronunciation sequence into a preset duration prediction model, and perform vectorization processing on each phoneme in the phoneme pronunciation sequence to obtain a phoneme vector sequence;

[0103] A calculation module 503, used to calculate the implicit feature vector corresponding to each phoneme vector in the phoneme vector sequence, and calculate the multi-dimensional Gaussian distribution of each phoneme according to the implicit feature vector;

[0104] The determination module 504 is used to align each audio frame in the audio sequence with each phoneme in the multi-dimensional Gaussian distribution to obtain an alignment relationship, and determine the pronunciation duration of each phoneme according to the alignment relationship.

[0105] In an embodiment of the present invention, the medical text data and the corresponding audio sequence generated by the intelligent diagnosis and treatment system are obtained through a pronunciation duration prediction device, the medical text data is converted into a phoneme pronunciation sequence, and input into a preset duration prediction model, each phoneme in the phoneme pronunciation sequence is vectorized, and the implicit feature vector and multidimensional Gaussian distribution corresponding to each phoneme vector are calculated; each audio frame in the audio sequence and each phoneme in the multidimensional Gaussian distribution are aligned to determine the pronunciation duration of each phoneme. The present invention improves the accuracy of the prediction by combining the bidirectional long short-term memory neural network in the preset duration prediction model with contextual information to predict the pronunciation duration.

[0106] See also Figure 6 Another embodiment of the pronunciation duration prediction device in the embodiment of the present invention includes:

[0107] An acquisition module 501 is used to acquire medical text data and corresponding audio sequences generated by the intelligent diagnosis and treatment system, and convert the medical text data into a phoneme pronunciation sequence;

[0108] A processing module 502 is used to input the phoneme pronunciation sequence into a preset duration prediction model, and perform vectorization processing on each phoneme in the phoneme pronunciation sequence to obtain a phoneme vector sequence;

[0109] A calculation module 503, used to calculate the implicit feature vector corresponding to each phoneme vector in the phoneme vector sequence, and calculate the multi-dimensional Gaussian distribution of each phoneme according to the implicit feature vector;

[0110] The determination module 504 is used to align each audio frame in the audio sequence with each phoneme in the multi-dimensional Gaussian distribution to obtain an alignment relationship, and determine the pronunciation duration of each phoneme according to the alignment relationship.

[0111] Wherein, the processing module 502 includes:

[0112] A conversion unit 5021, used for converting the phoneme pronunciation sequence into a feature dimension sequence of fixed dimension according to a preset window length and step length;

[0113] An extraction unit 5022 is used to input the feature dimension sequence into the phoneme vector layer of the preset duration prediction model, and extract the feature dimension and sequence position number of each phoneme in the feature dimension sequence;

[0114] The encoding unit 5023 is used to encode each phoneme in the feature dimension sequence according to the feature dimension and the sequence position number to obtain multiple encoding values;

[0115] The generating unit 5024 is used to perform vectorization processing on each of the encoding values ​​to obtain multiple phoneme vectors, and generate a phoneme vector sequence according to each of the phoneme vectors.

[0116] Wherein, the calculation module 503 includes:

[0117] A first calculation unit 5031 is used to calculate the implicit feature vector corresponding to each phoneme vector in the phoneme vector sequence according to the multi-head attention mechanism preset in the duration prediction model;

[0118] A Gaussian processing unit 5032, configured to perform Gaussian processing on the latent feature vector to obtain a Gaussian distribution;

[0119] A second calculation unit 5033 is used to calculate the mean and variance of the Gaussian distribution, and generate a mean vector and a variance diagonal matrix according to the mean and variance;

[0120] A construction unit 5034 is used to construct a functional relationship between the mean vector and the variance diagonal matrix to generate a multidimensional Gaussian function;

[0121] The third calculation unit 5035 is used to calculate the multidimensional Gaussian distribution of each of the phonemes according to a preset bidirectional long short-term memory network and a multidimensional Gaussian function.

[0122] The determining module 504 is specifically used for:

[0123] Calculating the alignment probability of each audio frame in the audio sequence and each phoneme in the multidimensional Gaussian distribution;

[0124] Aligning each audio frame in the audio sequence with each phoneme in the multidimensional Gaussian distribution according to the alignment probability to obtain an alignment relationship;

[0125] According to the alignment relationship, searching for the start position and the end position of each of the phonemes in the audio sequence;

[0126] The pronunciation duration of each phoneme is determined according to the start position and the end position corresponding to each phoneme.

[0127] The determining module 504 is further specifically configured to:

[0128] Calculating the amplitude spectrum of the audio sequence according to a preset short-time Fourier transform rule;

[0129] Inputting the magnitude spectrum into a preset Mel filter bank to convert the magnitude spectrum into a Mel spectrum frame sequence;

[0130] Extracting each mel-spectrogram frame in the mel-spectrogram frame sequence;

[0131] Each of the mel-spectrogram frames is aligned with each phoneme in the multidimensional Gaussian distribution to obtain an alignment relationship, and the pronunciation duration of each of the phonemes is determined according to the alignment relationship.

[0132] The pronunciation duration prediction device further includes a training module 505, and the training module 505 includes:

[0133] The sequence alignment unit 5051 is used to align the preset phoneme training sequence and the corresponding audio training sequence to obtain the pronunciation duration of each phoneme;

[0134] A vectorization unit 5052, used for vectorizing the phoneme training sequence to obtain a phoneme vector;

[0135] The duration prediction unit 5053 is used to input the phoneme vector into a preset machine learning model to obtain the predicted duration of each phoneme;

[0136] A function determination unit 5054, configured to determine a loss function according to the pronunciation duration and the prediction duration of each of the phonemes;

[0137] The adjustment unit 5055 is used to use the loss function to adjust the parameters of the machine learning model to obtain a duration prediction model.

[0138] In an embodiment of the present invention, the implicit feature vector of the phoneme vector sequence is calculated by a pronunciation duration prediction device, and the mean vector and variance diagonal matrix of the implicit feature vector, as well as the multidimensional Gaussian distribution are calculated, and then the audio sequence and the multidimensional Gaussian distribution are aligned to predict the pronunciation duration of the phoneme. The embodiment of the present invention combines the mean and variance to construct a mean vector and a variance diagonal matrix to predict the pronunciation duration of the phoneme, thereby improving the flexibility and accuracy of the pronunciation duration prediction.

[0139] See also Figure 7 , the following describes in detail an embodiment of the pronunciation duration prediction device in the embodiment of the present invention from the perspective of hardware processing.

[0140] Figure 71 is a schematic diagram of the structure of a pronunciation duration prediction device provided by an embodiment of the present invention. The pronunciation duration prediction device 700 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 710 (for example, one or more processors) and a memory 720, and one or more storage media 730 (for example, one or more mass storage devices) storing application programs 733 or data 732. Among them, the memory 720 and the storage medium 730 can be short-term storage or permanent storage. The program stored in the storage medium 730 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the pronunciation duration prediction device 700. Furthermore, the processor 710 may be configured to communicate with the storage medium 730 to execute a series of instruction operations in the storage medium 730 on the pronunciation duration prediction device 700.

[0141] The pronunciation duration prediction device 700 may also include one or more power supplies 740, one or more wired or wireless network interfaces 750, one or more input and output interfaces 760, and or, one or more operating systems 731, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that Figure 7 The structure of the pronunciation duration prediction device shown does not constitute a limitation on the pronunciation duration prediction device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0142] The server referred to in the present invention can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), as well as big data and artificial intelligence platforms.

[0143] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0144] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the pronunciation duration prediction method based on the duration prediction model.

[0145] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0146] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0147] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for predicting pronunciation duration based on a duration prediction model, characterized in that: The pronunciation duration prediction method based on the duration prediction model comprises: Acquire medical text data and a corresponding audio sequence generated by the intelligent diagnosis and treatment system, and convert the medical text data into a phoneme pronunciation sequence, wherein the phoneme pronunciation sequence includes a plurality of phonemes; Inputting the phoneme pronunciation sequence into a preset duration prediction model, vectorizing each phoneme in the phoneme pronunciation sequence through a phoneme vector layer to obtain a phoneme vector sequence, wherein the duration prediction model is used to predict the pronunciation duration of each phoneme in the phoneme pronunciation sequence; Calculate the implicit feature vector corresponding to each phoneme vector in the phoneme vector sequence through a fast Fourier transform layer, and calculate the multidimensional Gaussian distribution of each phoneme according to the implicit feature vector through a bidirectional long short-term memory neural network; Each audio frame in the audio sequence and each phoneme in the multi-dimensional Gaussian distribution are aligned to obtain an alignment relationship, and the pronunciation duration of each phoneme is determined according to the alignment relationship.

2. The method for predicting pronunciation duration based on the duration prediction model according to claim 1, characterized in that: The step of inputting the phoneme pronunciation sequence into a preset duration prediction model, performing vectorization processing on each phoneme in the phoneme pronunciation sequence, and obtaining a phoneme vector sequence includes: Converting the phoneme pronunciation sequence into a feature dimension sequence of fixed dimension according to a preset window length and step length; Inputting the feature dimension sequence into the phoneme vector layer of the preset duration prediction model, and extracting the feature dimension and sequence position number of each phoneme in the feature dimension sequence; According to the feature dimension and the sequence position number, encoding each phoneme in the feature dimension sequence to obtain multiple encoding values; Each of the encoding values ​​is vectorized to obtain a plurality of phoneme vectors, and a phoneme vector sequence is generated according to each of the phoneme vectors.

3. The method for predicting pronunciation duration based on the duration prediction model according to claim 2, characterized in that: The step of calculating the implicit feature vector corresponding to each phoneme vector in the phoneme vector sequence, and calculating the multi-dimensional Gaussian distribution of each phoneme according to the implicit feature vector comprises: According to the multi-head attention mechanism preset in the duration prediction model, calculating the implicit feature vector corresponding to each phoneme vector in the phoneme vector sequence, wherein the multi-head attention mechanism is an attention mechanism that selects multiple specific input information; Performing Gaussian processing on the latent feature vector to obtain a Gaussian distribution; Calculating the mean and variance of the Gaussian distribution, and generating a mean vector and a variance diagonal matrix based on the mean and variance; Constructing a functional relationship between the mean vector and the variance diagonal matrix to generate a multidimensional Gaussian function; According to the preset bidirectional long short-term memory network and the multidimensional Gaussian function, the multidimensional Gaussian distribution of each of the phonemes is calculated.

4. The method for predicting pronunciation duration based on the duration prediction model according to claim 3, characterized in that: The step of aligning each audio frame in the audio sequence with each phoneme in the multidimensional Gaussian distribution to obtain an alignment relationship, and determining the pronunciation duration of each phoneme according to the alignment relationship comprises: Calculating the alignment probability of each audio frame in the audio sequence and each phoneme in the multidimensional Gaussian distribution; Aligning each audio frame in the audio sequence with each phoneme in the multidimensional Gaussian distribution according to the alignment probability to obtain an alignment relationship; According to the alignment relationship, searching for the start position and the end position of each of the phonemes in the audio sequence; The pronunciation duration of each phoneme is determined according to the start position and the end position corresponding to each phoneme.

5. The method for predicting pronunciation duration based on the duration prediction model according to claim 3, characterized in that: The step of aligning each audio frame in the audio sequence with each phoneme in the multidimensional Gaussian distribution to obtain an alignment relationship, and determining the pronunciation duration of each phoneme according to the alignment relationship comprises: Calculating the amplitude spectrum of the audio sequence according to a preset short-time Fourier transform rule; Inputting the magnitude spectrum into a preset Mel filter bank to convert the magnitude spectrum into a Mel spectrum frame sequence; Extracting each mel-spectrogram frame in the mel-spectrogram frame sequence; Each of the mel-spectrogram frames is aligned with each phoneme in the multidimensional Gaussian distribution to obtain an alignment relationship, and the pronunciation duration of each of the phonemes is determined according to the alignment relationship.

6. The method for predicting pronunciation duration based on a duration prediction model according to any one of claims 1 to 5, characterized in that: Before inputting the phoneme pronunciation sequence into a preset duration prediction model and performing vectorization processing on each phoneme in the phoneme pronunciation sequence to obtain a phoneme vector sequence, the method further includes: Aligning the preset phoneme training sequence with the corresponding audio training sequence to obtain the pronunciation duration of each phoneme; Vectorizing the phoneme training sequence to obtain a phoneme vector; Inputting the phoneme vector into a preset machine learning model to obtain the predicted duration of each phoneme; Determine a loss function according to the pronunciation duration and prediction duration of each of the phonemes; The loss function is used to adjust the parameters of the machine learning model to obtain a duration prediction model.

7. A pronunciation duration prediction device, characterized in that: The pronunciation duration prediction device comprises: An acquisition module, used to acquire medical text data and corresponding audio sequences generated by the intelligent diagnosis and treatment system, and convert the medical text data into a phoneme pronunciation sequence; A processing module, used for inputting the phoneme pronunciation sequence into a preset duration prediction model, vectorizing each phoneme in the phoneme pronunciation sequence through a phoneme vector layer to obtain a phoneme vector sequence, wherein the duration prediction model is used for predicting the pronunciation duration of each phoneme in the phoneme pronunciation sequence; A calculation module, used for calculating the implicit feature vector corresponding to each phoneme vector in the phoneme vector sequence through a fast Fourier transform layer, and calculating the multidimensional Gaussian distribution of each phoneme according to the implicit feature vector through a bidirectional long short-term memory neural network; The determination module is used to align each audio frame in the audio sequence with each phoneme in the multidimensional Gaussian distribution to obtain an alignment relationship, and determine the pronunciation duration of each phoneme based on the alignment relationship.

8. The pronunciation duration prediction device according to claim 7, characterized in that: The processing module comprises: A conversion unit, used for converting the phoneme pronunciation sequence into a feature dimension sequence of fixed dimension according to a preset window length and step length; An extraction unit, used for inputting the feature dimension sequence into the phoneme vector layer of a preset duration prediction model, and extracting the feature dimension and sequence position number of each phoneme in the feature dimension sequence; An encoding unit, used for encoding each phoneme in the feature dimension sequence according to the feature dimension and the sequence position number to obtain multiple encoding values; A generating unit is used to perform vectorization processing on each of the encoding values ​​to obtain a plurality of phoneme vectors, and to generate a phoneme vector sequence according to each of the phoneme vectors.

9. A pronunciation duration prediction device, characterized in that: The pronunciation duration prediction device comprises: A memory and at least one processor, wherein instructions are stored in the memory, and the memory and the at least one processor are interconnected via a line; The at least one processor calls the instructions in the memory to enable the pronunciation duration prediction device to perform the steps of the pronunciation duration prediction method based on the duration prediction model as described in any one of claims 1 to 6.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the steps of the pronunciation duration prediction method based on the duration prediction model as described in any one of claims 1 to 6 are implemented.

Images