A speech generation method and system based on laryngeal vibration signal analysis

Abstract

The present application relates to the field of speech synthesis technology, and in particular to a speech generation method and system based on laryngeal vibration signal analysis, the method comprising: collecting laryngeal vibration signals through a laryngeal vibration sound generator; comparing vibration, pressure, and sound multi-dimensional data with laryngeal vibration data in a standard database, and calibrating the vibration, pressure, and sound multi-dimensional data based on the comparison results to obtain a calibrated laryngeal vibration signal; extracting an audio signal from the laryngeal vibration signal, extracting audio signal features through the ICEEMDAN algorithm, and identifying the emotional state of the user's current voice through a classification model based on the audio signal features; converting the calibrated laryngeal vibration signal into a digital signal, and converting the digital signal into text information; synthesizing and outputting speech based on the converted text information through text-to-speech technology, combined with the emotional state and selected timbre. The present application can solve the problems of insufficient voice cloning accuracy and single emotional expression.

Description

Technical Field

[0001] The present application relates to the field of speech synthesis technology, and in particular to a speech generation method and system based on laryngeal vibration signal analysis. Background Art

[0002] Current speech generation technology has evolved from rule-based systems (such as early rule-driven format synthesizers) to end-to-end models driven by deep learning. Traditional speech synthesis relies on predefined voice libraries and acoustic models, making it difficult to achieve personalized voice cloning and emotional adaptation.

[0003] In recent years, deep learning-based end-to-end modeling technologies (such as Tacotron 2 and the WaveNet architecture) have significantly improved the naturalness and fluency of speech by directly learning the text-to-speech mapping relationship. However, existing technologies still face problems such as insufficient accuracy in personalized voice cloning and limited emotional expression. Summary of the Invention

[0004] Based on this, it is necessary to provide a speech generation method and system based on laryngeal vibration signal analysis that can solve the problems of insufficient voice cloning accuracy and single emotional expression in response to the above technical problems.

[0005] In a first aspect, the present application provides a method for speech generation based on laryngeal vibration signal analysis, the method comprising:

[0006] The throat vibration signal is collected by a throat vibration sound generator, and the throat vibration signal carries multi-dimensional data of vibration, pressure and sound;

[0007] Comparing the vibration, pressure, and multi-dimensional sound data with throat vibration data in a standard database, and calibrating the vibration, pressure, and multi-dimensional sound data based on the comparison results to obtain a calibrated throat vibration signal;

[0008] Extracting an audio signal from the throat vibration signal, extracting audio signal features using the ICEEMDAN algorithm, and identifying the emotional state of the user's current voice using a classification model based on the audio signal features;

[0009] Converting the calibrated laryngeal vibration signal into a digital signal, and converting the digital signal into text information;

[0010] Based on the converted text information, the speech is synthesized and output through text-to-speech technology, combined with the emotional state and the selected timbre.

[0011] In one embodiment, before comparing the vibration, pressure, and sound multi-dimensional data with throat vibration data in a standard database, the method further includes:

[0012] performing filtering processing on the vibration, pressure, and sound multi-dimensional data;

[0013] The vibration, pressure and sound multi-dimensional data after filtering are denoised using a noise reduction algorithm.

[0014] In one embodiment, after calibrating the vibration, pressure, and sound multi-dimensional data based on the comparison results to obtain a calibrated throat vibration signal, the method further includes:

[0015] converting the calibrated throat vibration signal into an electrical signal, and compressing and encrypting the electrical signal;

[0016] Based on the processed electrical signals, a private database is established to store the user's personalized throat vibration data.

[0017] In one embodiment, extracting an audio signal from the throat vibration signal, extracting audio signal features using the ICEEMDAN algorithm, and identifying the emotional state of the user's current voice using a classification model based on the audio signal features includes:

[0018] Extracting an audio signal from the throat vibration signal, and decomposing the audio signal into multiple intrinsic modes using an ICEEMDAN algorithm;

[0019] Extracting multiple features of each intrinsic mode, wherein the multiple features of the intrinsic mode include statistical parameters, center frequency, energy, peak amplitude, and permutation entropy;

[0020] Based on the multiple features of each intrinsic pattern, classification is performed using a support vector machine to identify the emotional state of the user's current voice.

[0021] In one embodiment, synthesizing and outputting speech based on the converted text information by using text-to-speech technology, combined with the emotional state and the selected timbre, includes:

[0022] Mapping the audio signal to a low-dimensional latent space using text-to-speech technology to obtain a latent representation of the reference speech;

[0023] Mapping the converted text information to a potential representation consistent with the reference speech to obtain a potential representation of the text;

[0024] estimating the duration of speech using a trained duration predictor based on the latent representation of the reference speech and the latent representation of the text;

[0025] Combined with the emotional state and the selected timbre, the latent representation of the text and the duration of the speech are input into a trained speech autoencoder to synthesize and output speech.

[0026] In one embodiment, converting the calibrated throat vibration signal into a digital signal, and converting the digital signal into text information includes:

[0027] Converting the calibrated laryngeal vibration signal into a digital signal;

[0028] Decoding the digital signal into original speech features, and mapping the original speech features into phonemes through an acoustic model to obtain a phoneme sequence;

[0029] According to the phoneme sequence, a vocabulary sequence is predicted through a statistical language model, and text information is generated according to the vocabulary sequence.

[0030] In one embodiment, converting the calibrated throat vibration signal into a digital signal comprises:

[0031] The calibrated laryngeal vibration signal is sampled by a sampler at a preset frequency to collect amplitude values ​​of discrete time points;

[0032] Through the quantization method, the continuous amplitude value after sampling is approximated to a finite number of discrete values;

[0033] Convert the quantized discrete value into a binary code stream to form a digital signal.

[0034] In a second aspect, the present application further provides a speech generation device based on laryngeal vibration signal analysis. The device comprises:

[0035] A throat vibration signal acquisition module is used to collect throat vibration signals through a throat vibration sounder. The throat vibration signals carry multi-dimensional data of vibration, pressure and sound;

[0036] a data calibration module, configured to compare the vibration, pressure, and multi-dimensional sound data with throat vibration data in a standard database, and calibrate the vibration, pressure, and multi-dimensional sound data based on the comparison results to obtain a calibrated throat vibration signal;

[0037] An emotional state recognition module is used to extract an audio signal from the throat vibration signal, extract audio signal features using the ICEEMDAN algorithm, and identify the emotional state of the user's current voice using a classification model based on the audio signal features;

[0038] a throat vibration signal conversion module, configured to convert the calibrated throat vibration signal into a digital signal, and then convert the digital signal into text information;

[0039] The speech synthesis module is used to synthesize and output speech based on the converted text information through text-to-speech technology, combined with the emotional state and the selected timbre.

[0040] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are performed:

[0041] The throat vibration signal is collected by a throat vibration sound generator, and the throat vibration signal carries multi-dimensional data of vibration, pressure and sound;

[0042] Comparing the vibration, pressure, and multi-dimensional sound data with throat vibration data in a standard database, and calibrating the vibration, pressure, and multi-dimensional sound data based on the comparison results to obtain a calibrated throat vibration signal;

[0043] Extracting an audio signal from the throat vibration signal, extracting audio signal features using the ICEEMDAN algorithm, and identifying the emotional state of the user's current voice using a classification model based on the audio signal features;

[0044] Converting the calibrated laryngeal vibration signal into a digital signal, and converting the digital signal into text information;

[0045] Based on the converted text information, the speech is synthesized and output through text-to-speech technology, combined with the emotional state and the selected timbre.

[0046] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0047] The throat vibration signal is collected by a throat vibration sound generator, and the throat vibration signal carries multi-dimensional data of vibration, pressure and sound;

[0048] Comparing the vibration, pressure, and multi-dimensional sound data with throat vibration data in a standard database, and calibrating the vibration, pressure, and multi-dimensional sound data based on the comparison results to obtain a calibrated throat vibration signal;

[0049] Extracting an audio signal from the throat vibration signal, extracting audio signal features using the ICEEMDAN algorithm, and identifying the emotional state of the user's current voice using a classification model based on the audio signal features;

[0050] Converting the calibrated laryngeal vibration signal into a digital signal, and converting the digital signal into text information;

[0051] Based on the converted text information, the speech is synthesized and output through text-to-speech technology, combined with the emotional state and the selected timbre.

[0052] In summary, this application has the following beneficial technical effects:

[0053] By collecting multi-dimensional data on vibration, pressure and sound from throat vibration signals and performing multi-dimensional calibration in combination with a standard database, the accuracy of voice cloning has been significantly improved, making the synthesized voice extremely similar to the user's real voice in terms of timbre, intonation, pronunciation habits, etc.; through the ICEEMDAN algorithm, the characteristics of throat vibration signals are extracted, and the emotional state of the user's current voice is identified in real time through a classification model, achieving fast-response speech synthesis and high-accuracy emotion recognition; combining the emotional state and the selected timbre to synthesize and output speech, enhancing the expressiveness and appeal of speech synthesis and achieving emotionally rich speech expression. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 1 is a flow chart of a method for generating speech based on laryngeal vibration signal analysis in one embodiment;

[0055] Figure 2 is a flow chart of a speech generation method based on laryngeal vibration signal analysis in another embodiment;

[0056] Figure 3 is a structural block diagram of a speech generation device based on laryngeal vibration signal analysis in one embodiment;

[0057] Figure 4 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0058] The embodiment of the present invention provides a speech generation method and system based on laryngeal vibration signal analysis.

[0059] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0060] In the description of the embodiments disclosed herein, the term "including" and similar terms should be understood as open inclusion, i.e., "including but not limited to." The term "based on" should be understood as "based, at least in part, on." The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment." The terms "first," "second," etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0061] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 In one embodiment of the present invention, a method for generating speech based on laryngeal vibration signal analysis includes:

[0062] S100: collecting a laryngeal vibration signal through a laryngeal vibration sound generator.

[0063] Specifically, a laryngeal vibrator is used to detect vibration, pressure, sound and other signals in the user's throat, which serves as the starting component for raw data collection. A laryngeal vibrator is a device that is usually attached to the user's throat through a sensor (such as an accelerometer, microphone or pressure sensor) to collect multi-dimensional data in real time. The multi-dimensional data types collected by the laryngeal vibrator include vibration, pressure and sound. Vibration data includes detecting mechanical vibrations (e.g., frequency, amplitude and position) generated by the vibration of the throat muscles and vocal cords; pressure data includes measuring changes in air pressure when the vocal cords are closed (e.g., subglottal pressure); and sound data includes capturing sound waveforms (e.g., pitch, volume and spectral characteristics) through a microphone. These data are digitized and stored as time series signals for subsequent analysis. During collection, the device must ensure that the sensor is accurately positioned to reduce noise interference.

[0064] S200 , comparing the vibration, pressure, and sound multi-dimensional data with the throat vibration data in the standard database, and calibrating the vibration, pressure, and sound multi-dimensional data based on the comparison results to obtain a calibrated throat vibration signal.

[0065] Specifically, the multi-dimensional vibration, pressure, and sound data are compared with the laryngeal vibration data in the standard database. A similarity calculation method is used to calculate the similarity between the collected data and the laryngeal vibration data in the standard database. For example, if the deviation between the collected vibration amplitude and the database standard value exceeds a threshold, it is considered a mismatch. The pressure data may be compared with the standard vocal cord closure pattern to detect anomalies (such as low pressure indicating weakened vocal cord function). The comparison results will reveal data deviations (for example, insufficient vibration intensity or abnormal pressure), and the system will calibrate and correct them accordingly. For example, if the vibration amplitude is lower than the standard value (for example, the database standard is 50 dB, and the collected data is 40 dB), the system increases the signal gain to compensate for sensor sensitivity issues; if the pressure data does not match the standard pressure curve (such as high subglottal pressure), linear regression or filtering techniques are used to adjust the data offset; if the sound and vibration data are out of sync, the system aligns the time axis through correlation analysis.

[0066] S300: extracting an audio signal from the throat vibration signal, extracting audio signal features through the ICEEMDAN algorithm, and identifying the emotional state of the user's current voice through a classification model based on the audio signal features.

[0067] Among them, the CEEMDAN algorithm is an advanced signal decomposition technology used to process nonlinear and non-stationary signals. By introducing adaptive noise and multiple iterations, it effectively solves the modal aliasing problem in traditional empirical mode decomposition (EMD) and improves the accuracy and stability of decomposition.

[0068] Specifically, to accurately identify a user's emotional state, effective features must be extracted from the audio signal. First, the ICEEMDAN algorithm, a multi-feature extraction method, decomposes the audio signal into multiple intrinsic modes (IMFs). Next, features such as the mean, variance, kurtosis, skewness, energy, center frequency, peak amplitude, and permutation entropy are extracted for each IMF. Finally, these features are used to classify the four emotions of anger, happiness, sadness, and neutrality to identify the user's current emotional state.

[0069] S400: Convert the calibrated throat vibration signal into a digital signal, and convert the digital signal into text information.

[0070] Specifically, the conversion of laryngeal vibration signals (analog signals) to digital signals is a process that converts continuous signals into discrete signals. Its core steps include sampling, quantization, and encoding. This process enables analog signals to be processed and stored by computers or other digital systems. After the laryngeal vibration signals are converted into digital signals, they are converted into text through acoustic models. Finally, the text is corrected and optimized, including spelling correction, grammatical correction, and sentence segmentation, to improve readability and accuracy. In addition, semantic analysis can be further combined with task models (such as dialogue systems) to ensure that the output meets user intent.

[0071] S500, based on the converted text information, synthesizes and outputs speech through text-to-speech technology, combined with the emotional state and the selected timbre.

[0072] Specifically, based on the converted text information, text-to-speech technology is used to synthesize and output speech, combining emotional state and selected timbre parameters. Text-to-speech technology relies on a speech synthesis algorithm, using a neural network to convert text into natural, fluent speech. Furthermore, users can select different timbres (e.g., male, female, childlike) and emotions (e.g., cheerful, calm, sad, etc.) based on their needs, achieving diverse speech output.

[0073] In one embodiment, before comparing the vibration, pressure, and sound multi-dimensional data with the throat vibration data in the standard database, the method further includes:

[0074] The vibration, pressure and sound multi-dimensional data are filtered and processed; the filtered vibration, pressure and sound multi-dimensional data are denoised through a noise reduction algorithm.

[0075] Specifically, the collected data is filtered to remove interference signals such as high-frequency noise, and a noise reduction algorithm is used to improve the purity of the data. Filtering is one of the basic operations in data preprocessing, and its main purpose is to remove unnecessary noise by selectively retaining or suppressing certain frequency components. In throat vibration signal processing, high-frequency noise is usually related to environmental noise or interference during the sampling process. Therefore, the use of a low-pass filter can effectively remove high-frequency noise and retain low-frequency components, thereby improving the purity of the signal. Noise reduction algorithm is another effective means of removing noise, especially when the noise has a complex structure or non-uniform distribution. The noise reduction algorithm can remove noise more accurately while retaining the characteristics of the signal. The noise reduction algorithm can use an adaptive noise reduction algorithm to automatically adjust the noise reduction parameters according to the statistical characteristics of the signal, thereby more effectively removing noise.

[0076] In one embodiment, after calibrating the vibration, pressure, and sound multi-dimensional data based on the comparison results and obtaining the calibrated throat vibration signal, the method further includes:

[0077] The calibrated throat vibration signal is converted into an electrical signal, and the electrical signal is compressed and encrypted; based on the processed electrical signal, a private database is established to store the user's personalized throat vibration data.

[0078] Specifically, after signal acquisition, the throat vibration signal needs to be converted into an electrical signal and compressed to reduce the data volume for subsequent processing and storage. Compression technology can use lossless or lossy compression. During data transmission, to ensure data security, the electrical signal needs to be encrypted. Encryption technology can use symmetric or asymmetric encryption. After signal compression and encryption, the processed electrical signal is stored in a private database to store the user's personalized throat vibration information.

[0079] In this embodiment, the establishment of a private database provides users with the ability to store and analyze laryngeal tremor data for a long time. By regularly updating and maintaining the database, users can accumulate a large amount of historical data for use in predictive modeling.

[0080] In one embodiment, Figure 2 As shown, S300 includes:

[0081] S320, extracting an audio signal from the throat vibration signal, and decomposing the audio signal into multiple intrinsic modes using the ICEEMDAN algorithm;

[0082] S340, extracting multiple features of each intrinsic mode;

[0083] S360, based on the multiple features of each intrinsic pattern, uses support vector machine classification to identify the emotional state of the user's current voice.

[0084] Specifically, to identify different emotions in speech signals, multiple physical features are extracted from each order of IMF components in the framed signal. First, the speech signal is decomposed using the ICEEMDAN algorithm to obtain multi-order IMF components. Then, statistical parameters (including mean, variance, skewness, and kurtosis), center frequency, energy, peak amplitude, and permutation entropy are extracted from the time domain signal of each order IMF. These emotional speech features directly extracted from the IMF components complement each other and can comprehensively analyze emotional speech signals from different perspectives, facilitating more comprehensive emotional information extraction. Different types of emotional speech signals are affected by differences in statistical parameters such as energy distribution, symmetry, and the tail / top of the energy distribution. These variations can be captured by evaluating statistical parameters of the time domain signal under different modes, such as mean, variance, skewness, and kurtosis. Center frequency is related to the speaker's emotional state. For example, happy or angry speech signals may have a higher center frequency, sad speech signals may have a lower center frequency, and neutral speech signals may have a more stable center frequency. Peak amplitude reflects the amplitude and variability of emotional speech signals. These features may be related to the intensity and variability of the speaker's emotions. For example, a speech signal expressing happiness or surprise may have a larger peak amplitude, while a speech signal expressing sadness or disgust may have a smaller peak amplitude. Energy is related to the intensity and arousal of the speaker's emotions. For example, high-energy speech typically expresses emotions such as anger, excitement, or surprise, while low-energy speech may express emotions such as calmness, dullness, or sadness. Permutation entropy is a metric used to describe the complexity of time series or chaotic dynamical systems. It is used to quantify the randomness of a signal and can reflect the differences and similarities between emotional speech signals in different emotional states. After extracting features from the time domain signal of each order IMF, support vector machine classification is performed based on the extracted features to identify the emotional state of the user's current voice.

[0085] In one embodiment, synthesizing and outputting speech based on the converted text information using text-to-speech technology, combined with the emotional state and the selected timbre, includes:

[0086] Through text-to-speech technology, the audio signal is mapped to a low-dimensional latent space to obtain the latent representation of the reference speech; the converted text information is mapped to a latent representation consistent with the reference speech to obtain the latent representation of the text; based on the latent representation of the reference speech and the latent representation of the text, the duration of the speech is estimated through a trained duration predictor; combined with the emotional state and the selected timbre, the latent representation of the text and the duration of the speech are input into the trained speech autoencoder to synthesize and output the speech.

[0087] Specifically, the text-to-speech technology uses Supertonic TTS. First, a speech autoencoder is trained to map the audio signal into a low-dimensional latent space. Specifically, features are extracted from the audio signal to produce a mel-spectrogram. The mel-spectrogram is then compressed into a low-dimensional latent space representation through a series of convolutional layers (ConvNeXt blocks) and normalization layers (such as BatchNorm and LayerNorm), resulting in a latent representation of the reference speech. Next, the text information is processed at the character level and aligned with the speech using a cross-attention mechanism. This avoids the use of traditional G2P modules and external aligners, mapping the text information into a latent representation consistent with the reference speech. This latent representation accurately reflects the semantic features of both the text content and the reference speech. Secondly, a duration predictor is trained. The latent representation of the reference speech and the text are input into the trained duration predictor. Using a cross-attention mechanism and latent compression techniques, the duration of the speech is predicted. Finally, the latent representation of the text and the duration of the speech are input into the trained speech autoencoder, combining the emotional state and selected timbre. The speech is synthesized and output. The emotional state can be selected by the user.

[0088] In this embodiment, by compressing audio and text information into a low-dimensional space, the number of model parameters and computational overhead are reduced; ConvNeXt blocks are used to replace traditional convolutional layers to improve the model's inference speed and architectural efficiency; and dependencies such as the G2P module, text-to-speech aligner, and external pre-trained models are removed, simplifying the entire TTS pipeline.

[0089] In one embodiment, converting the calibrated throat vibration signal into a digital signal, and converting the digital signal into text information includes:

[0090] The calibrated throat vibration signal is converted into a digital signal; the digital signal is decoded into original speech features, and the original speech features are mapped into phonemes through an acoustic model to obtain a phoneme sequence; based on the phoneme sequence, a statistical language model is used to predict the vocabulary sequence, and text information is generated based on the vocabulary sequence.

[0091] Specifically, first, the calibrated laryngeal vibration signal is converted into a digital signal, and the digital signal is decoded into original speech features, which can characterize the tone, energy and frequency distribution of the speech; then, the original speech features are input into an acoustic model (such as a deep neural network / DNN, a recurrent neural network / RNN), which maps the original speech features into phonemes, which are the smallest units of speech, to obtain a phoneme sequence; finally, the phoneme sequence is input into a statistical language model (such as N-gram), which combines grammatical rules and contextual information to predict the most likely vocabulary sequence, generate text information, and perform error correction and optimization on the generated text information.

[0092] In one embodiment, converting the calibrated throat vibration signal into a digital signal includes:

[0093] The calibrated throat vibration signal is sampled by a sampler at a preset frequency to collect the amplitude value of discrete time points; the continuous amplitude value after sampling is approximated to a finite number of discrete values ​​through a quantization method; the quantized discrete values ​​are converted into a binary code stream to form a digital signal.

[0094] Specifically, converting the calibrated laryngeal vibration signal into a digital signal mainly involves three steps: sampling, quantization, and encoding. First, the laryngeal vibration signal is sampled at a specific frequency (e.g., 8000 sampling times per second) by a sampler to collect the amplitude values ​​of discrete time points. During the sampling process, an instantaneous value of the signal is "intercepted" within a specific time interval, thus forming a discrete time series. Second, the continuous amplitude values ​​obtained by sampling are converted into a finite number of discrete values. Since the amplitude of the laryngeal vibration signal is continuous, while digital signals can only represent a finite number of discrete values, each continuous amplitude value obtained by sampling needs to be approximated with a certain accuracy. Finally, the quantized discrete values ​​are converted into a binary code stream (e.g., PCM format) to form a digital signal.

[0095] In one embodiment, Figure 3 As shown, a speech generation device based on laryngeal vibration signal analysis is provided, comprising: a laryngeal vibration signal acquisition module 10, a data calibration module 20, an emotional state recognition module 30, a laryngeal vibration signal conversion module 40 and a speech synthesis module 50, wherein:

[0096] The throat vibration signal acquisition module 10 is used to collect the throat vibration signal through the throat vibration sound generator. The throat vibration signal carries multi-dimensional data of vibration, pressure and sound;

[0097] The data calibration module 20 is used to compare the vibration, pressure and sound multi-dimensional data with the throat vibration data in the standard database, and calibrate the vibration, pressure and sound multi-dimensional data based on the comparison results to obtain a calibrated throat vibration signal;

[0098] The emotional state recognition module 30 is used to extract the audio signal from the throat vibration signal, extract the audio signal features using the ICEEMDAN algorithm, and identify the emotional state of the user's current voice using a classification model based on the audio signal features;

[0099] a throat vibration signal conversion module 40 for converting the calibrated throat vibration signal into a digital signal, and then converting the digital signal into text information;

[0100] The speech synthesis module 50 is used to synthesize and output speech based on the converted text information through text-to-speech technology, combined with the emotional state and the selected timbre.

[0101] In one embodiment, the speech generation device based on laryngeal vibration signal analysis also includes a denoising module for filtering the vibration, pressure and sound multi-dimensional data; and denoising the filtered vibration, pressure and sound multi-dimensional data through a noise reduction algorithm.

[0102] In one embodiment, the speech generation device based on laryngeal vibration signal analysis also includes a personalized data storage module, which is used to convert the calibrated laryngeal vibration signal into an electrical signal and compress and encrypt the electrical signal; based on the processed electrical signal, a private database is established to store the user's personalized laryngeal vibration data.

[0103] In one embodiment, the emotional state recognition module 30 is also used to extract audio signals from the throat vibration signal, and decompose the audio signal into multiple intrinsic modes through the ICEEMDAN algorithm; extract multiple features of each intrinsic mode, and the multiple features of the intrinsic mode include statistical parameters, center frequency, energy, peak amplitude and permutation entropy; according to the multiple features of each intrinsic mode, classification is performed through a support vector machine to identify the emotional state of the user's current voice.

[0104] In one embodiment, the speech synthesis module 50 is also used to map the audio signal to a low-dimensional latent space through text-to-speech technology to obtain a latent representation of the reference speech; map the converted text information to a latent representation consistent with the reference speech to obtain a latent representation of the text; estimate the duration of the speech through a trained duration predictor based on the latent representation of the reference speech and the latent representation of the text; and input the latent representation of the text and the duration of the speech into a trained speech autoencoder in combination with the emotional state and the selected timbre to synthesize and output the speech.

[0105] In one embodiment, the throat vibration signal conversion module 40 is also used to convert the calibrated throat vibration signal into a digital signal; decode the digital signal into original speech features, and map the original speech features into phonemes through an acoustic model to obtain a phoneme sequence; predict the vocabulary sequence based on the phoneme sequence through a statistical language model, and generate text information based on the vocabulary sequence.

[0106] In one embodiment, the throat vibration signal conversion module 40 is also used to collect the amplitude values ​​of the calibrated throat vibration signal at discrete time points through a sampler at a preset frequency; through a quantization method, the sampled continuous amplitude values ​​are approximated to a finite number of discrete values; and the quantized discrete values ​​are converted into a binary code stream to form a digital signal.

[0107] Each module in the aforementioned speech generation device based on laryngeal vibration signal analysis can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0108] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store infrared image data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a speech generation method based on laryngeal vibration signal analysis is implemented.

[0109] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0110] The above are all preferred embodiments of the present application, and are not intended to limit the scope of protection of the present application. Therefore, any equivalent changes made based on the structure, shape, and principle of the present application should be included in the scope of protection of the present application.

Claims

1. A speech generation method based on laryngeal vibration signal analysis, characterized in that: include: The throat vibration signal is collected by a throat vibration sound generator, and the throat vibration signal carries multi-dimensional data of vibration, pressure and sound; Comparing the vibration, pressure, and multi-dimensional sound data with throat vibration data in a standard database, and calibrating the vibration, pressure, and multi-dimensional sound data based on the comparison results to obtain a calibrated throat vibration signal; Extracting an audio signal from the throat vibration signal, extracting audio signal features using the ICEEMDAN algorithm, and identifying the emotional state of the user's current voice using a classification model based on the audio signal features; Converting the calibrated laryngeal vibration signal into a digital signal, and converting the digital signal into text information; Based on the converted text information, the speech is synthesized and output through text-to-speech technology, combined with the emotional state and the selected timbre.

2. The method for speech generation based on laryngeal vibration signal analysis according to claim 1, characterized in that: Before comparing the vibration, pressure and sound multi-dimensional data with the throat vibration data in the standard database, the method further includes: performing filtering processing on the vibration, pressure, and sound multi-dimensional data; The vibration, pressure and sound multi-dimensional data after filtering are denoised using a noise reduction algorithm.

3. The method for speech generation based on laryngeal vibration signal analysis according to claim 1, characterized in that: After calibrating the vibration, pressure, and sound multi-dimensional data based on the comparison results to obtain a calibrated throat vibration signal, the method further includes: converting the calibrated throat vibration signal into an electrical signal, and compressing and encrypting the electrical signal; Based on the processed electrical signals, a private database is established to store the user's personalized throat vibration data.

4. The method for speech generation based on laryngeal vibration signal analysis according to claim 1, characterized in that: The extracting of the audio signal from the throat vibration signal, extracting the audio signal features by the ICEEMDAN algorithm, and identifying the emotional state of the user's current voice by a classification model based on the audio signal features includes: Extracting an audio signal from the throat vibration signal, and decomposing the audio signal into multiple intrinsic modes using an ICEEMDAN algorithm; Extracting multiple features of each intrinsic mode, wherein the multiple features of the intrinsic mode include statistical parameters, center frequency, energy, peak amplitude, and permutation entropy; Based on the multiple features of each intrinsic pattern, classification is performed using a support vector machine to identify the emotional state of the user's current voice.

5. The method for speech generation based on laryngeal vibration signal analysis according to claim 1, characterized in that: The synthesizing and outputting of speech based on the converted text information by using text-to-speech technology in combination with the emotional state and the selected timbre includes: Mapping the audio signal to a low-dimensional latent space using text-to-speech technology to obtain a latent representation of the reference speech; Mapping the converted text information to a potential representation consistent with the reference speech to obtain a potential representation of the text; estimating the duration of speech using a trained duration predictor based on the latent representation of the reference speech and the latent representation of the text; Combined with the emotional state and the selected timbre, the latent representation of the text and the duration of the speech are input into a trained speech autoencoder to synthesize and output speech.

6. The method for generating speech based on laryngeal vibration signal analysis according to claim 1, characterized in that: The step of converting the calibrated throat vibration signal into a digital signal and converting the digital signal into text information includes: Converting the calibrated laryngeal vibration signal into a digital signal; Decoding the digital signal into original speech features, and mapping the original speech features into phonemes through an acoustic model to obtain a phoneme sequence; According to the phoneme sequence, a vocabulary sequence is predicted through a statistical language model, and text information is generated according to the vocabulary sequence.

7. The method for generating speech based on laryngeal vibration signal analysis according to claim 6, characterized in that: The converting the calibrated throat vibration signal into a digital signal comprises: The calibrated laryngeal vibration signal is sampled by a sampler at a preset frequency to collect amplitude values ​​of discrete time points; Through the quantization method, the continuous amplitude value after sampling is approximated to a finite number of discrete values; Convert the quantized discrete value into a binary code stream to form a digital signal.

8. A speech generation device based on laryngeal vibration signal analysis, characterized in that: include: A throat vibration signal acquisition module is used to collect throat vibration signals through a throat vibration sounder. The throat vibration signals carry multi-dimensional data of vibration, pressure and sound; a data calibration module, configured to compare the vibration, pressure, and multi-dimensional sound data with throat vibration data in a standard database, and calibrate the vibration, pressure, and multi-dimensional sound data based on the comparison results to obtain a calibrated throat vibration signal; An emotional state recognition module is used to extract an audio signal from the throat vibration signal, extract audio signal features using the ICEEMDAN algorithm, and identify the emotional state of the user's current voice using a classification model based on the audio signal features; a throat vibration signal conversion module, configured to convert the calibrated throat vibration signal into a digital signal, and then convert the digital signal into text information; The speech synthesis module is used to synthesize and output speech based on the converted text information through text-to-speech technology, combined with the emotional state and the selected timbre.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images