Speech synthesis method based on electroencephalogram emotion

Abstract

The invention discloses a speech synthesis method based on electroencephalogram emotion, and relates to the field of intelligent speech synthesis. Comprising the following steps: acquiring electroencephalogram emotion data after a testee hears a voice segment by adopting an electroencephalogram signal acquisition instrument, carrying out emotion labeling, obtaining labeled emotion extreme value group data, and then preprocessing the labeled emotion extreme value group data; performing electroencephalogram emotion measurement model training on the convolutional neural network through the preprocessed emotion extreme value group data to obtain a trained electroencephalogram emotion measurement model; outputting a recognition result of the trained electroencephalogram emotion measurement model as input of a vits model, and performing emotion speech synthesis model training to obtain a trained emotion speech synthesis model; and performing voice synthesis on the movie and television play to be dubbed through the trained emotion voice synthesis model, and outputting a final voice synthesis result. According to the method, the electroencephalogram emotion measurement model and the emotion speech synthesis model are provided, speech generation can be optimized under the emotion measurement result, and the speech which meets the common-emotional requirements of audiences and is rich in emotion is synthesized.

Description

[0001] This application is a divisional application. The application number of the original application is 202410603270.5, the application date is May 15, 2024, and the name of the invention is "A Cantonese speech intelligent synthesis method and system based on EEG emotion measurement". Technical Field

[0002] The present invention relates to the technical field of intelligent speech synthesis, and in particular to a speech synthesis method based on electroencephalogram emotions. Background Art

[0003] Speech synthesis refers to the use of computers to automatically generate speech from text. In the film and television industry, automatic dubbing programs can automatically generate speech from the text lines, emotional expressions, timbre and other elements in the script, and then match the speech to the picture, thereby greatly reducing the dubbing cost. However, since film and television production needs to achieve empathy, the generated speech has extremely high emotional requirements. Therefore, emotional factors have become the focus of research on speech synthesis in film and television.

[0004] Thanks to the development of deep neural networks, intelligent speech synthesis has made great progress. Speech synthesis based on deep neural networks usually achieves text-to-speech conversion by building a parameterized neural network and using sample data in the form of text-speech pairs to model the mapping relationship between text and speech. This simple data-driven speech synthesis method lacks the mining and modeling of finer-grained factors such as timbre and emotion.

[0005] The usual approach to emotional speech synthesis is to enable the model to learn the emotional style of speech through explicit labels, that is, by manually labeling the speech with text labels that express emotions, the model learns the rhythm of speech during the speech synthesis process, thereby achieving the effect of emotional speech generation. However, explicit text labels can only enable the model to learn the average style of sample data and the basic rhythm of speech, and still lack fine-grained analysis of emotional rhythm. In addition, this form of speech annotation in the form of text labels is highly dependent on the model designer's way of modeling emotional metrics and the subjective will of people, and the effect of emotional style learning and expression needs to be improved.

[0006] There are two existing methods for emotion measurement modeling: discrete and continuous. The mainstream discrete method includes several basic emotional states and related extensions, such as "anger", "expectation", "fear", "sadness", "trust", "surprise", "joy", etc. In addition, there are methods based on the Palette Theory, which can further use basic emotional states as primary colors to generate other emotions. There are also emotion wheel representation methods and attribute-based or layered emotion quantification methods.

[0007] Compared with the discrete method of measuring emotions, the continuous method can represent more detailed emotional states and has higher accuracy. The continuous method usually uses several basic coordinate axes to characterize emotions. The commonly used method is the Valence-Arousal bipolar emotional quadrant system, which describes emotions from the two dimensions of valence and arousal.

[0008] Studies have shown that changes in the potential information of the cerebral cortex can represent a lot of information related to human cognition. When people hear the emotional characteristics in the audio, it will stimulate their imagination, which will cause changes in the potential of the cerebral cortex. This potential change can be represented by an electroencephalogram (EEG). Through signal processing methods, more fine-grained information contained in the EEG can be extracted, such as information about people's emotional changes in the audio.

[0009] In general, since different audiences have different emotional empathy points for film and television content, the speech synthesis method based on early explicit label training cannot provide differentiated expressions for specific audiences. Based on this problem, a more fine-grained emotion modeling method is needed to guide the optimization of speech synthesis effects.

[0010] Therefore, there is an urgent need to propose a speech synthesis method based on EEG emotions to solve the limitations of existing technologies. Summary of the invention

[0011] In view of this, the present invention provides a speech synthesis method based on EEG emotions, which optimizes speech generation under the emotion measurement results by constructing an EEG emotion measurement model and an emotional speech synthesis model, and synthesizes emotion-rich speech that meets the audience's empathy needs.

[0012] In order to achieve the above object, the present invention adopts the following technical solution:

[0013] A method for speech synthesis based on EEG emotion comprises the following steps:

[0014] S1. Data acquisition: Use an EEG signal acquisition device to obtain the EEG emotional data of the tester after hearing the voice clip;

[0015] S2. Data labeling: labeling the collected EEG emotion data to obtain the labeled emotion extreme value group data;

[0016] S3. Data preprocessing: preprocessing the emotion extreme value group data to obtain preprocessed emotion extreme value group data;

[0017] S4. EEG emotion measurement model training: The EEG emotion measurement model is trained on the convolutional neural network using the preprocessed emotion extreme value group data to obtain a trained EEG emotion measurement model;

[0018] S5. Emotional speech synthesis model training: output the recognition result of the trained EEG emotion measurement model as the input of the vits model, so as to train the emotional speech synthesis model and obtain the trained emotional speech synthesis model;

[0019] S6. Speech synthesis: Use the trained emotional speech synthesis model to perform speech synthesis on the dubbed film and television dramas and output the final speech synthesis result.

[0020] In the above method, optionally, the specific content of using an EEG signal collector in S1 to obtain the EEG emotion data of the tester after hearing the voice clip is:

[0021] The tester puts on the EEG collection helmet, calibrates the electrode position, and turns on the recorder and data collection software in sequence. The tester completes the calibration process of opening and closing eyes and clicking the mouse according to the instructions of the data collection software. The computer calculates the signal calibration time deviation during the calibration process. The tester will hear a voice clip, and the EEG collection helmet collects the tester's EEG data and marks the time when the voice clip starts and ends in the EEG data.

[0022] In the above method, optionally, in S2, the collected EEG emotion data is emotionally labeled, and the specific content of the labeled emotion extreme value group data is:

[0023] First, feature processing is performed on the EEG emotion data obtained in S1 to filter out the EEG emotion data that can represent the extreme values ​​of emotion, and the emotion polarity of the EEG emotion data is labeled.

[0024] In the above method, optionally, the emotion extreme value group data is an a extreme value group, a b extreme value group and a c extreme value group for classifying six emotions according to the bipolar characteristics of emotions.

[0025] In the above method, optionally, in S3, the emotion extreme value group data is preprocessed to remove noise to obtain the emotion extreme value group data after the noise is removed.

[0026] In the above method, optionally, in S4, the convolutional neural network is trained with the preprocessed extreme value group data to obtain the EEG emotion measurement model, and the specific content of the trained EEG emotion measurement model is:

[0027] S41. Input signal: The input signal has two sections, including anchor sample data anchor and real-time input EEG data input. The anchor sample data is the emotion extreme value group data after noise removal obtained in S3.

[0028] S42. Feature extraction: The EEG emotion measurement model mainly captures the time period of emotion fluctuations. Therefore, the window sliding method is used to frame the emotion extreme value group data after noise removal. The framed data is expressed as:

[0029]

[0030]

[0031] in, C is the number of channels, T is the time length, is the real number domain, M is the window size, is the splicing operation, N is the number of frames;

[0032] Each frame is extracted by convolutional neural network and represented as:

[0033]

[0034] Among them, K is a two-dimensional convolution kernel, and the convolution is performed along the channel vertically and along the time axis horizontally. n For each frame signal, The feature map represents the output result of each frame signal after convolution, D is a D-dimensional vector, i is the vertical position index of the feature map pixel, j is the horizontal position index of the feature map pixel, c is the vertical position index of the convolution kernel, and m is the horizontal position index of the convolution kernel;

[0035] S43. Anchored coherent noise removal module based on attention mechanism: The attention mechanism is applied to eliminate features similar to the anchored data. The features of the anchored sample data are taken as K, and the features of the real-time input EEG data are taken as Q and V, which are expressed as:

[0036]

[0037]

[0038] The distance is expressed as:

[0039]

[0040] The distance attention mechanism is expressed as:

[0041]

[0042] Among them, d ij is the distance between the i-th feature and the j-th feature;

[0043] S44. Classifier classifies: A input After the features are compressed by the multi-layer convolutional neural network, they are input into the fully connected neural network for sentiment measurement prediction:

[0044]

[0045]

[0046] Among them, Z is the feature vector flattened after the convolution operation, the vector dimension is O, ANN is the fully connected neural network function, It is a scalar value between 0 and 1 representing the emotion measure. The closer the predicted value is to 0, the more similar the emotion of the input EEG signal input_trail is to the EEG signal anchor_trail used for anchoring. Conversely, the more dissimilar it is, the more similar the emotion measure of the input EEG signal is determined.

[0047] S45. Train each sample labeled with the emotion scale separately to obtain three trained EEG emotion measurement models

[0048] In the above method, optionally, the recognition result of the trained EEG emotion measurement model is output in S5 as the input of the vits model, so as to train the emotional speech synthesis model, and the specific content of the trained emotional speech synthesis model is:

[0049] The emotional speech synthesis model is divided into model learning speech reconstruction training and model emotion enhancement learning training;

[0050] S51. Model learning speech reconstruction: Use the preprocessed emotional extreme value group data obtained in S3 to train the model, learn speech reconstruction, and embed the emotional information in the text encoder in the form of a vector; the loss function is the generated speech mel spectrum graph Mel spectrogram with real samples X mel The gap between them, that is, whether the speech can be reconstructed, the loss value for each sample is expressed as:

[0051]

[0052] S52. Model emotion enhancement learning: Use the trained EEG emotion measurement model obtained in S4 to perform emotion recognition on the Cantonese speech generated by AI. Import the voice actor's feedback signal on the AI ​​dubbing emotion through the device to calibrate the encoding loss function of the AI ​​dubbing emotion encoder, and then calculate the loss value to complete the model enhancement learning training for the emotional speech synthesis model. The total loss value calculation formula is expressed as:

[0053] loss stage2 =αloss recon +βloss emotion

[0054]

[0055] Among them, α and β are the weight coefficients of reconstruction loss and sentiment loss in the total loss of model reinforcement learning.

[0056] A Cantonese speech intelligent synthesis system based on EEG emotion measurement, applying any of the above-mentioned speech synthesis methods based on EEG emotion, comprising: a data acquisition module, a data annotation module, a data preprocessing module, an EEG emotion measurement model training module, an emotional speech synthesis model training module, and a speech synthesis module;

[0057] A data acquisition module is connected to the input end of the data annotation module and is used to acquire the EEG emotion data of the tester after hearing the voice clip using an EEG signal collector;

[0058] A data labeling module is connected to the input end of the data preprocessing module and is used to label the collected EEG emotion data to obtain the labeled emotion extreme value group data;

[0059] A data preprocessing module is connected to the input end of the EEG emotion measurement model training module and is used to preprocess the emotion extreme value group data to obtain the preprocessed emotion extreme value group data;

[0060] The EEG emotion measurement model training module is connected to the input end of the emotional speech synthesis model training module and is used to train the EEG emotion measurement model of the convolutional neural network through the preprocessed emotion extreme value group data to obtain a trained EEG emotion measurement model;

[0061] The emotional speech synthesis model training module is connected to the input end of the speech synthesis module and is used to output the recognition result of the trained EEG emotion measurement model as the input of the vits model, so as to train the emotional speech synthesis model and obtain the trained emotional speech synthesis model;

[0062] The speech synthesis module is used to perform speech synthesis on the dubbed TV series and movies through the trained emotional speech synthesis model, and output the final speech synthesis result.

[0063] It can be seen from the above technical solution that compared with the prior art, the present invention provides a speech synthesis method based on EEG emotions, which has the following beneficial effects: the present invention proposes an EEG emotion measurement model and an emotional speech synthesis model. The emotional speech synthesis model can convert the text in the script into speech. When the audience wears a non-invasive EEG device, they listen to the synthesized speech and generate an electroencephalogram. The electroencephalogram generates an emotion measurement through the EEG emotion measurement model, which is conducive to optimizing speech generation under the emotion measurement results and synthesizing emotionally rich speech that meets the audience's empathy needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0065] Figure 1 A flow chart of a Cantonese speech intelligent synthesis method based on EEG emotion measurement provided by the present invention;

[0066] Figure 2 A schematic diagram of EEG emotion data annotation provided by the present invention;

[0067] Figure 3 A schematic diagram of six EEG emotion measurements provided by the present invention;

[0068] Figure 4 A flowchart of the present invention for preprocessing the noise removal of the extreme value group data;

[0069] Figure 5 A schematic diagram of the structure of the EEG emotion measurement model provided by the present invention;

[0070] Figure 6 A schematic diagram of characteristic distribution provided by the present invention;

[0071] Figure 7 The emotional speech synthesis model framework diagram provided by the present invention;

[0072] Figure 8 A dubbing flow chart based on an EEG emotion measurement model and an emotional speech synthesis model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0073] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] Reference Figure 1 As shown, the present invention discloses a speech synthesis method based on EEG emotion, comprising the following steps:

[0075] S1. Data acquisition: Use an EEG signal acquisition device to obtain the EEG emotional data of the tester after hearing the voice clip;

[0076] S2. Data labeling: labeling the collected EEG emotion data to obtain the labeled emotion extreme value group data;

[0077] S3. Data preprocessing: preprocessing the emotion extreme value group data to obtain preprocessed emotion extreme value group data;

[0078] S4. EEG emotion measurement model training: The EEG emotion measurement model is trained on the convolutional neural network using the preprocessed emotion extreme value group data to obtain a trained EEG emotion measurement model;

[0079] S5. Emotional speech synthesis model training: output the recognition result of the trained EEG emotion measurement model as the input of the vits model, so as to train the emotional speech synthesis model and obtain the trained emotional speech synthesis model;

[0080] S6. Speech synthesis: Use the trained emotional speech synthesis model to perform speech synthesis on the dubbed film and television dramas and output the final speech synthesis result.

[0081] Furthermore, the specific contents of the EEG emotional data obtained by the EEG signal collector after the tester hears the voice clip in S1 are:

[0082] The tester puts on the EEG collection helmet, calibrates the electrode position, and turns on the recorder and data collection software in sequence. The tester completes the calibration process of opening and closing eyes and clicking the mouse according to the instructions of the data collection software. The computer calculates the signal calibration time deviation during the calibration process. The tester will hear a voice clip, and the EEG collection helmet collects the tester's EEG data and marks the time when the voice clip starts and ends in the EEG data.

[0083] Specifically, add physiological saline to the electrodes of the multi-point EEG device and install the electrodes. The tester wears an EEG collection helmet and calibrates the electrode position. The recorder and data acquisition software are turned on in sequence to enter the software calibration phase. The tester completes the calibration process of opening and closing eyes and clicking the mouse according to the software instructions. The computer calculates the signal calibration time deviation T during the calibration process. offset The tester then listened to the emotional voice clip. This process collected EEG data and marked the timestamps of the start and end of the voice clip in the data. begin 、T end Finally, the calculated signal is combined to calibrate the time deviation T offset Label the EEG data. The time index of the EEG data segment is: T begin +T offset :T end +T offset The timestamp of human EEG data is shown as follows: Figure 2 As shown. The EEG and the corresponding speech annotation samples are (Si ,X i ,c),S i is the i-th EEG sample, X i is the i-th audio sample, where the emotion category is c.

[0084] Furthermore, in S2, the collected EEG emotion data is emotionally labeled, and the specific content of the labeled emotion extreme value group data is as follows:

[0085] First, feature processing is performed on the EEG emotion data obtained in S1 to filter out the EEG emotion data that can represent the extreme values ​​of emotion, and the emotion polarity of the EEG emotion data is labeled.

[0086] Furthermore, the emotion extreme value group data is an a extreme value group, a b extreme value group, and a c extreme value group that classify six emotions according to the bipolar characteristics of emotions.

[0087] Specifically, six basic emotions are classified according to their polar opposite characteristics, including three extreme values: [sadness, joy], [anger, expectation], and [fear, surprise]. Each group is called an emotion scale, with the central intersection representing neutral emotions, and the continuous values ​​between the two extreme values ​​representing the intensity of the group of emotions, such as Figure 3 The sample is represented as (S i ,X i ,e y ), where S represents EEG data, X represents speech data, e represents the emotion measurement value, y∈{a,b,c}, a represents the extreme value group [sadness, joy], b represents the extreme value group [anger, expectation], c represents the extreme value group [fear, surprise], e∈[0,1], and e is usually 1 or 0 when annotated.

[0088] Furthermore, in S3, the emotion extreme value group data is preprocessed to remove noise, so as to obtain the emotion extreme value group data after the noise is removed.

[0089] Specifically, the EEG data collected by safer non-invasive EEG acquisition equipment will inevitably introduce more noise, so the process of re-analyzing EEG focuses on removing noise, which includes incoherent noise and coherent noise. Incoherent noise refers to noise whose frequency characteristics are quite different from those of useful signals. This type of noise is manifested as additive noise in the frequency domain and is relatively easy to remove. Coherent noise refers to noise whose frequency characteristics are similar to those of useful signals. This type of noise is easily mixed in and is not easy to remove. The noise removal process is as follows: Figure 4 shown.

[0090] Further, such as Figure 5 As shown, in S4, the EEG emotion measurement model is trained on the convolutional neural network through the preprocessed emotion extreme value group data, and the specific content of the trained EEG emotion measurement model is:

[0091] S41. Input signal: The input signal has two sections, including anchor sample data anchor and real-time input EEG data input. The anchor sample data is the emotion extreme value group data after noise removal obtained in S3.

[0092] S42. Feature extraction: The EEG emotion measurement model mainly captures the time period of emotion fluctuations. Therefore, the window sliding method is used to frame the emotion extreme value group data after noise removal. The framed data is expressed as:

[0093]

[0094]

[0095] in, C is the number of channels, T is the time length, is the real number domain, M is the window size, is the splicing operation, N is the number of frames;

[0096] like Figure 6 As shown, each frame is extracted by convolutional neural network and represented as:

[0097]

[0098] Among them, K is a two-dimensional convolution kernel, and the convolution is performed along the channel vertically and along the time axis horizontally. n For each frame signal, The feature map represents the output result of each frame signal after convolution, D is a D-dimensional vector, i is the vertical position index of the feature map pixel, j is the horizontal position index of the feature map pixel, c is the vertical position index of the convolution kernel, and m is the horizontal position index of the convolution kernel;

[0099] S43. Anchored coherent noise removal module based on attention mechanism: The attention mechanism is applied to eliminate features similar to the anchored data. The features of the anchored sample data are taken as K, and the features of the real-time input EEG data are taken as Q and V, which are expressed as:

[0100]

[0101]

[0102] The distance is expressed as:

[0103]

[0104] The distance attention mechanism is expressed as:

[0105]

[0106] Among them, d ij is the distance between the i-th feature and the j-th feature;

[0107] S44. Classifier classifies: A input After the features are compressed by the multi-layer convolutional neural network, they are input into the fully connected neural network for sentiment measurement prediction:

[0108]

[0109]

[0110] Among them, Z is the feature vector flattened after the convolution operation, the vector dimension is O, ANN is the fully connected neural network function, It is a scalar value between 0 and 1 representing the emotion measure. The closer the predicted value is to 0, the more similar the emotion of the input EEG signal input_trail is to the EEG signal anchor_trail used for anchoring. Conversely, the more dissimilar it is, the more similar the emotion measure of the input EEG signal is determined.

[0111] S45. Train each sample labeled with the emotion scale separately to obtain three trained EEG emotion measurement models

[0112] Specifically, the emotion extreme value group data after noise removal in the S42 feature extraction contains multiple frequency components. According to research, the EEG in the 0Hz to 64Hz frequency band has the greatest impact on emotions. Therefore, filtering the emotion extreme value group data after noise removal can remove most of the incoherent background noise, such as the electrical signals generated by eye movements and muscle movements.

[0113] Furthermore, S5 outputs the recognition result of the trained EEG emotion measurement model as the input of the vits model, thereby training the emotional speech synthesis model. The specific content of the trained emotional speech synthesis model is:

[0114] The emotional speech synthesis model is divided into model learning speech reconstruction training and model emotion enhancement learning training;

[0115] S51. Model learning speech reconstruction: Use the preprocessed emotional extreme value group data obtained in S3 to train the model, learn speech reconstruction, and embed the emotional information in the text encoder in the form of a vector; the loss function is the generated speech mel spectrum graph Mel spectrogram with real samples X mel The gap between them, that is, whether the speech can be reconstructed, the loss value for each sample is expressed as:

[0116]

[0117] S52. Model emotion enhancement learning: Use the trained EEG emotion measurement model obtained in S4 to perform emotion recognition on the Cantonese speech generated by AI. Import the voice actor's feedback signal on the AI ​​dubbing emotion through the device to calibrate the encoding loss function of the AI ​​dubbing emotion encoder, and then calculate the loss value to complete the model enhancement learning training for the emotional speech synthesis model. The total loss value calculation formula is expressed as:

[0118] loss stage2 =αloss recon +βloss emotion

[0119]

[0120] Among them, α and β are the weight coefficients of reconstruction loss and sentiment loss in the total loss of model reinforcement learning.

[0121] Specifically, the specific steps of the S52 model emotion enhancement learning are as follows: the subject (professional voice actor) wears an EEG acquisition device, evaluates the voice generated by the model and performs an emotion selection operation, selects from {sadness, joy, anger, expectation, fear, surprise}, and selects the emotion category and measure l. While the subject is making the selection operation, the system will automatically record the operation timestamp T begin , and then calculate the correction value T offset , and then use the corrected annotation timestamp T label , intercepting EEG signals as EEG emotion measurement model The model will output Then the loss value is calculated and the Cantonese speech generation model is trained again.

[0122] The loss function includes the error between the model-generated speech and the real speech, and the error between the emotion measurement of AI dubbing by professional voice actors monitored by EEG equipment and the real emotion label. Here, the cross entropy form is used. The loss value for each sample can be expressed as:

[0123] loss stage2 =αloss recon +βloss emotion

[0124]

[0125] Among them, α and β are hyperparameters, which are expressed as the weight coefficients of reconstruction loss and sentiment loss in the total loss of model reinforcement learning.

[0126] Specifically, the basic framework of the Cantonese speech emotion speech synthesis model formed by the present invention is VITSS, which is an end-to-end speech synthesis model using adversarial learning conditional variational autoencoder. Speech emotion information is embedded in the text encoder of the VITSS model, and the speech emotion information can come from manual annotation or EEG emotion recognition results. The emotion speech synthesis model is divided into model learning speech reconstruction training and model emotion enhancement learning training. The training process is as follows: Figure 7 shown.

[0127] In a specific embodiment, in order to solve the problem of insufficient emotional expression in traditional Cantonese speech synthesis of film and television dramas trained by explicit text emotion labels, a method for intelligent synthesis of Cantonese speech of film and television dramas based on EEG emotion measurement is proposed. Figure 8 As shown, the specific method is: prepare Cantonese dialogue text according to the content of the film and television drama, and then the dubbing actor wears the EEG device; the dubbing program is started and time calibration is performed; the text content is input into the emotional speech synthesis model; the dubbing actor evaluates the generation effect of the model; if satisfied with the generation effect, the speech generation result is saved; if dissatisfied, the dubbing actor reselects the emotion, the system will automatically obtain the EEG data and input the EEG emotion measurement model to calculate the measurement to optimize the emotional speech synthesis model, and then regenerate the speech, the dubbing actor continues to evaluate until satisfied with the generation effect, then ends the speech generation and saves the speech generation result.

[0128] and Figure 1 Corresponding to the method described above, the embodiment of the present invention also provides a Cantonese speech intelligent synthesis system based on EEG emotion measurement, which is used for Figure 1 The specific implementation of the method includes: data acquisition module, data annotation module, data preprocessing module, EEG emotion measurement model training module, emotion speech synthesis model training module, speech synthesis module;

[0129] A data acquisition module is connected to the input end of the data annotation module and is used to acquire the EEG emotion data of the tester after hearing the voice clip using an EEG signal collector;

[0130] A data labeling module is connected to the input end of the data preprocessing module and is used to label the collected EEG emotion data to obtain the labeled emotion extreme value group data;

[0131] A data preprocessing module is connected to the input end of the EEG emotion measurement model training module and is used to preprocess the emotion extreme value group data to obtain the preprocessed emotion extreme value group data;

[0132] The EEG emotion measurement model training module is connected to the input end of the emotional speech synthesis model training module and is used to train the EEG emotion measurement model of the convolutional neural network through the preprocessed emotion extreme value group data to obtain a trained EEG emotion measurement model;

[0133] The emotional speech synthesis model training module is connected to the input end of the speech synthesis module and is used to output the recognition result of the trained EEG emotion measurement model as the input of the vits model, so as to train the emotional speech synthesis model and obtain the trained emotional speech synthesis model;

[0134] The speech synthesis module is used to perform speech synthesis on the dubbed TV series and movies through the trained emotional speech synthesis model, and output the final speech synthesis result.

[0135] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0136] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech synthesis method based on EEG emotion, characterized in that: The following steps are involved: S1. Data acquisition: Use an EEG signal acquisition device to obtain the EEG emotional data of the tester after hearing the voice clip; S2. Data labeling: labeling the collected EEG emotion data to obtain the labeled emotion extreme value group data; S3. Data preprocessing: preprocessing the emotion extreme value group data to obtain preprocessed emotion extreme value group data; S4. EEG emotion measurement model training: The EEG emotion measurement model is trained on the convolutional neural network using the preprocessed emotion extreme value group data to obtain a trained EEG emotion measurement model; S5. Emotional speech synthesis model training: output the recognition result of the trained EEG emotion measurement model as the input of the vits model, so as to train the emotional speech synthesis model and obtain the trained emotional speech synthesis model; S6. Speech synthesis: Use the trained emotional speech synthesis model to perform speech synthesis on the dubbed TV series and movies, and output the final speech synthesis result; The specific contents of S5 are: The emotional speech synthesis model is divided into model learning speech reconstruction training and model emotion enhancement learning training; S51. Model learning speech reconstruction: Use the preprocessed emotional extreme value group data obtained in S3 to train the model, learn speech reconstruction, and embed the emotional information in the text encoder in the form of a vector; the loss function is the generated speech mel spectrum graph Mel spectrogram with real samples X mel The gap between them, that is, whether the speech can be reconstructed, the loss value for each sample is expressed as: S52. Model emotion enhancement learning: Use the trained EEG emotion measurement model obtained in S4 to perform emotion recognition on the Cantonese speech generated by AI, import the voice actor's feedback signal on the AI ​​dubbing emotion through the device, and use it to calibrate the encoding loss function of the AI ​​dubbing emotion encoder, and then calculate the loss value to complete the model enhancement learning training for the emotional speech synthesis model; The loss function includes the error between the model-generated speech and the real speech, and the error between the emotion measurement of AI dubbing by professional voice actors monitored by EEG equipment and the real emotion label. The loss value for each sample can be expressed in the form of cross entropy: loss stage2 =aloss recon +βloss emotion Among them, α and β are the weight coefficients of reconstruction loss and sentiment loss in the total loss of model reinforcement learning.

2. The method for speech synthesis based on EEG emotion according to claim 1, characterized in that: The specific contents of the EEG emotional data obtained by the EEG signal collector after the tester heard the voice clip in S1 are as follows: Fill the electrodes of the multi-point EEG device with saline and install the electrodes; the tester wears the EEG acquisition helmet, calibrates the electrode position, turns on the recorder and data acquisition software in sequence, and enters the software calibration phase; the tester completes the calibration process of opening and closing eyes and clicking the mouse according to the instructions of the data acquisition software, and the computer calculates the signal calibration time deviation T during the calibration process offset , the tester will hear a voice clip with emotion. During this process, the EEG data of the tester is collected by the EEG helmet and the timestamps of the start and end of the voice clip are marked in the EEG data. begin , T end ; Finally, the calculated signal is combined to calibrate the time deviation T offset Label the EEG data. The time index of the EEG data segment is T begin +T offset :T end +T offset .

3. The method for speech synthesis based on EEG emotion according to claim 1, characterized in that: In S2, the collected EEG emotion data is emotionally labeled, and the specific content of the labeled emotion extreme value group data is as follows: First, feature processing is performed on the EEG emotion data obtained in S1 to filter out the EEG emotion data that can represent the extreme values ​​of emotion, and the emotion polarity of the EEG emotion data is labeled.

4. The method for speech synthesis based on EEG emotion according to claim 3, characterized in that: The emotional extreme value group data are extreme value group a, extreme value group b and extreme value group c, which classify six emotions according to the polar opposite characteristics of emotions; each extreme value group is called an emotional scale, the central intersection represents neutral emotions, and the continuous values ​​between the two extreme values ​​represent the intensity of the group of emotions; The sample is represented as (S i ,X i ,e y ), where S represents EEG data, S i represents the i-th EEG sample, X represents the speech data, X i represents the i-th audio sample, e represents the emotion measurement value, y∈{a,b,c}, a represents the [sadness, joy] extreme value group, b represents the [anger, expectation] extreme value group, c represents the [fear, surprise] extreme value group, e∈[0,1], and e is 1 or 0 when marked.

5. The method for speech synthesis based on EEG emotion according to claim 1, characterized in that: In S3, the emotion extreme value group data is preprocessed to remove noise, so as to obtain the emotion extreme value group data after removing noise; wherein the noise includes incoherent noise and coherent noise.

6. The method for speech synthesis based on EEG emotion according to claim 1, characterized in that: In S4, the EEG emotion measurement model is trained on the convolutional neural network through the preprocessed emotion extreme value group data, and the specific content of the trained EEG emotion measurement model is: S41. Input signal: The input signal has two sections, including anchor sample data anchor and real-time input EEG data input. The anchor sample data is the emotion extreme value group data after noise removal obtained in S3. S42. Feature extraction: The EEG emotion measurement model mainly captures the time period of emotion fluctuations. Therefore, the window sliding method is used to frame the emotion extreme value group data after noise removal. The framed data is expressed as: in, C is the number of channels, T is the time length, is the real number domain, M is the window size, is the splicing operation, N is the number of frames; Each frame is extracted by convolutional neural network and represented as: Among them, K is a two-dimensional convolution kernel, and the convolution is performed along the channel vertically and along the time axis horizontally. n For each frame signal, The feature map represents the output result of each frame signal after convolution, D is a D-dimensional vector, i is the vertical position index of the feature map pixel, j is the horizontal position index of the feature map pixel, c is the vertical position index of the convolution kernel, and m is the horizontal position index of the convolution kernel; S43. Anchored coherent noise removal module based on attention mechanism: The attention mechanism is applied to eliminate features similar to the anchored data. The features of the anchored sample data are taken as K, and the features of the real-time input EEG data are taken as Q and V, which are expressed as: The distance is expressed as: The distance attention mechanism is expressed as: Among them, d ij is the distance between the i-th feature and the j-th feature; S44. Classifier classifies: A input After the features are compressed by the multi-layer convolutional neural network, they are input into the fully connected neural network for sentiment measurement prediction: Among them, Z is the feature vector flattened after the convolution operation, the vector dimension is O, ANN is the fully connected neural network function, It is a scalar value between 0 and 1 representing the emotion measure. The closer the predicted value is to 0, the more similar the emotion of the input EEG signal input_trail is to the EEG signal anchor_trail used for anchoring. Conversely, the more dissimilar it is, the more similar the emotion measure of the input EEG signal is determined. S45. Train each sample labeled with the emotion scale separately to obtain three trained EEG emotion measurement models

Images