Speech conversion method and device based on emotion analysis, equipment and storage medium

Abstract

The invention belongs to the field of artificial intelligence, finance and medical treatment, and relates to a speech conversion method based on sentiment analysis, which comprises the following steps: inputting a preprocessed standard text data set into a pre-constructed conversion synthesis model, performing sentiment analysis through a sentiment analysis structure, combining an acoustic structure with a sentiment analysis result to obtain predicted acoustic features, and converting the predicted acoustic features into speech data. And performing voice waveform synthesis through a voice generator, adjusting training according to a preset loss function to obtain a final emotion voice synthesis model, inputting the to-be-converted text into the emotion voice synthesis model, and synthesizing target voice. The invention also provides a speech conversion device and equipment based on emotion analysis, and a storage medium. In addition, the invention also relates to a block chain technology, and the to-be-converted text can be stored in a block chain. According to the invention, the emotion expression ability is enhanced, and the coherence and fluency of voice are enhanced.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence, financial technology and digital medical technology, and in particular to a speech conversion method, device, equipment and storage medium based on sentiment analysis. Background Art

[0002] In the current medical and financial fields, text-to-speech (TTS) technology, as an important means of information transmission, has been widely used in multiple scenarios such as remote medical consultation, intelligent customer service, and financial information broadcasting. Existing TTS products and technologies have made certain progress in converting text to speech and can provide basic speech output functions. However, current TTS products and technologies still face many deficiencies and defects in practical applications, which limits their in-depth application in fields such as medicine and finance.

[0003] First, most existing TTS systems lack natural emotional expression when generating speech, causing the output speech to sound monotonous and mechanical. In the medical field, emotional communication between doctors and patients is crucial, and speech lacking emotional expression may not accurately convey the doctor's care and instructions, affecting the treatment effect. Similarly, in the financial field, personalized emotional voice services can improve customer satisfaction and loyalty, but the lack of emotion in existing TTS technology makes it difficult to meet this demand.

[0004] Secondly, in order to improve processing speed and efficiency, some TTS systems simplify data during processing, which often leads to the loss of emotional and semantic information in speech. In the fields of medicine and finance, the accuracy and completeness of information are crucial, and any loss of information may lead to misunderstandings or risks. Therefore, the problem of information loss caused by data simplification needs to be solved urgently.

[0005] In addition, existing TTS technologies often rely on pre-recorded voice clips and have poor adaptability to real-time input text and changing emotional expressions. In scenarios with high real-time requirements such as medicine and finance, this lack of adaptability may lead to lags and distortions in information transmission, affecting user experience and decision-making results.

[0006] Finally, many TTS systems are unable to generate personalized voices based on the speaker's characteristics and emotional needs, and lack diversity. In the medical field, different roles such as doctors, patients, and family members may require different voice styles to meet their specific communication needs. In the financial field, personalized voice services are also an important means to enhance brand image and customer stickiness. However, the lack of personalization of existing TTS technology limits its application potential in these areas. Summary of the invention

[0007] The purpose of the embodiments of the present application is to propose a speech conversion method, device, equipment and storage medium based on sentiment analysis to solve the technical problems that the speech generated by the existing TTS technology lacks natural emotional expression, has poor adaptability to real-time input text and changing emotional expressions, and cannot generate personalized speech according to the speaker's characteristics and emotional needs.

[0008] In order to solve the above technical problems, the embodiment of the present application provides a speech conversion method based on sentiment analysis, which adopts the following technical solution:

[0009] Acquire a sample data set, wherein the sample data set includes a large number of paired data of text data samples and corresponding acoustic feature samples;

[0010] Preprocessing the text data sample to obtain a standard text data set;

[0011] Inputting the standard text dataset into a pre-built conversion-synthesis model, wherein the conversion-synthesis model includes a sentiment analysis structure, an acoustic structure, and a speech generator;

[0012] Performing sentiment classification and sentiment embedding on the standard text data set through the sentiment analysis structure to obtain a text sentiment embedding vector;

[0013] Extracting acoustic features by fusing the standard text data set and the text sentiment embedding vector through the acoustic structure to obtain predicted acoustic features;

[0014] Inputting the predicted acoustic features and the acoustic feature samples into the speech generator to synthesize speech waveforms, respectively, to obtain predicted speech and real speech;

[0015] Calculating the loss according to the predicted acoustic feature, the acoustic feature sample, the predicted speech and the real speech according to a preset loss function;

[0016] Adjusting the model parameters of the conversion synthesis model based on the loss, continuing the iterative training until the iteration stop condition is met, and obtaining the final emotional speech synthesis model;

[0017] The text to be converted is obtained, and the text to be converted is input into the emotional speech synthesis model to synthesize the target speech.

[0018] In order to solve the above technical problems, the embodiment of the present application further provides a speech conversion device based on sentiment analysis, which adopts the following technical solution:

[0019] An acquisition module, used to acquire a sample data set, wherein the sample data set includes a large number of paired data of text data samples and corresponding acoustic feature samples;

[0020] A preprocessing module, used for preprocessing the text data sample to obtain a standard text data set;

[0021] An input module, used for inputting the standard text dataset into a pre-built conversion synthesis model, wherein the conversion synthesis model includes a sentiment analysis structure, an acoustic structure and a speech generator;

[0022] A sentiment analysis module, used to perform sentiment classification and sentiment embedding on the standard text data set through the sentiment analysis structure to obtain a text sentiment embedding vector;

[0023] An acoustic feature extraction module, used for extracting acoustic features by fusing the standard text data set and the text sentiment embedding vector with the acoustic structure to obtain predicted acoustic features;

[0024] A speech prediction module, used for inputting the predicted acoustic features and the acoustic feature samples into the speech generator respectively to perform speech waveform synthesis, so as to obtain predicted speech and real speech;

[0025] A loss calculation module, used to calculate the loss according to the predicted acoustic features, the acoustic feature samples, the predicted speech and the real speech according to a preset loss function;

[0026] An iteration module, used for adjusting the model parameters of the conversion synthesis model based on the loss, continuing the iterative training until the iteration stop condition is met, and obtaining a final emotional speech synthesis model;

[0027] The speech generation module is used to obtain the text to be converted, input the text to be converted into the emotional speech synthesis model, and synthesize the target speech.

[0028] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0029] The computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the speech conversion method based on sentiment analysis as described above are implemented.

[0030] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0031] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the speech conversion method based on sentiment analysis as described above are implemented.

[0032] Compared with the prior art, this application has the following beneficial effects:

[0033] The present application provides a speech conversion method based on sentiment analysis, which can improve the text quality by preprocessing the text samples in the sample data set, help to more accurately understand the text content, and accurately identify the emotional tendency in the text; through the sentiment analysis structure, sentiment classification and sentiment embedding are performed on the standard text data set, so as to realize the automatic recognition and expression of text sentiment and enhance the emotional expression ability; through the acoustic structure, the standard text data set and the text sentiment embedding vector are fused to extract acoustic features, and the acoustic features are input into the speech generator for speech waveform synthesis, which can give the speech more emotional colors, make the synthesized speech more natural, reduce monotony and mechanical feeling, improve the accuracy of speech recognition, enhance the coherence and fluency of speech, and improve user experience and satisfaction; converting text to speech through the emotional speech synthesis model can improve the adaptability to real-time input, and can also generate personalized speech according to the speaker's characteristics and emotional needs, and improve the diversity of speech synthesis. In addition, the emotional speech synthesis model can handle multiple speaker characteristics and multiple emotional scenes, providing possibilities for wider applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0036] Figure 2 is a flow chart of an embodiment of a speech conversion method based on sentiment analysis according to the present application;

[0037] Figure 3 is a structural schematic diagram of an embodiment of a speech conversion device based on sentiment analysis according to the present application;

[0038] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0040] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0041] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0042] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0043] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0044] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0045] The server 103 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal device 101 .

[0046] It should be noted that the speech conversion method based on sentiment analysis provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the speech conversion device based on sentiment analysis is generally arranged in the server / terminal device.

[0047] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0048] Continue to refer Figure 2 , shows a flow chart of an embodiment of a speech conversion method based on sentiment analysis according to the present application, comprising the following steps:

[0049] Step S201, obtaining a sample data set, wherein the sample data set includes a large number of paired data of text data samples and corresponding acoustic feature samples.

[0050] Collect speech data sets. Speech data sets can be collected through professional recording studio equipment or obtained through public speech databases, such as LJSpeech, Emo-DB, etc. Preprocess the collected original speech data sets, including removing noise, adjusting speech quality, etc., to obtain the preprocessed speech data sets, obtain the text data sets corresponding to the speech data sets, extract the acoustic features of the speech data in the speech data sets, and annotate the text data sets with the extracted acoustic features, so as to match the text data with the corresponding acoustic features, thereby establishing a mapping relationship between text and speech, and providing supervised learning data for subsequent model training.

[0051] In this embodiment, the electronic device (eg Figure 1The server / terminal device shown in the figure can receive the acquired sample data set through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0052] The text dataset with annotated acoustic features is used as a sample dataset. The sample dataset contains a large amount of paired data, and each paired data includes a text data feature and its corresponding acoustic feature sample.

[0053] Among them, the acoustic features include Mel spectrogram, duration information, pitch (F0) and energy. The Mel spectrogram represents a visual representation of the frequency content of the speech signal; the duration information represents the duration of each phoneme or syllable; the pitch (F0) represents the fundamental frequency of the speech signal, which is related to human pitch perception; and the energy represents the intensity or loudness of the speech signal.

[0054] Step S202, preprocessing the text data samples to obtain a standard text data set.

[0055] In this embodiment, the text data samples are preprocessed so as to standardize the text data samples into standard text data. Specifically, the preprocessing includes cleaning, normalization, word segmentation, etc., wherein cleaning includes removing irrelevant characters and special symbols, removing stop words, spell checking, grammar correction, removing duplicate content, ASCII conversion, lowercase conversion, abbreviation expansion, number expansion and blank compression, for example, the abbreviation "Dr." is expanded to "doctor" and the number "2" is converted to "two". Normalization is to scale or standardize the text data to conform to a specific data distribution or range, for example, using a normalization method to scale the data value to the range of [0,1].

[0056] Step S203, inputting the standard text data set into a pre-built conversion-synthesis model, where the conversion-synthesis model includes a sentiment analysis structure, an acoustic structure, and a speech generator.

[0057] In this embodiment, the pre-built conversion synthesis model includes a sentiment analysis structure, an acoustic structure and a speech generator. The sentiment analysis structure is used to perform sentiment analysis on the text, and the acoustic structure is used to convert the graphics elements (written characters) in the text into phonemes (sound units). The corresponding acoustic features of the text data sample are predicted based on the converted phonemes combined with the sentiment analysis results of the sentiment analysis structure. The acoustic features are converted into audible speech waveforms through the speech generator, and the synthesized speech also has natural emotional expression.

[0058] Step S204, sentiment classification and sentiment embedding are performed on the standard text data set through the sentiment analysis structure to obtain a text sentiment embedding vector.

[0059] The sentiment analysis structure performs sentiment analysis on standard text data in the standard text dataset. Sentiment analysis includes sentiment classification and sentiment embedding generation.

[0060] In this embodiment, the sentiment analysis structure includes a pre-trained sentiment classification model and an embedding generation layer. The standard text data set is input into the sentiment classification model to perform text sentiment classification to obtain text sentiment categories; the text sentiment categories are sentimentally embedded through the embedding generation layer to generate a text sentiment embedding vector.

[0061] The sentiment classification model is a pre-trained model that has learned a large number of language patterns and sentiment expressions. The sentiment classification model can perform sentiment classification on standard text data in the standard text dataset. The embedding generation layer generates sentiment embeddings based on the results of sentiment classification.

[0062] In some optional implementations of this embodiment, the sentiment classification model includes an input layer, an embedding layer, a spatial expansion layer, an encoding layer, and a fully connected layer. The step of inputting the standard text data set into the sentiment classification model to perform text sentiment classification to obtain the text sentiment category includes:

[0063] Preprocess the standard text data set through the input layer to obtain input text data;

[0064] The input text data is converted into a vector through the embedding layer to obtain a text embedding vector;

[0065] The text embedding vector is input into the spatial expansion layer, and the text expansion vector is obtained by concatenating the text embedding vector and the introduced prompt vector;

[0066] The emotional features in the text expansion vector are extracted through the self-attention mechanism of the encoding layer to obtain the emotional semantic features;

[0067] The sentiment semantic features are input into the fully connected layer for classification to obtain the text sentiment category.

[0068] The sentiment classification model uses the improved RoBERTa model. The standard text dataset is preprocessed through the input layer to obtain input text data in a data format suitable for model processing. The input text data is input into the embedding layer for vector conversion. The vector conversion of the embedding layer includes character embedding (token embedding), segmentation embedding (segmentation embedding) and position embedding (position embedding) of the input text data.

[0069] Among them, character embedding is used to represent the main semantic information of a word; segment embedding is used to represent the structure of a sentence to distinguish different sentences or paragraphs; position embedding provides information about the position of a word in a sentence, adding temporal information to the attention mechanism. The vectors corresponding to character embedding, segment embedding, and position embedding are added together to obtain the text embedding vector.

[0070] The spatial expansion layer introduces a learnable prompt vector pool to spatially expand the text embedding vector to enrich the feature representation. Specifically, the text embedding vector is input into the spatial expansion layer, and the pre-trained prompt vector pool is obtained through the spatial expansion layer. The prompt vector pool contains S vectors, where S is a positive integer; the pre-trained k prompt vectors are obtained from the prompt vector pool, where k is a positive integer and is less than or equal to S; the text embedding vector is concatenated with the k prompt vectors to obtain the text expansion vector. By introducing the prompt vector, additional information or constraints are introduced on the basis of the original feature space, thereby expanding the feature space, allowing the model to capture more features related to emotional information, thereby improving the accuracy of sentiment classification.

[0071] The encoding layer is a plurality of stacked connected Transformer modules, including a plurality of stacked connected encoders and a plurality of stacked connected decoders. The semantic and contextual features of the text extension vector are extracted through the encoder's multi-head attention mechanism to obtain the text encoding features. The text encoding features are input into the decoder. The global dependency between the text encoding features is captured through the multi-head attention mechanism and the cross attention mechanism to obtain the emotional semantic features.

[0072] The emotional semantic features are input into the fully connected layer, and the classification probability distribution of the emotional category is output through the Softmax activation function. The text emotional category is obtained according to the classification probability distribution.

[0073] Text sentiment classification through sentiment classification models can efficiently and accurately identify the emotional expression of the text and improve the accuracy of sentiment classification.

[0074] The embedding generation layer can use the embedding layer in the deep learning model, such as the BERT embedding layer, the Transformer embedding layer, etc.; it can also use the word embedding model, such as Word2Vec, GloVe, FastText, etc. The embedding generation layer performs sentiment embedding according to the text sentiment category and generates the corresponding text sentiment embedding vector.

[0075] By performing sentiment classification and sentiment embedding on standard text datasets, the accuracy of text sentiment recognition can be improved, which helps voice convey the emotional color in the text more accurately. Sentiment embedding can generate personalized voice based on the emotional needs of the input text, which helps to flexibly adjust according to the emotional needs of different scenarios and improve adaptability to the input text and expression of emotional changes.

[0076] Step S205, extracting acoustic features by fusing the standard text data set and the text sentiment embedding vector through acoustic structure to obtain predicted acoustic features.

[0077] The core goal of acoustic structure training is to build a model that can convert text information into acoustic features. The acoustic structure includes a phoneme conversion layer, an acoustic encoder, a variance adapter, and an acoustic decoder. The standard text dataset and text sentiment embedding vector are input into the acoustic structure, and processed by the phoneme conversion layer, acoustic encoder, variance adapter, and acoustic decoder in turn to obtain the predicted acoustic features that integrate sentiment information.

[0078] In this embodiment, the step of extracting acoustic features by fusing a standard text dataset and a text sentiment embedding vector through an acoustic structure to obtain predicted acoustic features includes:

[0079] The standard text dataset and text sentiment embedding vector are mapped to phonemes through the phoneme conversion layer to obtain the phoneme embedding sequence;

[0080] Extract features from the phoneme embedding sequence through an acoustic encoder to obtain a phoneme feature sequence;

[0081] Input the phoneme feature sequence into the variance adapter for acoustic feature prediction to obtain an acoustic feature hidden sequence;

[0082] The acoustic feature hidden sequence is decoded by the acoustic decoder to obtain the predicted acoustic features.

[0083] In some optional implementations of this embodiment, the phoneme conversion layer adopts a grapheme to phoneme conversion (G2P) model, and the G2P model includes an input layer, a feature extraction layer, a sequence modeling layer, and an output layer. Specifically, the input text data set is preprocessed through the input layer, including operations such as word segmentation and part-of-speech tagging, to form a text sequence suitable for model processing; the input text sequence and text sentiment embedding vector are feature extracted through the feature extraction layer to capture the phoneme features of the text sequence and the sentiment features of the text sentiment embedding vector; the sentiment features are embedded into the phoneme features, and input into the sequence modeling layer for sequence modeling to obtain a phoneme representation vector that integrates each phoneme feature and sentiment feature; the phoneme representation vector is decoded through the output layer using a beam search-based decoding algorithm to convert it into a phoneme symbol sequence to obtain a complete phoneme embedding sequence.

[0084] In some optional implementations, the acoustic encoder includes multiple layers of self-attention layers and 1D convolution layers; position information is added to the phoneme embedding sequence through position encoding to obtain a phoneme embedding sequence containing the position information; the phoneme embedding sequence containing the position information is input into the multiple layers of self-attention layers, and the phoneme embedding sequence is processed using the multiple layers of self-attention mechanism to capture the dependency between phonemes to obtain a phoneme semantic sequence; and feature extraction of the phoneme semantic sequence is performed through the 1D convolution layer to obtain a phoneme feature sequence to enhance the model's understanding of the phoneme sequence.

[0085] In some optional implementations, the variance adapter includes a duration predictor, a pitch predictor, and an energy predictor, which are used to predict the duration, pitch, and energy of a phoneme, respectively. Specifically, the phoneme feature sequence is input into the duration predictor, and the logarithmic duration value of each phoneme is predicted to obtain a predicted duration value; the pitch logarithmic value of each phoneme in the phoneme feature sequence is predicted by the pitch predictor to obtain a predicted pitch value; the energy level of each phoneme in the phoneme feature sequence is predicted by the energy predictor to obtain predicted energy information; the predicted duration value, the predicted pitch value, and the predicted energy information are embedded and converted to obtain an acoustic feature vector; the acoustic feature vector and the phoneme feature sequence are concatenated to obtain an acoustic feature hidden sequence.

[0086] In this embodiment, the duration predictor uses the first fully connected layer to predict the logarithmic duration value of each phoneme based on the phoneme feature sequence output by the acoustic encoder, and converts the logarithmic duration value into the actual predicted duration value through an exponential function. The pitch predictor uses the second fully connected layer to predict the logarithmic value of the pitch of each phoneme based on the phoneme feature sequence, and converts the logarithmic value of the pitch into the actual predicted pitch value (Hz) through an exponential function. The energy predictor is used to predict the energy level of each phoneme, usually scaled by a sigmoid function to ensure that the energy value is within a reasonable range. The predicted duration value, predicted pitch value, and predicted energy information are feature integrated and embedded in a high-dimensional space to obtain an acoustic feature vector; the acoustic feature vector is added to the phoneme feature sequence to form a new adaptive hidden sequence containing acoustic feature information, namely, an acoustic feature hidden sequence.

[0087] By using the variance adapter to predict duration, pitch, and energy, the naturalness, intelligibility, expressiveness, and clarity of speech can be significantly improved, making the synthesized speech closer to natural human pronunciation, thereby meeting the needs of various application scenarios and having better adaptability to multiple scenarios.

[0088] In some optional implementations, the acoustic decoder includes a self-attention layer, a 1D convolution layer, and an upsampling and residual connection layer, which uses the self-attention layer and the 1D convolution layer to convert the adaptive acoustic feature hidden sequence into an acoustic feature spectrum sequence, and gradually increases the size of the feature map through upsampling and residual connection until the target speech length is reached, and outputs the predicted acoustic features.

[0089] In a specific example, the acoustic decoder is a Mel spectrogram decoder. Mel spectrogram is a type of acoustic feature and is commonly used in speech synthesis. Therefore, the acoustic feature hidden sequence is directly decoded into a Mel spectrogram.

[0090] By fusing text phonemes and text emotions through acoustic structure to predict acoustic features, it is possible to learn the correspondence between input text and acoustic features, and capture the corresponding acoustic features and conversion rules of the text, which helps to improve the naturalness, fluency, intelligibility and accuracy of synthesized speech, and optimize the efficiency and quality of speech synthesis.

[0091] In some optional implementations, during the training of the conversion synthesis model, the acoustic structure is first trained. The loss function in the training process is used to measure the difference between the acoustic features predicted by the model and the actual acoustic features, and the model parameters are optimized to minimize the difference. The loss function may include:

[0092] 1) Mel-spectrogram loss: measures the difference between the predicted Mel-spectrogram and the true Mel-spectrogram, usually calculated using L1 loss or L2 loss.

[0093] 2) Duration loss: measures the difference between the predicted phoneme duration and the true phoneme duration.

[0094] 3) Pitch loss: measures the difference between the predicted pitch and the true pitch.

[0095] 4) Energy loss: measures the difference between the predicted energy level and the actual energy level.

[0096] 5) Total loss: The weighted sum of all the above losses is obtained to obtain the total loss, which is used for end-to-end training of the model.

[0097] The loss function is used to calculate the loss value between the predicted acoustic features and the true acoustic features. According to the loss value, the model parameters of the acoustic structure are updated through the back propagation algorithm. Use an optimization algorithm (such as Adam or SGD) to adjust the model parameters to minimize the loss function. After adjusting the model parameters, continue to iterate the training of the acoustic structure until the performance on the validation set reaches a satisfactory level or the predetermined number of iterations is reached. In order to prevent overfitting, dropout or L2 regularization can be introduced during the training process.

[0098] Step S206: input the predicted acoustic features and the acoustic feature samples into a speech generator to synthesize speech waveforms, thereby obtaining predicted speech and real speech.

[0099] In the speech synthesis stage, the goal is to convert acoustic features into audible speech waveforms while ensuring that the synthesized speech has natural emotional expression.

[0100] In this embodiment, the speech generator includes multiple convolutional layers and residual blocks. The predicted acoustic features and acoustic feature samples are respectively input into the speech generator, and the predicted acoustic features and acoustic feature samples are convolved through multiple convolutional layers to obtain corresponding predicted spectral features and sample spectral features. The predicted spectral features and sample spectral features are converted into waveforms through the residual block to obtain corresponding predicted speech and real speech.

[0101] In a specific example, when the predicted acoustic feature output by the acoustic structure is a predicted Mel spectrogram, the predicted Mel spectrogram is converted into a high-resolution audio waveform through multiple convolutional layers and residual blocks, that is, predicted speech is generated.

[0102] By using a speech generator to synthesize speech waveforms from acoustic features, the naturalness and fluency of speech synthesis can be improved, multi-language and multi-style speech synthesis can be supported, the diversity of speech synthesis can be increased, and personalized customization of speech synthesis can be achieved, thereby enhancing the flexibility and scalability of speech synthesis.

[0103] Step S207, calculating the loss according to the preset loss function based on the predicted acoustic features, the acoustic feature samples, the predicted speech and the real speech.

[0104] In the model training phase, the speech synthesis structure includes a speech generator and a speech discriminator. The speech generator attempts to generate realistic audio waveforms, while the speech discriminator attempts to distinguish between real and generated waveforms. The speech discriminator evaluates the difference between the generated audio waveform and the real waveform, that is, the difference between the predicted speech and the real speech, to guide the training of the speech generator.

[0105] In this embodiment, a speech discriminator corresponding to the speech generator is obtained, and the predicted speech and the real speech are input into the speech discriminator to calculate the adversarial loss; the spectrum loss between the predicted acoustic features and the acoustic feature samples is calculated; the L1 loss is calculated according to the predicted speech and the real speech; the adversarial loss, the spectrum loss and the L1 loss are weighted and summed to obtain the final total loss. Among them, the spectrum loss is the Mel spectrogram loss.

[0106] Specifically, the differential privacy sensitivity is calculated based on the predicted speech and the real speech; based on the differential privacy sensitivity, the privacy loss value, namely the adversarial loss, is calculated using the Markov formula.

[0107] By combining multiple loss functions to calculate losses, we can enhance the robustness of the model, maintain the accuracy of speech features, optimize the training process, improve model performance, balance the quality and diversity of generated speech, and flexibly adapt to different application scenarios and needs.

[0108] Step S208, adjusting the model parameters of the conversion synthesis model based on the loss, continuing the iterative training until the iteration stop condition is met, and obtaining the final emotional speech synthesis model.

[0109] Specifically, the Adam or SGD optimizer is used to adjust the model parameters of the conversion synthesis model according to the loss, and the adjusted model is iteratively trained until the iteration stop condition is met, that is, the number of iterations reaches the preset number or the loss does not change significantly, and the final emotional speech synthesis model is output.

[0110] Step S209, obtaining the text to be converted, inputting the text to be converted into the emotional speech synthesis model, and synthesizing the target speech.

[0111] In this embodiment, the acquired text to be converted is input into the emotional speech synthesis model, and a personalized and emotionally rich target speech is obtained by combining emotional analysis and speech synthesis technology. The emotional speech synthesis model can generate emotional speech based on the context information of the text, realize the automatic recognition and expression of text emotions, and improve the naturalness and expressiveness of the speech.

[0112] It should be emphasized that in order to further ensure the privacy and security of the text to be converted, the above text to be converted can also be stored in a node of a blockchain.

[0113] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0114] This application can improve the text quality by preprocessing the text samples in the sample data set, which helps to more accurately understand the text content and accurately identify the emotional tendency in the text; through the sentiment analysis structure, the standard text data set is sentimentally classified and sentimentally embedded, so as to realize the automatic recognition and expression of text emotions and enhance the emotional expression ability; through the acoustic structure, the standard text data set and the text emotion embedding vector are integrated to extract acoustic features, and the acoustic features are input into the speech generator for speech waveform synthesis, so as to give the speech more emotional colors, make the synthesized speech more natural, reduce the monotony and mechanical feeling, improve the accuracy of speech recognition, enhance the coherence and fluency of speech, and improve the user experience and satisfaction; convert text to speech through the emotional speech synthesis model, so as to improve the adaptability to real-time input, and generate personalized speech according to the speaker's characteristics and emotional needs, so as to improve the diversity of speech synthesis. In addition, the emotional speech synthesis model can handle multiple speaker characteristics and multiple emotional scenarios, providing the possibility for a wider range of applications.

[0115] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0116] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0117] The speech conversion method based on sentiment analysis of this application gives the speech synthesis system a deeper level of naturalness and emotional expression ability. For Ping An Group, the application of the speech conversion method based on sentiment analysis will greatly enhance the interactivity and affinity of its financial services. In the insurance business, this method can provide customers with a more personalized and empathetic service experience through emotionally rich voice interaction, especially in scenarios such as claims consultation and customer care, it can more effectively convey warmth and understanding. In the banking business, the application of this method makes the automatic voice service no longer a cold machine language, but can adjust the tone and expression according to the customer's emotions and needs, enhancing the customer's trust and satisfaction. In addition, in the field of asset management and investment consulting, this method can provide a more vivid and persuasive way of conveying information, helping customers better understand investment products and market dynamics. Overall, the application of this application will make Ping An Group's services more humane, enhance customer loyalty, and enhance its brand image, thereby gaining an advantage in the highly competitive financial market.

[0118] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0119] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0120] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a speech conversion device based on sentiment analysis, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0121] like Figure 3 As shown, the speech conversion device 300 based on sentiment analysis described in this embodiment includes: an acquisition module 301, a preprocessing module 302, an input module 303, a sentiment analysis module 304, an acoustic feature extraction module 305, a speech prediction module 306, a loss calculation module 307, an iteration module 308 and a speech generation module 309. Among them:

[0122] The acquisition module 301 is used for acquiring a sample data set, wherein the sample data set includes a large number of paired data of text data samples and corresponding acoustic feature samples;

[0123] The preprocessing module 302 is used to preprocess the text data sample to obtain a standard text data set;

[0124] The input module 303 is used to input the standard text data set into a pre-built conversion synthesis model, wherein the conversion synthesis model includes a sentiment analysis structure, an acoustic structure and a speech generator;

[0125] The sentiment analysis module 304 is used to perform sentiment classification and sentiment embedding on the standard text data set through the sentiment analysis structure to obtain a text sentiment embedding vector;

[0126] The acoustic feature extraction module 305 is used to extract acoustic features by fusing the standard text data set and the text sentiment embedding vector through the acoustic structure to obtain predicted acoustic features;

[0127] The speech prediction module 306 is used to input the predicted acoustic features and the acoustic feature samples into the speech generator to perform speech waveform synthesis to obtain predicted speech and real speech;

[0128] The loss calculation module 307 is used to calculate the loss according to the predicted acoustic features, the acoustic feature samples, the predicted speech and the real speech according to a preset loss function;

[0129] The iteration module 308 is used to adjust the model parameters of the conversion synthesis model based on the loss, continue iterative training until the iteration stop condition is met, and obtain the final emotional speech synthesis model;

[0130] The speech generation module 309 is used to obtain the text to be converted, input the text to be converted into the emotional speech synthesis model, and synthesize the target speech.

[0131] It should be emphasized that in order to further ensure the privacy and security of the text to be converted, the above text to be converted can also be stored in a node of a blockchain.

[0132] The above-mentioned speech conversion device 300 based on sentiment analysis can improve the text quality by preprocessing the text samples in the sample data set, which helps to more accurately understand the text content and accurately identify the emotional tendency in the text; through the sentiment analysis structure, the standard text data set is sentimentally classified and sentimentally embedded, so as to realize the automatic recognition and expression of text emotions and enhance the emotional expression ability; through the acoustic structure, the standard text data set and the text sentiment embedding vector are integrated to extract acoustic features, and the acoustic features are input into the speech generator for speech waveform synthesis, which can give the speech more emotional colors, make the synthesized speech more natural, reduce monotony and mechanical feeling, improve the accuracy of speech recognition, enhance the coherence and fluency of speech, improve user experience and satisfaction, and convert text to speech through the emotional speech synthesis model, which can improve the adaptability to real-time input, and can also generate personalized speech according to the characteristics and emotional needs of the speaker, so as to improve the diversity of speech synthesis.

[0133] In some optional implementations, the sentiment analysis structure includes a pre-trained sentiment classification model and an embedding generation layer, and the sentiment analysis module 304 includes:

[0134] A classification submodule, used for inputting the standard text data set into the sentiment classification model to perform text sentiment classification and obtain text sentiment categories;

[0135] The sentiment embedding submodule is used to perform sentiment embedding on the text sentiment category through the embedding generation layer to generate a text sentiment embedding vector.

[0136] By performing sentiment classification and sentiment embedding on standard text datasets, the accuracy of text sentiment recognition can be improved, which helps speech convey the emotional color in the text more accurately and improves adaptability to input text and emotional expression changes.

[0137] In some optional implementations of this embodiment, the sentiment classification model includes an input layer, an embedding layer, a spatial expansion layer 、 The encoding layer and the fully connected layer, the classification submodule includes:

[0138] An input unit, used for preprocessing the standard text data set through the input layer to obtain input text data;

[0139] An embedding unit, used for performing vector conversion on the input text data through the embedding layer to obtain a text embedding vector;

[0140] An expansion unit, used for inputting the text embedding vector into the spatial expansion layer, and obtaining a text expansion vector by concatenating the text embedding vector and the introduced prompt vector;

[0141] An encoding unit, used for extracting the sentiment features in the text extension vector through the self-attention mechanism of the encoding layer to obtain sentiment semantic features;

[0142] The classification unit is used to input the emotional semantic features into the fully connected layer for classification to obtain the text emotional category.

[0143] Text sentiment classification through sentiment classification models can efficiently and accurately identify the emotional expression of the text and improve the accuracy of sentiment classification.

[0144] In some optional implementations, the acoustic structure includes a phoneme conversion layer, an acoustic encoder, a variance adapter, and an acoustic decoder, and the acoustic feature extraction module 305 includes:

[0145] A phoneme conversion submodule, used for mapping the standard text data set and the text sentiment embedding vector into phonemes through the phoneme conversion layer to obtain a phoneme embedding sequence;

[0146] An acoustic coding submodule, used for performing feature extraction on the phoneme embedding sequence through the acoustic encoder to obtain a phoneme feature sequence;

[0147] An adapter module, used for inputting the phoneme feature sequence into the variance adapter to perform acoustic feature prediction to obtain an acoustic feature hidden sequence;

[0148] The acoustic decoding submodule is used to decode the acoustic feature hidden sequence through the acoustic decoder to obtain the predicted acoustic feature.

[0149] By fusing text phonemes and text emotions through acoustic structure to predict acoustic features, it is possible to learn the correspondence between input text and acoustic features, and capture the corresponding acoustic features and conversion rules of the text, which helps to improve the naturalness, fluency, intelligibility and accuracy of synthesized speech, and optimize the efficiency and quality of speech synthesis.

[0150] In some optional implementations of this embodiment, the variance adapter includes a duration predictor, a pitch predictor, and an energy predictor, and the adapter module is further used for:

[0151] Inputting the phoneme feature sequence into the duration predictor, predicting the logarithmic duration value of each phoneme, and obtaining a predicted duration value;

[0152] The pitch predictor is used to predict the pitch logarithm of each phoneme in the phoneme feature sequence to obtain a predicted pitch value;

[0153] Predicting the energy level of each phoneme in the phoneme feature sequence by the energy predictor to obtain predicted energy information;

[0154] Embedding and converting the predicted duration value, the predicted pitch value, and the predicted energy information to obtain an acoustic feature vector;

[0155] The acoustic feature vector and the phoneme feature sequence are concatenated to obtain an acoustic feature hidden sequence.

[0156] By using the variance adapter to predict duration, pitch, and energy, the naturalness, intelligibility, expressiveness, and clarity of speech can be significantly improved, making the synthesized speech closer to natural human pronunciation, thereby meeting the needs of various application scenarios and having better adaptability to multiple scenarios.

[0157] In some optional implementations, the speech generator includes a plurality of convolutional layers and residual blocks, and the speech prediction module 306 is further configured to:

[0158] Performing convolution operations on the predicted acoustic features and the acoustic feature samples respectively through the multiple convolution layers to obtain predicted spectrum features and sample spectrum features accordingly;

[0159] The predicted spectral features and the sample spectral features are converted into waveforms through the residual block to obtain corresponding predicted speech and real speech.

[0160] By using a speech generator to synthesize speech waveforms from acoustic features, the naturalness and fluency of speech synthesis can be improved, multi-language and multi-style speech synthesis can be supported, the diversity of speech synthesis can be increased, and personalized customization of speech synthesis can be achieved, thereby enhancing the flexibility and scalability of speech synthesis.

[0161] In some optional implementations of this embodiment, the loss calculation module 307 is further used to:

[0162] Acquire a speech discriminator corresponding to the speech generator, and input the predicted speech and the real speech into the speech discriminator to calculate the adversarial loss;

[0163] Calculating a spectral loss between the predicted acoustic feature and the acoustic feature sample;

[0164] Calculate L1 loss based on the predicted speech and the real speech;

[0165] The adversarial loss, the spectrum loss and the L1 loss are weightedly summed to obtain the final loss.

[0166] By combining multiple loss functions to calculate losses, we can enhance the robustness of the model, maintain the accuracy of speech features, optimize the training process, and improve model performance.

[0167] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0168] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 4 with a memory 41, a processor 42, and a network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0169] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0170] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of a speech conversion method based on sentiment analysis, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0171] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the speech conversion method based on sentiment analysis.

[0172] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0173] By preprocessing the text samples in the sample data set, the text quality can be improved, which helps to more accurately understand the text content and accurately identify the emotional tendency in the text; through the sentiment analysis structure, the standard text data set is subjected to sentiment classification and sentiment embedding, which realizes the automatic recognition and expression of text emotions and enhances the emotional expression ability; through the acoustic structure, the standard text data set and the text sentiment embedding vector are integrated to extract acoustic features, and the acoustic features are input into the speech generator for speech waveform synthesis, which can give the speech more emotional colors, make the synthesized speech more natural, reduce monotony and mechanical feeling, improve the accuracy of speech recognition, enhance the coherence and fluency of speech, improve user experience and satisfaction, and convert text to speech through the emotional speech synthesis model, which can improve the adaptability to real-time input, and can also generate personalized speech according to the speaker's characteristics and emotional needs, and improve the diversity of speech synthesis.

[0174] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the speech conversion method based on sentiment analysis as described above.

[0175] By preprocessing the text samples in the sample data set, the text quality can be improved, which helps to more accurately understand the text content and accurately identify the emotional tendency in the text; through the sentiment analysis structure, the standard text data set is subjected to sentiment classification and sentiment embedding, which realizes the automatic recognition and expression of text emotions and enhances the emotional expression ability; through the acoustic structure, the standard text data set and the text sentiment embedding vector are integrated to extract acoustic features, and the acoustic features are input into the speech generator for speech waveform synthesis, which can give the speech more emotional colors, make the synthesized speech more natural, reduce monotony and mechanical feeling, improve the accuracy of speech recognition, enhance the coherence and fluency of speech, improve user experience and satisfaction, and convert text to speech through the emotional speech synthesis model, which can improve the adaptability to real-time input, and can also generate personalized speech according to the speaker's characteristics and emotional needs, and improve the diversity of speech synthesis.

[0176] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0177] It should be noted that the non-Company's software tools or components that appear in the embodiments of the present application are merely examples and do not represent actual use.

[0178] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A speech conversion method based on sentiment analysis, characterized in that: The steps include: Acquire a sample data set, wherein the sample data set includes a large number of paired data of text data samples and corresponding acoustic feature samples; Preprocessing the text data sample to obtain a standard text data set; Inputting the standard text dataset into a pre-built conversion-synthesis model, wherein the conversion-synthesis model includes a sentiment analysis structure, an acoustic structure, and a speech generator; Performing sentiment classification and sentiment embedding on the standard text data set through the sentiment analysis structure to obtain a text sentiment embedding vector; Extracting acoustic features by fusing the standard text data set and the text sentiment embedding vector through the acoustic structure to obtain predicted acoustic features; Inputting the predicted acoustic features and the acoustic feature samples into the speech generator to synthesize speech waveforms, respectively, to obtain predicted speech and real speech; Calculating the loss according to the predicted acoustic feature, the acoustic feature sample, the predicted speech and the real speech according to a preset loss function; Adjusting the model parameters of the conversion synthesis model based on the loss, continuing the iterative training until the iteration stop condition is met, and obtaining the final emotional speech synthesis model; The text to be converted is obtained, and the text to be converted is input into the emotional speech synthesis model to synthesize the target speech.

2. The method for voice conversion based on sentiment analysis according to claim 1, characterized in that: The sentiment analysis structure includes a pre-trained sentiment classification model and an embedding generation layer. The step of performing sentiment classification and sentiment embedding on the standard text dataset through the sentiment analysis structure to obtain a text sentiment embedding vector includes: Inputting the standard text data set into the sentiment classification model to perform text sentiment classification to obtain text sentiment categories; The text sentiment category is sentimentally embedded through the embedding generation layer to generate a text sentiment embedding vector.

3. The method for voice conversion based on sentiment analysis according to claim 2, characterized in that: The sentiment classification model includes an input layer, an embedding layer, and a spatial expansion layer. 、 The coding layer and the fully connected layer, the step of inputting the standard text data set into the sentiment classification model to perform text sentiment classification, and obtaining the text sentiment category comprises: Preprocessing the standard text data set through the input layer to obtain input text data; Performing vector conversion on the input text data through the embedding layer to obtain a text embedding vector; Inputting the text embedding vector into the spatial expansion layer, and obtaining a text expansion vector by concatenating the text embedding vector and the introduced prompt vector; Extracting the sentiment features in the text extension vector through the self-attention mechanism of the encoding layer to obtain sentiment semantic features; The sentiment semantic features are input into the fully connected layer for classification to obtain text sentiment categories.

4. The method for voice conversion based on sentiment analysis according to claim 1, characterized in that: The acoustic structure includes a phoneme conversion layer, an acoustic encoder, a variance adapter and an acoustic decoder. The acoustic feature extraction is performed by fusing the standard text data set and the text sentiment embedding vector through the acoustic structure to obtain the predicted acoustic feature. The step includes: Mapping the standard text data set and the text sentiment embedding vector into phonemes through the phoneme conversion layer to obtain a phoneme embedding sequence; Extracting features from the phoneme embedding sequence by using the acoustic encoder to obtain a phoneme feature sequence; Inputting the phoneme feature sequence into the variance adapter to perform acoustic feature prediction to obtain an acoustic feature hidden sequence; The acoustic feature hidden sequence is decoded by the acoustic decoder to obtain the predicted acoustic feature.

5. The method for voice conversion based on sentiment analysis according to claim 4, characterized in that: The variance adapter includes a duration predictor, a pitch predictor and an energy predictor, and the step of inputting the phoneme feature sequence into the variance adapter for acoustic feature prediction to obtain an acoustic feature hidden sequence includes: Inputting the phoneme feature sequence into the duration predictor, predicting the logarithmic duration value of each phoneme, and obtaining a predicted duration value; The pitch predictor is used to predict the pitch logarithm of each phoneme in the phoneme feature sequence to obtain a predicted pitch value; Predicting the energy level of each phoneme in the phoneme feature sequence by the energy predictor to obtain predicted energy information; Embedding and converting the predicted duration value, the predicted pitch value, and the predicted energy information to obtain an acoustic feature vector; The acoustic feature vector and the phoneme feature sequence are concatenated to obtain an acoustic feature hidden sequence.

6. The method for voice conversion based on sentiment analysis according to claim 1, characterized in that: The speech generator includes a plurality of convolutional layers and residual blocks. The steps of inputting the predicted acoustic features and the acoustic feature samples into the speech generator for speech waveform synthesis to obtain the predicted speech and the real speech include: Performing convolution operations on the predicted acoustic features and the acoustic feature samples respectively through the multiple convolution layers to obtain predicted spectrum features and sample spectrum features accordingly; The predicted spectral features and the sample spectral features are converted into waveforms through the residual block to obtain corresponding predicted speech and real speech.

7. The method for voice conversion based on sentiment analysis according to claim 1, characterized in that: The step of calculating the loss according to the predicted acoustic feature, the acoustic feature sample, the predicted speech and the real speech according to the preset loss function comprises: Acquire a speech discriminator corresponding to the speech generator, and input the predicted speech and the real speech into the speech discriminator to calculate the adversarial loss; Calculating a spectral loss between the predicted acoustic feature and the acoustic feature sample; Calculate L1 loss based on the predicted speech and the real speech; The adversarial loss, the spectrum loss and the L1 loss are weightedly summed to obtain the final loss.

8. A speech conversion device based on sentiment analysis, characterized in that: include: An acquisition module, used to acquire a sample data set, wherein the sample data set includes a large number of paired data of text data samples and corresponding acoustic feature samples; A preprocessing module, used for preprocessing the text data sample to obtain a standard text data set; An input module, used for inputting the standard text dataset into a pre-built conversion synthesis model, wherein the conversion synthesis model includes a sentiment analysis structure, an acoustic structure and a speech generator; A sentiment analysis module, used to perform sentiment classification and sentiment embedding on the standard text data set through the sentiment analysis structure to obtain a text sentiment embedding vector; An acoustic feature extraction module, used for extracting acoustic features by fusing the standard text data set and the text sentiment embedding vector with the acoustic structure to obtain predicted acoustic features; A speech prediction module, used for inputting the predicted acoustic features and the acoustic feature samples into the speech generator respectively to perform speech waveform synthesis, so as to obtain predicted speech and real speech; A loss calculation module, used to calculate the loss according to the predicted acoustic features, the acoustic feature samples, the predicted speech and the real speech according to a preset loss function; An iteration module, used for adjusting the model parameters of the conversion synthesis model based on the loss, continuing the iterative training until the iteration stop condition is met, and obtaining a final emotional speech synthesis model; The speech generation module is used to obtain the text to be converted, input the text to be converted into the emotional speech synthesis model, and synthesize the target speech.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the speech conversion method based on sentiment analysis as described in any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the speech conversion method based on sentiment analysis as described in any one of claims 1 to 7 are implemented.

Images