Voice message generation method and device, computer equipment, and storage medium

Abstract

The present embodiment provides a method and device for generating a voice message based on facial expression recognition, a computer device, and a storage medium, which belongs to the field of artificial intelligence technology. The method for generating a voice message based on facial expression recognition includes: obtaining voice data and its corresponding facial image, performing voice recognition on the voice data to obtain a text message, and performing facial expression recognition on the facial image to obtain an expression message; inputting the text message and the expression message into a first model, and the first model obtains an answer text message based on the text message and the expression message, and finally performs voice conversion on the answer text message to obtain a corresponding answer voice message. The embodiment of the present application adds facial images to the chatbot, and by recognizing facial images, it can more accurately judge the current situation, and the first model obtains an answer text message based on the text message and the expression message, and converts the answer text message into a voice reply message, thereby improving the accuracy of the voice reply message.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method and apparatus for generating a voice message, a computer device, and a storage medium. Background Art

[0002] With the development of computer technology, communication methods such as instant messaging tools and text messaging are becoming increasingly popular. These methods not only enable communication between people, but also enable communication between people and artificial intelligence systems. For example, chatbots are artificial intelligence systems that communicate with people through these methods.

[0003] Currently, chatbots are categorized into two types: active and passive. Active interaction occurs when the bot proactively interacts with humans by sharing or recommending interesting information. Passive interaction occurs when the user initiates a conversation, and the bot understands and responds accordingly. Most chatbots currently used by users are passive. However, these passive chatbots offer limited functionality, responding only to text generated by the user's voice recognition. This single recognition method often compromises the accuracy of the chatbot's voice responses. Summary of the Invention

[0004] The main purpose of the embodiments of the present application is to propose a voice message generation method and device based on expression recognition, computer equipment, and storage medium, aiming to improve the accuracy of voice reply messages.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present application provides a method for generating a voice message based on expression recognition, the method comprising:

[0006] Get voice data and its corresponding facial image:

[0007] Performing voice recognition on the voice data to obtain a text message;

[0008] Perform expression recognition on the facial image to obtain an expression message:

[0009] The text message and the emoticon message are input into a first model, and the first model obtains a reply text message according to the text message and the emoticon message:

[0010] The answer text message is converted into voice to obtain a corresponding answer voice message.

[0011] In some embodiments, performing speech recognition on the speech data to obtain a text message includes:

[0012] Performing an integral transformation on the time domain signal of the voice data to obtain a frequency domain signal;

[0013] constructing a plane space according to the time domain signal and the frequency domain signal;

[0014] Performing a convolution operation on the speech data in the plane space by a first neural network to obtain a speech sequence and a sequence length;

[0015] Slicing the speech sequence according to the sequence length to obtain a plurality of slice sequences;

[0016] The plurality of slice sequences are subjected to text conversion by a second neural network to obtain the text message.

[0017] In some embodiments, performing expression recognition on the facial image to obtain an expression message includes:

[0018] performing self-attention screening on the facial image through a third neural network to obtain transformation parameters;

[0019] Performing a distortion transformation on the facial image according to the transformation parameters to obtain a transformed image;

[0020] Performing expression recognition on the facial image and the transformed image through a fourth neural network to obtain the expression message.

[0021] In some embodiments, the fourth neural network includes a convolutional layer, a fully connected layer, and a classifier; performing expression recognition on the facial image and the transformed image by the fourth neural network to obtain an expression message includes:

[0022] performing feature extraction on the facial image and the transformed image through the convolution layer to obtain a plurality of image feature vectors;

[0023] Splicing the plurality of image feature vectors through the fully connected layer to obtain an image splicing vector;

[0024] The image splicing vector is subjected to expression classification by the classifier to obtain the expression message.

[0025] In some embodiments, before inputting the text message and the emoticon message into the first model and the first model obtaining a reply text message based on the text message and the emoticon message, the process includes:

[0026] Obtain a message data set; wherein the message data set includes a plurality of question sample data, a plurality of preset expressions, and a plurality of answer sample data, the question sample data and the preset expressions correspond one-to-one to form a binding group, and each binding group has a mapping relationship with the answer sample data;

[0027] Performing word segmentation on the plurality of question sample data to obtain a plurality of question word segmentation data;

[0028] Performing word segmentation on the plurality of answer sample data to obtain a plurality of answer word segmentation data;

[0029] Obtaining a first original model;

[0030] The first original model is trained according to the plurality of question segmentation data, the plurality of answer segmentation data and the plurality of preset expressions to obtain the first model.

[0031] In some embodiments, the first original model includes an encoder and a decoder; and the first original model is trained based on the plurality of question segmentation data, the plurality of answer segmentation data, and the plurality of preset expressions to obtain the first model, including:

[0032] Inputting the plurality of question segmentation data and the plurality of answer segmentation data into the encoder for first encoding to obtain sample encoded data;

[0033] Inputting the plurality of preset expressions into the encoder for second encoding to obtain expression encoding data;

[0034] Splicing the sample coding data and the expression coding data to obtain sample splicing data;

[0035] Inputting the sample splicing data into the decoder for decoding to obtain sample decoded data;

[0036] Calculating a loss function of the first original model according to the sample splicing data and the sample decoded data to obtain a loss value;

[0037] The first original model is updated according to the loss value to obtain a first model.

[0038] In some embodiments, converting the reply text message into voice to obtain a corresponding reply voice message includes:

[0039] Performing voice conversion on the answer text message to obtain a preliminary voice message;

[0040] transforming the preliminary voice message to obtain a spectrogram;

[0041] extracting audio features of the spectrogram;

[0042] Decoding the audio features through a fifth neural network model to obtain audio data corresponding to each frame;

[0043] The audio data is synthesized to obtain a corresponding answer voice message.

[0044] A second aspect of the present disclosure provides a device for generating a voice message based on expression recognition, comprising:

[0045] Data acquisition module, used to obtain voice data and its corresponding facial images:

[0046] A speech recognition module, configured to perform speech recognition on the speech data to obtain a text message;

[0047] The expression recognition module is used to perform expression recognition on the facial image to obtain an expression message:

[0048] A text message acquisition module is used to input the text message and the expression message into a first model, and the first model obtains a reply text message according to the text message and the expression message:

[0049] The voice message acquisition module is used to perform voice conversion on the answer text message to obtain a corresponding answer voice message.

[0050] A third aspect of the embodiments of the present disclosure provides a computer device, comprising a memory and a processor, wherein the memory stores a program, and when the program is executed by the processor, the processor is used to execute the method described in any one of the embodiments of the first aspect of the application.

[0051] The fourth aspect of the embodiments of the present disclosure proposes a storage medium, which is a computer-readable storage medium. The storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method described in any one of the embodiments of the first aspect of the present application.

[0052] The embodiment of the present disclosure proposes a method and apparatus for generating a voice message based on expression recognition, a computer device, and a storage medium. By acquiring voice data and its corresponding facial image, the method performs voice recognition on the voice data to obtain a text message, and performs expression recognition on the facial image to obtain an expression message. The text message and the expression message are input into a first model, and the first model obtains a reply text message based on the text message and the expression message. Finally, the reply text message is voice-converted to obtain a corresponding reply voice message. The embodiment of the present application adds facial images to the chatbot. By recognizing the facial image, the current situation can be judged more accurately. The first model obtains a reply text message based on the text message and the expression message, and converts the reply text message into a voice reply message, thereby improving the accuracy of the voice reply message. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a first flow chart of a method for generating a voice message based on expression recognition provided by an embodiment of the present disclosure;

[0054] Figure 2 yes Figure 1 Flowchart of step S200 in FIG.

[0055] Figure 3 yes Figure 1 Flowchart of step S300 in FIG.

[0056] Figure 4 yes Figure 3 Flowchart of step S330 in FIG.

[0057] Figure 5 yes Figure 1 Flowchart of step S500 in FIG.

[0058] Figure 6 is a second flow chart of the method for generating a voice message based on expression recognition provided by an embodiment of the present disclosure;

[0059] Figure 7 This is a practical application flow chart of the method for generating a voice message based on expression recognition provided by an embodiment of the present disclosure;

[0060] Figure 8 This is a block diagram of the module structure of the voice message generation device based on expression recognition provided by an embodiment of the present disclosure;

[0061] Figure 9 It is a schematic diagram of the hardware structure of the computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0063] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention pertains. The terms used herein are for the purpose of describing embodiments of the present invention only and are not intended to limit the present invention.

[0065] First, let’s analyze some of the terms used in this application:

[0066] Artificial intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0067] A chatbot is a computer program that communicates via conversation or text. It can simulate human conversation and pass the Turing test. Chatbots can be used for practical purposes, such as customer service or obtaining information. Some chatbots incorporate natural language processing systems, but most simple systems simply interpret input keywords and search a database for the most appropriate response. Chatbots are part of virtual assistants (such as Google Assistant) and can be integrated into many organizations' applications, websites, and instant messaging platforms (such as Facebook Messenger). Non-assistant applications include chatrooms for entertainment purposes, research and promotion of specific products, and social bots.

[0068] Convolutional Neural Networks (CNNs) are a type of feedforward neural network with a deep structure that incorporates convolutional computations. They are one of the leading algorithms in deep learning. CNNs possess representational learning capabilities, enabling shift-invariant classification of input information based on their hierarchical structure. With the advent of deep learning theory and improvements in numerical computing equipment, CNNs have rapidly developed and are being applied to fields such as computer vision and natural language processing. CNNs are modeled after biological visual perception and are capable of both supervised and unsupervised learning. The shared convolution kernel parameters within the hidden layers and the sparsity of inter-layer connections enable CNNs to learn grid-like topology features, such as pixels and audio, with minimal computational effort, resulting in robust results and no additional feature engineering required.

[0069] A recurrent neural network (RNN) is a type of recursive neural network that takes sequence data as input and recurses in the direction of the sequence's evolution, with all nodes (recurrent units) connected in a chain-like fashion. Bidirectional RNNs (Bi-RNNs) and long short-term memory networks (LSTMs) are common RNNs. RNNs possess memory, parameter sharing, and Turing completeness, making them advantageous for learning nonlinear features of sequences. RNNs have applications in natural language processing (NLP), such as speech recognition, language modeling, and machine translation, and are also used in various time series forecasting applications. RNNs, constructed by incorporating convolutional neural networks, can handle computer vision problems involving sequence inputs.

[0070] The Gated Recurrent Unit (GRU) is a gating mechanism in recurrent neural networks. Similar to other gating mechanisms, it aims to address the vanishing / exploding gradient problem in standard RNNs while preserving long-term information in the sequence. The GRU performs just as well as the LSTM on many sequence tasks, such as speech recognition, but has fewer parameters than the LSTM, consisting of only a reset gate and an update gate.

[0071] CTC (Connectionist temporal classification): is a loss function used in sequence labeling problems, primarily for aligning input and output labels. Traditional sequence labeling algorithms require perfect alignment of input and output symbols at every moment, while CTC expands the label set and adds empty elements. After labeling the sequence with the expanded label set, all predicted sequences that can be converted to true sequences through a mapping function are correct predictions. This means that the predicted sequence can be obtained without the need for data alignment. Its objective function is to maximize the sum of the probabilities of all correct predicted sequences. A forward-backward algorithm is used to find all correct predicted sequences.

[0072] Region of interest (ROI): In machine vision and image processing, the area to be processed is outlined in the image being processed in the form of a box, circle, ellipse, irregular polygon, etc., which is called a ROI.

[0073] OpenCV: is a cross-platform computer vision and machine learning software library released under the Apache 2.0 license (open source), and can run on Linux, Windows, Android, and Mac OS operating systems. It is lightweight and efficient, consisting of a series of C functions and a small number of C++ classes. It also provides interfaces for languages such as Python, Ruby, and MATLAB, and implements many common algorithms in image processing and computer vision. OpenCV is written in C++, has C++, Python, Java, and MATLAB interfaces, and supports Windows, Linux, Android, and Mac OS. OpenCV mainly focuses on real-time vision applications and uses MMX and SSE instructions when available. It now also provides support for C#, Ch, Ruby, and GO.

[0074] VGG model (Visual Geometry Group Network): This network is related to the work at ILSVRC 2014. The main work is to prove that increasing the depth of the network can affect the final performance of the network to a certain extent. VGG has two structures, namely VGG16 and VGG19. There is no essential difference between the two, except that the network depth is different. One improvement of VGG16 over AlexNet is the use of several consecutive 3x3 convolution kernels instead of the larger convolution kernels (11x11, 7x7, 5x5) in AlexNet. For a given receptive field (the local size of the input image related to the output), using stacked small convolution kernels is better than using large convolution kernels, because multiple layers of nonlinear layers can increase the depth of the network to ensure the learning of more complex patterns, and the cost is relatively small (fewer parameters).

[0075] Embedding: Embedding is a vector representation that uses a low-dimensional vector to represent an object. This object can be a word, a product, a movie, and so on. The property of this embedding vector is that it allows objects corresponding to vectors with similar distances to have similar meanings. For example, the distance between embedding(Avengers) and embedding(Iron Man) will be very close, but the distance between embedding(Avengers) and embedding(Gone with the Wind) will be farther. Embedding is essentially a mapping from semantic space to vector space, while trying to maintain the relationship between the original samples in the semantic space in the vector space. For example, two semantically similar words will be relatively close in the vector space. Embedding can encode objects using low-dimensional vectors while preserving their meaning. It is often used in machine learning. When building machine learning models, objects are encoded as low-dimensional dense vectors and then passed to the DNN to improve efficiency.

[0076] Cross Entropy: A key concept in Shannon's information theory, it is primarily used to measure the difference between two probability distributions. The performance of a language model is typically measured using cross entropy and perplexity. Cross entropy refers to the difficulty of using the model to recognize text, or from a compression perspective, how many bits are used to encode each word on average. Perplexity refers to the average number of branches used to represent the text using the model, and its reciprocal can be considered the average probability of each word. Smoothing involves assigning a probability value to unobserved N-gram combinations to ensure that a word sequence can always obtain a probability value through the language model. Commonly used smoothing techniques include Turing estimation, deletion interpolation smoothing, Katz smoothing, and Kneser-Ney smoothing.

[0077] Jieba word segmenter: Jieba word segmenter, also known as Jieba word segmenter, is an open source word segmenter. Chinese word segmentation is a basic step in Chinese text processing and a basic module for Chinese human-computer natural language interaction. When performing Chinese natural language processing, word segmentation is usually required first. Among them, Jieba word segmenter is often used for word segmentation. Jieba word segmentation algorithm uses a prefix dictionary to achieve efficient word graph scanning, generating a directed acyclic graph (DAG) consisting of all possible word generation situations of Chinese characters in a sentence, and then uses dynamic programming to find the maximum probability path and find the maximum segmentation combination based on word frequency. For unregistered words, the HMM model based on the word-forming ability of Chinese characters is adopted, and the Viterbi algorithm is used. Jieba word segmentation supports three word segmentation modes: the first is the precise mode, which attempts to segment the sentence most accurately and is suitable for text analysis; the second is the full mode, which scans all the words in the sentence that can be formed into words. It is very fast, but cannot resolve ambiguity; the third is the search engine mode, which is based on the precise mode and further segments long words to improve the recall rate, making it suitable for search engine word segmentation.

[0078] Analyzer word segmenter: Analyzer word segmenter is a component that specializes in word segmentation. It generally consists of three parts: Character Filters, Tokenizer (segmenting words according to rules), and Token Filters. Character Filters are mainly used to process raw text, such as removing HTML and special characters; Tokenizer is used to segment words according to rules; Token Filters are used to process segmented words, including lowercase, deleting stopwords, and adding synonyms.

[0079] Encoder: Encoding is to convert the input sequence into a vector of fixed length; decoding (decoder) is to convert the previously generated fixed vector into an output sequence; the input sequence can be text, voice, image, video; the output sequence can be text or image.

[0080] Word2vec (word to vector) is a family of related models used to generate word vectors. These models are shallow, two-layer neural networks trained to reconstruct linguistic text. The network is represented by words and is required to guess the adjacent positions of the input words. Under the bag-of-words model in Word2vec, word order is unimportant. After training, the Word2vec model maps each word to a vector that represents the relationship between words. This vector serves as the hidden layer of the neural network.

[0081] Attention Mechanism: The attention mechanism enables a neural network to focus on a subset of its inputs (or features), selecting specific inputs. It can be applied to any type of input, regardless of its shape. Given limited computing power, the attention mechanism is a primary means of addressing information overload, allocating computing resources to more important tasks.

[0082] Seq2Seq is an important RNN model, also known as the Encoder-Decoder model, which can be understood as an N×M model. The model consists of two parts: the Encoder, which encodes sequence information of any length into a vector c. The Decoder, on the other hand, decodes the context vector c and outputs it as a sequence.

[0083] Short-time Fourier Transform (STFT) is only applicable to stationary signals. The whistle signal of dolphins is a non-stationary signal, and its frequency characteristics change with time. In order to capture this time-varying feature, it is necessary to perform time-frequency analysis on the signal. Commonly used methods include short-time Fourier transform, wavelet transform, and Hilbert-Huang transform.

[0084] The Mel-Frequency Cepstrum is a linear transformation of the logarithmic energy spectrum based on the nonlinear Mel scale of sound frequencies. The Mel-Frequency Cepstrum Coefficients (MFCCs) are the coefficients that make up the Mel-Frequency Cepstrum. They are derived from the cepstrum of an audio clip. The difference between the cepstrum and the Mel-Frequency Cepstrum is that the Mel-Frequency Cepstrum divides the frequency bands into equally spaced intervals on the Mel scale, which better approximates the human auditory system than the linearly spaced bands used in the normal logarithmic cepstrum. This nonlinear representation can improve the representation of sound signals in a variety of fields.

[0085] Griffin-lim: A vocoder commonly used in speech synthesis, it converts the acoustic parameters generated by the speech synthesis system into a speech waveform. This vocoder requires no training or pre-knowledge of the phase spectrum. Instead, it estimates phase information from the relationship between frames to reconstruct the speech waveform.

[0086] Softmax classifier: It is a generalization of the logistic regression classifier in the face of multiple classifications, and the output is the probability value of belonging to different categories.

[0087] With the development of computer technology, communication methods such as instant messaging tools and text messaging are becoming increasingly popular. These methods not only enable communication between people, but also between humans and artificial intelligence systems. For example, chatbots are artificial intelligence systems that communicate with humans through these methods. Currently, most chatbots users encounter are passive, but these passive chatbots offer limited functionality, responding only to text messages generated by the user's voice recognition. However, this single recognition method often compromises the accuracy of the chatbot's generated voice responses.

[0088] Based on this, the embodiments of the present disclosure provide a method and apparatus for generating a voice message based on expression recognition, a computer device, and a storage medium, which can improve the accuracy of text emotion classification.

[0089] The embodiments of the present disclosure provide a method and apparatus for generating a voice message based on expression recognition, a computer device, and a storage medium, which are specifically described through the following embodiments. First, the method for generating a voice message based on expression recognition in the embodiments of the present disclosure is described.

[0090] The method for generating a voice message based on expression recognition provided by the embodiment of the present disclosure relates to the field of artificial intelligence. The method for generating a voice message based on expression recognition provided by the embodiment of the present disclosure can be applied in a terminal, can be applied in a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer or a smart watch, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the method for generating a voice message based on expression recognition, etc., but is not limited to the above forms.

[0091] The disclosed embodiments can be used in a wide variety of general or specialized computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0092] Reference Figure 1 According to the first aspect of the embodiment of the present disclosure, the method for generating a voice message based on expression recognition includes but is not limited to steps S100 to S500.

[0093] Step S100: Acquire voice data and its corresponding facial image:

[0094] Step S200, performing speech recognition on the speech data to obtain a text message;

[0095] Step S300: Perform facial expression recognition on the facial image to obtain an expression message:

[0096] Step S400: Input the text message and the emoticon message into the first model, and the first model obtains a reply text message based on the text message and the emoticon message:

[0097] Step S500: convert the answer text message into voice to obtain a corresponding answer voice message.

[0098] In step S100 of some embodiments, the microphone captures the voice data emitted by the user, that is, the content of the user's speech to the chat robot; while capturing the voice data emitted by the user, the camera is used to capture the image of the user while speaking, specifically, the user's facial image can be captured. In actual applications, some images captured by the camera may not capture the user's facial area, or the image may not only contain the user's facial area. In this case, the images captured by the camera need to be further screened. Specifically, images that do not contain the user's facial area can be deleted. In order to further improve the accuracy of expression recognition, the area of interest of the image, such as the face area, can also be detected, where the face area is the area that needs to be focused on in the expression of the embodiment of this application.

[0099] In some embodiments, the CascadeClassifier function in the open source openCV can also be used to automatically detect all face areas in the image to achieve face detection and positioning of the image.

[0100] In step S200 of some embodiments, after the voice data sent by the user is collected, the voice data needs to be converted into text to obtain a text message.

[0101] In step S300 of some embodiments, after capturing a user's facial image, the facial image needs to be processed for expression classification. For example, it is necessary to determine which expression a particular facial image represents and generate a corresponding expression message based on the expression, such as a text vector or image vector corresponding to the expression, for the first model to generate a response text message. In this embodiment of the present application, expressions can be classified into happy, sad, angry, neutral, surprised, and afraid, among others.

[0102] In step S400 of some embodiments, a text message and an emoticon message are input into a first model, and the first model obtains a reply text message based on the text message and the emoticon message.

[0103] In step S500 of some embodiments, the answer text message is converted into voice to obtain a corresponding answer voice message. After the answer voice message is generated, the chat robot makes a corresponding voice reply to the user according to the voice message.

[0104] In some embodiments, as Figure 2 As shown, step S200 specifically includes but is not limited to steps S210 to S250.

[0105] Step S210, performing an integral transformation on the time domain signal of the speech data to obtain a frequency domain signal;

[0106] Step S220, constructing a plane space according to the time domain signal and the frequency domain signal;

[0107] Step S230, performing a convolution operation on the speech data in a plane space through the first neural network to obtain a speech sequence and a sequence length;

[0108] Step S240, slicing the speech sequence according to the sequence length to obtain multiple slice sequences;

[0109] Step S250: Perform text conversion on the plurality of slice sequences through a second neural network to obtain a text message.

[0110] In step S210 of some embodiments, the time domain signal of the voice data is integrally transformed to obtain a frequency domain signal. In the embodiment of the present application, the integral transform can use Fourier transform, where the Fourier transform converts the original difficult-to-process time domain signal into an easy-to-analyze frequency domain signal. The function of the fast Fourier transform is to transform the digital signal in the time domain into the frequency domain, and the positions with higher energy can be analyzed in the frequency domain. These positions may be the frequency bands where the sounds that need attention are located.

[0111] In step S220 of some embodiments, the time domain signal and the frequency domain signal are combined into a two-dimensional space, that is, a plane space.

[0112] In step S230 of some embodiments, a convolution operation is performed on the speech data in a planar space by a first neural network to obtain a speech sequence and a length of the sequence. The first neural network is composed of multiple CNNs and is configured to perform a convolution operation on the speech data to obtain a speech sequence and a length of the speech sequence.

[0113] In step S240 of some embodiments, the speech sequence is sliced according to the sequence length. Specifically, the speech data is modeled. During the modeling process, the speech sequence can be cut into multiple slices to obtain a slice sequence. For example, the speech sequence can be cut into N slices to obtain N slice sequences.

[0114] In step S250 of some embodiments, a plurality of slice sequences are converted into text by a second neural network to obtain a text message. Specifically, the second neural network can be an RNN network, and the RNN network is applied to a plurality of GRU units, and the N slices obtained in step S240 are used as the N inputs of the RNN, and the text message output by the RNN is obtained, thereby completing the process of converting the voice data into a text message. It should be noted that when the number of time steps is large or the time step is small, the gradient of the RNN network is more likely to decay or explode. Although clipping the gradient can deal with gradient explosion, it cannot solve the problem of gradient decay, which makes it difficult for the RNN network to capture the dependency relationship with a large time step distance in the time series in practice. Based on this, the embodiment of the present application adopts GRU units in the RNN network, which can better capture the dependency relationship with a large time step distance in the time series and control the flow of information, so as to achieve a better model training effect and make the converted text message more accurate.

[0115] In some embodiments, the first neural network and the second neural network may constitute a speech model, and the speech model may be capable of converting speech data into text messages. In order to further improve the training effect of the speech model, it is considered to optimize the speech model using a loss function, such as a CTC loss function. Specifically, the loss function is shown in Formula (1), where X represents a given speech segment, Z represents the text corresponding to X, Π is a product operation, p represents probability, p(Z|X) represents the probability of outputting X given X, and L represents the output probability of Z corresponding to X. The loss function is minimized by minimizing the product of probabilities. Specifically, the loss function can be minimized by using a strategy of removing duplicates when the same letter appears multiple times in a row and a strategy of removing spaces.

[0116]

[0117] In some embodiments, as Figure 3 As shown, step S300 specifically includes but is not limited to steps S310 to S330.

[0118] Step S310, performing self-attention screening on the facial image through a third neural network to obtain transformation parameters;

[0119] Step S320, performing a distortion transformation on the facial image according to the transformation parameters to obtain a transformed image;

[0120] Step S330: Perform expression recognition on the facial image and the transformed image through a fourth neural network to obtain an expression message.

[0121] In step S310 of some embodiments, the facial image is self-attention screened by a third neural network to obtain a transformation parameter, i.e., parameter θ. The third neural network in the embodiment of the present application refers to a self-attention network, which is composed of two convolutional layers and two fully connected layers and can locate the key areas of the face. Since different expressions have different key areas, for example, when the user is angry, the key areas of their facial expression are the eyebrows and eyes; when the user is happy, the key areas of their facial expression are the mouth; when the user is surprised, the key areas of their facial expression are the mouth and eyes, etc., the use of a self-attention network can more accurately classify facial expressions.

[0122] In step S320 of some embodiments, the facial image is warped and transformed according to the transformation parameter θ to obtain a transformed image. Specifically, the transformation parameter can be of various types. For example, if the transformation parameter is a transformation direction, and the transformation direction is a 90-degree clockwise rotation, the direction of the facial image can be transformed according to the transformation parameter; if the transformation parameter is an upward flip, the facial image can be mirror-flipped according to the transformation parameter, etc., so as to determine which areas in the facial image belong to key areas related to expression.

[0123] In step S330 of some embodiments, feature extraction processing is performed on the facial image to obtain a corresponding feature vector, and the feature vector and the transformed image obtained in step S320 are input into the fourth neural network to output the classification result of the expression. In practical applications, the VGG-19 network can be used to extract features from facial images. It should be noted that the classified message described in this application is embodied in various forms, such as output in the form of an expression image or in the form of expression text. If it is in the form of an expression image, the expression image is converted into a vector to obtain an expression message; if it is in the form of expression text, the expression text is converted into a vector to obtain an expression message.

[0124] In some embodiments, the fourth neural network includes a convolutional layer, a fully connected layer, and a classifier, such as Figure 4 As shown, step S330 specifically includes but is not limited to steps S331 to S333.

[0125] Step S331, performing feature extraction on the facial image and the transformed image through a convolutional layer to obtain multiple image feature vectors;

[0126] Step S332: splicing multiple image feature vectors through a fully connected layer to obtain an image splicing vector;

[0127] Step S333: performing expression classification on the image splicing vector through a classifier to obtain an expression message.

[0128] In step S331 of some embodiments, the facial image and the transformed image are input into a convolutional layer of a fourth neural network, and feature extraction processing is performed on the facial image through the convolutional layer to obtain multiple image feature vectors.

[0129] In step S332 of some embodiments, multiple image feature vectors are input into a fully connected layer, and the multiple image feature vectors are spliced by the fully connected layer to obtain an image splicing vector.

[0130] In step S333 of some embodiments, the image splicing vector is input to a classifier, which outputs a classification result of the expression, and an expression message is obtained based on the classification result. In practical applications, the classifier referred to in this application can be a Softmax classifier, etc.

[0131] In some embodiments, the third neural network and the fourth neural network may constitute an expression recognition model, and the expression recognition model may be capable of classifying facial expressions. In order to further improve the training effect of the expression recognition model, it is considered to use a loss function to optimize the expression recognition model, such as a cross entropy loss function, as shown in formula (2), where M is the number of categories, y ic is the true category, p icis the predicted probability that the observed sample i belongs to a category C among M categories.

[0132]

[0133] In some embodiments, a face photo, i.e., a face image mentioned in the embodiment of the present application, is subjected to feature extraction by VGG-19 to obtain an image feature vector corresponding to the face image; at the same time, the face photo is input into a self-attention network to generate a parameter θ, and T is obtained according to the parameter θ. θ (G). Where, T θ (G) is equivalent to making an affine transformation on the input, and θ is the parameter of the transformation, which is equivalent to generating a distorted transformed sample of the input face photo, that is, a transformed image, which helps the neural network find important areas related to expressions in the face. Next, feature extraction is performed on the transformed image to obtain the image feature vector corresponding to the transformed image. Finally, the image feature vector corresponding to the facial image and the image feature vector corresponding to the transformed image are input into two fully connected layers, and the fully connected layer outputs the classification result of the expression. The embodiment of the present application introduces an attention mechanism, which can locate different key areas of the face according to different expressions, so that the neural network pays attention to the areas related to expressions in the face, making the expression recognition effect more accurate.

[0134] In some embodiments, as Figure 5 As shown, step S500 specifically includes but is not limited to steps S510 to S550.

[0135] Step S510, converting the reply text message into voice to obtain a preliminary voice message;

[0136] Step S520, transforming the preliminary voice message to obtain a spectrogram;

[0137] Step S530, extracting audio features of the spectrogram;

[0138] Step S540: decoding the audio features using a fifth neural network model to obtain audio data corresponding to each frame;

[0139] Step S550: synthesize the audio data to obtain a corresponding answer voice message.

[0140] In step S510 of some embodiments, the reply text message is converted into voice to obtain a preliminary voice message. In actual applications, the voice conversion can be performed using software such as OCR text recognition.

[0141] In step S520 of some embodiments, the preliminary voice message is transformed to obtain a spectrogram. Specifically, the preliminary voice message refers to the sound signal corresponding to the reply text message. The STFT can convert the sound signal into a corresponding two-dimensional signal, thereby obtaining a spectrogram. Specifically, the STFT principle is to divide a long signal into frames, add a window, and then perform a Fourier transform (FFT) on each frame. Finally, the results of each frame are stacked along another dimension to obtain a two-dimensional signal form similar to a picture, thereby obtaining the corresponding spectrogram.

[0142] In step S530 of some embodiments, an encoder is used to extract MFCC audio features of the spectrogram.

[0143] In step S540 of some embodiments, a fifth neural network based on a self-attention mechanism is used to decode the audio features to obtain audio data corresponding to each frame. Specifically, the fifth neural network is an RNN network, specifically composed of two GRU network layers, where each GRU network layer includes 256 GRU units.

[0144] In step S550 of some embodiments, since the phase change pattern between consecutive frames needs to be considered when generating audio from the spectrum, after obtaining the audio corresponding to each frame, it is necessary to use the Griffin_lim reconstruction algorithm to fine-tune the phase change between consecutive frames, thereby generating continuous frame audio and obtaining the corresponding answer voice message. It should be noted that in the case where the phase change between consecutive frames is large, it is necessary to obtain an intermediate phase so that the phase change of the continuous frame audio is not too large, thereby affecting the effect of generating the answer voice message. In addition, the embodiments of the present application can also transform the output audio parameters such as voice intonation according to different expressions, so that the robot can make more appropriate responses.

[0145] In some embodiments, as Figure 6 As shown, before step S400, the step is further included: constructing a first model, specifically including but not limited to steps S610 to S650.

[0146] Step S610, obtaining a message data set;

[0147] Step S620, performing word segmentation on the plurality of question sample data to obtain a plurality of question word segmentation data;

[0148] Step S630, performing word segmentation on the plurality of answer sample data to obtain a plurality of answer word segmentation data;

[0149] Step S640, obtaining a first original model;

[0150] Step S650: training the first original model based on the plurality of question segmentation data, the plurality of answer segmentation data and the plurality of preset expressions to obtain a first model.

[0151] In step S610 of some embodiments, a message dataset for model training is obtained. The message dataset includes a plurality of question sample data, a plurality of preset expressions, and a plurality of answer sample data. The question sample data and the preset expressions correspond one-to-one to form a binding group, and each binding group has a mapping relationship with the answer sample data.

[0152] In step S620 of some embodiments, a Chinese word segmentation tool, Jieba or Analyzer, is used to perform word segmentation processing on a plurality of question sample data to obtain a plurality of question word segmentation data.

[0153] In step S630 of some embodiments, a Chinese word segmentation tool such as Jieba or Analyzer is used to perform word segmentation processing on the plurality of answer sample data to obtain a plurality of answer word segmentation data.

[0154] In step S640 of some embodiments, a first original model is obtained, where the first original model may specifically be a Seq2seq model that has not been trained.

[0155] In step S650 of some embodiments, the first original model is trained based on a plurality of question segmentation data, a plurality of answer segmentation data, and a plurality of preset expressions to obtain a first model.

[0156] In some embodiments, step S650 further includes but is not limited to the following steps:

[0157] Inputting a plurality of question segmentation data and a plurality of answer segmentation data into an encoder for first encoding to obtain sample encoded data;

[0158] Inputting a plurality of preset expressions into an encoder for second encoding to obtain expression encoding data;

[0159] Splicing the sample coding data and the expression coding data to obtain sample splicing data;

[0160] Input the sample splicing data into the decoder for decoding to obtain sample decoded data;

[0161] Calculate the loss function of the first original model according to the sample splicing data and the sample decoding data to obtain a loss value;

[0162] The first original model is updated according to the loss value to obtain a first model.

[0163] More specifically, a plurality of question word segmentation data and a plurality of answer word segmentation data are input into the encoder for first encoding to obtain sample encoding data. The encoder refers to word2vec, and the generated sample encoding data is a word embedding vector. At the same time, a plurality of preset expressions are input into word2vec for second encoding to obtain expression encoding data. Then, the sample encoding data and the expression encoding data are input into the Seq2seq model for training. Specifically, the sample encoding data and the expression encoding data are spliced by the Seq2seq model to obtain sample splicing data, and the sample splicing data is input into the decoder for decoding to obtain sample decoded data; based on the sample splicing data and the sample decoded data, the loss function of the first original model is calculated, such as the cross entropy loss function, to obtain the loss value; based on the loss value, the first original model is updated to obtain the first model. In order to solve the problem of information loss caused by the decoder in Seq2seq only accepting the last output of the encoder and being away from the previous output, the embodiment of the present application also uses an attention model to focus on some key positions of the problem.

[0164] In some embodiments, as Figure 7 As shown, the present application adopts multiple modules to realize the process of the voice message generation method based on expression recognition. Specifically, the modules include: a voice recognition module, an expression recognition module, a text understanding module and a voice conversion module. The specific method includes: the voice recognition module recognizes the voice information of the user speaking to the chat robot, and converts the voice information into corresponding text. At the same time, the camera obtains the image of the user when speaking, and captures the face area to obtain the face area image, and inputs the face area image into the expression recognition module, and the expression recognition module recognizes the corresponding expression. The text obtained by the voice recognition module and the expression obtained by the expression recognition module are input into the text understanding module, and the text understanding module generates a text answer based on the text and the expression. The text is input into the voice conversion module to generate a voice answer, thereby completing the process of the voice message generation method based on expression recognition.

[0165] The embodiment of the present disclosure proposes a method for generating voice messages based on expression recognition. By acquiring voice data and its corresponding facial image, the method performs voice recognition on the voice data to obtain a text message, and performs expression recognition on the facial image to obtain an expression message. The text message and the expression message are input into a first model, and the first model obtains a reply text message based on the text message and the expression message. Finally, the reply text message is voice-converted to obtain a corresponding reply voice message. The embodiment of the present application incorporates facial images into the chatbot. By recognizing the facial images, the current scenario can be more accurately judged. The first model obtains a reply text message based on the text message and the expression message, and converts the reply text message into a voice reply message, thereby improving the accuracy of the voice reply message.

[0166] The embodiment of the present disclosure also provides a device for generating a voice message based on expression recognition, such as Figure 8 As shown, the above-mentioned method for generating a voice message based on expression recognition can be implemented. The voice message generating device based on expression recognition includes: a data acquisition module 710, a speech recognition module 720, an expression recognition module 730, a text message acquisition module 740 and a voice message acquisition module 750, wherein the data acquisition module 710 is used to acquire speech data and its corresponding facial image; the speech recognition module 720 is used to perform speech recognition on the speech data to obtain a text message; the expression recognition module 730 is used to perform expression recognition on the facial image to obtain an expression message; the text message acquisition module 740 is used to input the text message and the expression message into the first model, and the first model obtains an answer text message based on the text message and the expression message; the voice message acquisition module 750 is used to perform voice conversion on the answer text message to obtain a corresponding answer voice message. The embodiment of the present application adds facial images to the chatbot. By recognizing the facial images, the current situation can be judged more accurately, and the first model obtains the answer text message based on the text message and the expression message, and converts the answer text message into a voice reply message, thereby improving the accuracy of the voice reply message.

[0167] The voice message generation device based on expression recognition in the embodiment of the present disclosure is used to execute the voice message generation method based on expression recognition in the above embodiment. Its specific processing process is the same as the voice message generation method based on expression recognition in the above embodiment, and will not be repeated here.

[0168] The present disclosure also provides a computer device, including:

[0169] at least one processor, and

[0170] a memory communicatively connected to at least one processor; wherein,

[0171] The memory stores instructions, which are executed by at least one processor so that the at least one processor implements a method as described in any one of the embodiments of the first aspect of the present application when executing the instructions.

[0172] The following combination Figure 9 The hardware structure of the computer device is described in detail. The computer device includes: a processor 810, a memory 820, an input / output interface 830, a communication interface 840 and a bus 850.

[0173] The processor 810 may be implemented as a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present disclosure.

[0174] The memory 820 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 820 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 820 and is called by the processor 810 to execute the voice message generation method based on expression recognition in the embodiments of this disclosure.

[0175] Input / output interface 830, used to implement information input and output;

[0176] Communication interface 840, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.); and

[0177] bus 850 , which transmits information between the various components of the device (e.g., processor 810 , memory 820 , input / output interface 830 , and communication interface 840 );

[0178] The processor 810 , the memory 820 , the input / output interface 830 and the communication interface 840 are connected to each other in communication within the device via a bus 850 .

[0179] An embodiment of the present disclosure further provides a storage medium, which is a computer-readable storage medium and stores computer-executable instructions. The computer-executable instructions are used to enable a computer to execute the voice message generation method based on expression recognition according to an embodiment of the present disclosure.

[0180] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0181] The embodiments described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems.

[0182] It will be understood by those skilled in the art that Figures 1 to 7 The technical solutions shown in the figures do not constitute a limitation on the embodiments of the present disclosure, and may include more or fewer steps than those shown in the figures, or a combination of certain steps, or different steps.

[0183] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0184] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0185] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0186] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0187] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0188] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0189] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0190] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0191] The preferred embodiments of the present disclosure are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present disclosure. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present disclosure should be within the scope of the present disclosure.

Claims

1. A method for generating a voice message based on facial expression recognition, characterized in that: include: Get voice data and its corresponding facial image: Performing voice recognition on the voice data to obtain a text message; performing self-attention screening on the facial image through a third neural network to obtain transformation parameters; Performing a distortion transformation on the facial image according to the transformation parameters to obtain a transformed image; Perform expression recognition on the facial image and the transformed image through a fourth neural network to obtain an expression message: The text message and the emoticon message are input into a first model, and the first model obtains a reply text message according to the text message and the emoticon message: Performing voice conversion on the answer text message to obtain a corresponding answer voice message; The fourth neural network includes a convolutional layer, a fully connected layer, and a classifier; and performing expression recognition on the facial image and the transformed image by the fourth neural network to obtain an expression message includes: performing feature extraction on the facial image and the transformed image through the convolution layer to obtain a plurality of image feature vectors; Splicing the plurality of image feature vectors through the fully connected layer to obtain an image splicing vector; The image splicing vector is subjected to expression classification by the classifier to obtain the expression message.

2. The method according to claim 1, characterized in that The performing speech recognition on the speech data to obtain a text message includes: Performing an integral transformation on the time domain signal of the voice data to obtain a frequency domain signal; constructing a plane space according to the time domain signal and the frequency domain signal; Performing a convolution operation on the speech data in the plane space by a first neural network to obtain a speech sequence and a sequence length; Slicing the speech sequence according to the sequence length to obtain a plurality of slice sequences; The plurality of slice sequences are subjected to text conversion by a second neural network to obtain the text message.

3. The method according to claim 1, characterized in that Before inputting the text message and the emoticon message into the first model and the first model obtaining a reply text message according to the text message and the emoticon message, the method includes: Obtain a message data set; wherein the message data set includes a plurality of question sample data, a plurality of preset expressions, and a plurality of answer sample data, the question sample data and the preset expressions correspond one-to-one to form a binding group, and each binding group has a mapping relationship with the answer sample data; Performing word segmentation on the plurality of question sample data to obtain a plurality of question word segmentation data; Performing word segmentation on the plurality of answer sample data to obtain a plurality of answer word segmentation data; Obtaining a first original model; The first original model is trained according to the plurality of question segmentation data, the plurality of answer segmentation data and the plurality of preset expressions to obtain the first model.

4. The method according to claim 3, characterized in that The first original model includes an encoder and a decoder; the first original model is trained according to the plurality of question segmentation data, the plurality of answer segmentation data, and the plurality of preset expressions to obtain a first model, including: Inputting the plurality of question segmentation data and the plurality of answer segmentation data into the encoder for first encoding to obtain sample encoded data; Inputting the plurality of preset expressions into the encoder for second encoding to obtain expression encoding data; Splicing the sample coding data and the expression coding data to obtain sample splicing data; Inputting the sample splicing data into the decoder for decoding to obtain sample decoded data; Calculating a loss function of the first original model according to the sample splicing data and the sample decoded data to obtain a loss value; The first original model is updated according to the loss value to obtain a first model.

5. The method according to any one of claims 1 to 4, characterized in that The voice conversion of the answer text message to obtain a corresponding answer voice message includes: Performing voice conversion on the answer text message to obtain a preliminary voice message; transforming the preliminary voice message to obtain a spectrogram; extracting audio features of the spectrogram; Decoding the audio features through a fifth neural network model to obtain audio data corresponding to each frame; The audio data is synthesized to obtain a corresponding answer voice message.

6. A voice message generation device based on expression recognition, characterized in that: include: A data acquisition module, used to obtain voice data and its corresponding facial image; A speech recognition module, configured to perform speech recognition on the speech data to obtain a text message; an expression recognition module configured to perform self-attention screening on the facial image using a third neural network to obtain transformation parameters; perform a distortion transformation on the facial image according to the transformation parameters to obtain a transformed image; and perform expression recognition on the facial image and the transformed image using a fourth neural network to obtain an expression message; A text message acquisition module, configured to input the text message and the expression message into a first model, wherein the first model obtains a reply text message according to the text message and the expression message; A voice message acquisition module, configured to convert the answer text message into voice to obtain a corresponding answer voice message; The fourth neural network includes a convolutional layer, a fully connected layer, and a classifier; and performing expression recognition on the facial image and the transformed image by the fourth neural network to obtain an expression message includes: performing feature extraction on the facial image and the transformed image through the convolution layer to obtain a plurality of image feature vectors; Splicing the plurality of image feature vectors through the fully connected layer to obtain an image splicing vector; The image splicing vector is subjected to expression classification by the classifier to obtain the expression message.

7. A computer device, characterized in that: The computer device includes a memory and a processor, wherein the memory stores a program, and when the program is executed by the processor, the processor is configured to perform: The method according to any one of claims 1 to 5.

8. A storage medium, wherein the storage medium is a computer-readable storage medium, characterized in that: The computer readable storage device contains a computer program, and when the computer program is executed by a computer, the computer is configured to perform: The method according to any one of claims 1 to 5.

Images