Facial expression data generation large model training and generation method based on emotion discrimination

Abstract

The invention provides a facial expression data generation large model training and generation method based on emotion discrimination, and the model training method comprises the steps: obtaining a training sample; the method comprises the following steps: extracting fusion features of training voices, training texts and emotion categories, inputting the fusion features into a facial data generation large model, outputting predicted facial parameters, and inputting the fusion features into the model, so that the model can utilize the features to predict facial parameters matched with the input voices, texts and emotion categories. The model is helped to generate facial expressions consistent with input data emotion, and the authenticity and accuracy of emotion expression are improved; comparing the difference between the predicted face parameter and the training sample to obtain a loss value of the large model generated by the face data, adjusting the parameter of the large model generated by the face data based on a relationship between the loss value and a preset standard, and gradually reducing the difference between the predicted face parameter and the training sample by the model by adjusting the parameter of the model, therefore, the prediction accuracy is improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of virtual image data processing, and in particular to a method for training and generating a large model based on facial expression data for emotion discrimination. Background Art

[0002] Virtual humans are created and used in the non-physical world using computer technology, and possess multiple human characteristics (such as appearance, performance, and interaction). Voice and facial movements are correlated, and different voices correspond to different facial expressions.

[0003] In related technologies, when virtual humans are performing language expressions and other behaviors, the emotional expression of the corresponding language is poorly integrated with facial expressions, and only focuses on the movement and synchronization information of the lower half of the face and lips. There is a lack of adjustment and adaptation of facial emotional expressiveness, which affects the user's emotional expression through virtual humans. Summary of the Invention

[0004] This disclosure proposes a method for training and generating a large model based on facial expression data for emotion discrimination.

[0005] The first embodiment of the present disclosure proposes a method for generating a large model training method based on facial expression data for emotion discrimination, the method comprising:

[0006] Acquire training samples, where the training samples include training speech, training text, and emotion categories;

[0007] Extracting fusion features of the training speech, the training text, and the emotion category;

[0008] Inputting the fused features into facial data to generate a large model, and outputting predicted facial parameters corresponding to the training sample;

[0009] Comparing the difference between the predicted facial parameters and the training sample to obtain a loss value for generating a large model based on the facial data;

[0010] Based on the relationship between the loss value and the preset standard, the parameters of the large model generated by the facial data are adjusted.

[0011] In the embodiment of the present disclosure, extracting the fusion features of the training speech, the training text, and the emotion category includes:

[0012] Calling a preset language model to extract text features of the training text; and calling a preset speech model to extract audio features of the training speech;

[0013] Encoding the emotion category to obtain emotion features;

[0014] The text features, the audio features and the emotional features are fused to obtain fused features.

[0015] In an embodiment of the present disclosure, the training sample further includes time information, and the feature fusion of the text feature, the audio feature, and the emotion feature to obtain a target fusion feature includes:

[0016] Splicing the text features and the audio features corresponding to the same moment to obtain an initial fusion feature;

[0017] The initial fusion feature is matched with the emotion feature to obtain the target fusion feature.

[0018] In an embodiment of the present disclosure, the audio feature includes at least one of a tone feature, a speech rate feature, and a sound intensity feature.

[0019] In the embodiment of the present disclosure, comparing the difference between the predicted facial parameters and the training sample to obtain the loss value of the facial data to generate the large model includes:

[0020] comparing the predicted text corresponding to the vector features at each moment in the predicted facial parameters with the first difference of the training text at the corresponding moment;

[0021] Inputting the predicted facial parameters into a preset emotion recognition model to obtain a preset emotion;

[0022] comparing the preset emotion with the emotion category to obtain a second difference;

[0023] The loss value is calculated based on the first difference and the second difference.

[0024] In an embodiment of the present disclosure, the training sample further includes training facial data, and the comparing the difference between the predicted facial parameters and the training sample to obtain a loss value for generating a large model from the facial data includes:

[0025] comparing the vector features of the predicted facial parameters at each moment with the vector features of the training facial data at the corresponding moment to obtain a first difference;

[0026] comparing the preset emotion with the emotion category to obtain a second difference;

[0027] The loss value is calculated based on the first difference and the second difference.

[0028] A second embodiment of the present disclosure provides a method for generating facial expression data based on emotion discrimination, the method comprising:

[0029] Get generated data;

[0030] extracting features of the generated data;

[0031] The features are input into facial data to generate a large model, and the virtual facial data is output.

[0032] In the embodiment of the present disclosure, the generated data includes audio data and / or text data.

[0033] In an embodiment of the present disclosure, extracting features of the generated data includes:

[0034] If the generated data is audio data, performing text recognition on the audio data to obtain text features; performing feature extraction on the text features to obtain text features; and performing feature extraction on the audio data to obtain audio features;

[0035] If the production data includes audio features and text features, feature extraction is performed on the text features to obtain text features; and feature extraction is performed on the audio data to obtain audio features.

[0036] The technical solutions provided in the embodiments of the present disclosure have at least the following technical effects or advantages:

[0037] The embodiment of the present disclosure provides a method for training and generating a large model for generating facial expression data based on emotion discrimination, wherein the model training method includes: obtaining training samples, wherein the training samples include training speech, training text, and emotion categories, thereby ensuring the diversity of the training sample data; extracting fusion features of the training speech, training text, and emotion categories, inputting the fusion features into the facial data generation large model, and outputting predicted facial parameters corresponding to the training samples; by inputting the fusion features into the model, the model can use these features to predict facial parameters that match the input speech, text, and emotion categories; on this basis, comparing the difference between the predicted facial parameters and the training samples to obtain a loss value of the facial data generation large model; wherein based on the relationship between the loss value and the preset standard, the parameters of the facial data generation large model are adjusted; by adjusting the parameters of the model, the model can gradually reduce the difference between its predicted facial parameters and the training samples, thereby helping the model to generate facial expressions that are consistent with the emotions of the input data, improving the authenticity and accuracy of the emotional expression, and thus improving its prediction accuracy.

[0038] Additional aspects and advantages of the present disclosure will be given in part in the description below and in part will be obvious from the description below, or will be learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Various other advantages and benefits will become apparent to those skilled in the art by reading the detailed description of the preferred embodiment below. The accompanying drawings are only for the purpose of illustrating the preferred embodiment and are not to be considered as limiting the present disclosure. In addition, the same reference numerals are used throughout the accompanying drawings to represent the same components.

[0040] In the attached figure:

[0041] Figure 1 A flowchart of a method for generating a large model training based on facial expression data for emotion discrimination provided by an embodiment of the present disclosure is shown;

[0042] Figure 2 A schematic diagram of a method for generating a large model training based on facial expression data for emotion discrimination provided by an embodiment of the present disclosure is shown;

[0043] Figure 3 A schematic diagram showing another method for generating a large model training based on facial expression data for emotion discrimination provided by an embodiment of the present disclosure is shown;

[0044] Figure 4 A flowchart of a method for generating facial expression data based on emotion discrimination provided by another embodiment of the present disclosure is shown;

[0045] Figure 5 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown;

[0046] Figure 6 A schematic diagram of a storage medium provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0047] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0048] It should be noted that, unless otherwise specified, the technical or scientific terms used in the present disclosure should have the common meanings understood by those skilled in the art to which the present disclosure belongs.

[0049] A 3D digital human is a virtual character developed based on computer and artificial intelligence technologies. Created through computer technology, it features realistic appearance, voice, movements, and expressions, and is capable of real-time interaction with users, providing a truly interactive experience. The creation of a 3D digital human requires 3D modeling, using specialized modeling software to design the human's appearance, skeletal structure, and other aspects. Motion capture technology captures the movements and facial expressions of a real actor and transfers this data to the virtual human, enabling it to simulate a variety of human movements and expressions. The application of artificial intelligence technology enables 3D digital humans to respond to natural language and recognize emotions, enabling natural and fluid interactions with users.

[0050] The facial expression data of a digital avatar is primarily expressed through facial movements. Related technologies typically capture facial data using hardware devices such as cameras and Faceware. This captured facial data is then used to drive the facial expression data of the digital avatar, achieving facial movements. However, when cameras, Faceware, and other hardware devices capture facial data, they are susceptible to environmental lighting effects. In environments with excessively strong or weak lighting, the camera's capture quality may be affected, thereby compromising the accuracy and smoothness of the digital avatar's facial movements.

[0051] Facial expression data is strongly correlated with speech, and speech is more easily acquired than facial data. Therefore, driving facial movements of digital avatars based on speech is more convenient than using facial data alone. However, existing approaches that use speech to drive facial movements focus on the movement and synchronization of the lower half of the face and lips, lacking the ability to adjust the facial expression. This, in turn, results in a poor user experience.

[0052] In view of this, the embodiment of the present disclosure proposes a method for generating a large model training based on facial expression data for emotion discrimination. Figure 1 As shown, the method for generating a large model training method based on facial expression data for emotion discrimination according to an embodiment of the present disclosure may include the following steps:

[0053] In step S101, a training sample is obtained.

[0054] Among them, the training samples include training speech, training text, and emotion categories.

[0055] Exemplarily, training samples can be obtained through open source data sets, Internet data, and data collected from own devices. The acquired data can cover voices of different age groups and genders, and the corresponding voices have clear emotional classification. Specifically, it can be the voice of the target object within a preset time period, the emotional category of the voice is sadness, and the text of the corresponding voice.

[0056] The training text can be the corresponding text extracted from the training speech. Specifically, Automatic Speech Recognition (ASR) can be used to recognize the text in the speech to obtain the training text. The emotion category can be a manual classification of the emotion of the speech, such as sadness, joy, excitement, etc.

[0057] The training samples are a certain number of speech, text, and emotion categories corresponding to different age groups and genders. After obtaining the corresponding training samples, the training samples can be divided into a training set and a test set, where the ratio of the training set to the test set can be 8:2.

[0058] In an optional embodiment, after obtaining the training samples, in order to further ensure the accuracy of the model training, the obtained training samples can also be preprocessed. The corresponding preprocessing process can be to clean the training samples and cut them into semantically complete segments, thereby improving the quality of each training sample to a certain extent and improving the accuracy of subsequent model training.

[0059] In step S102, fusion features of training speech, training text and emotion categories are extracted.

[0060] For example, to extract fused features from the training speech, training text, and emotion categories, a neural network model can be used to extract features from the training speech, training text, and emotion categories separately and then fuse them. Specifically, acoustic features can be extracted from the training speech, using a large speech model such as HuBERT (Hidden Unit BERT). These features include lexical and non-lexical features, such as language units in the speech, such as words and phrases, as well as non-lexical features such as speech attributes such as intonation, speaking rate, and voice intensity. These features can reflect the prosody and sound quality of the speech. Natural language processing techniques can be used to extract text features, such as bag-of-words models, TF-IDF, and word embeddings. These features can reflect the content and semantic information of the text. Emotional features can be extracted from the speech and text using emotion recognition techniques. Emotional features can include basic emotion categories such as joy, sadness, and anger, as well as more complex emotion categories such as happiness, frustration, and anxiety. Emotional features can be extracted using machine learning algorithms (such as support vector machines and random forests) or deep learning algorithms (such as recurrent neural networks and convolutional neural networks).

[0061] The extracted speech, text, and sentiment features are combined to form a fused feature vector containing multiple pieces of information. This method is simple and easy to implement, but it requires attention to the differences in dimensionality and scale between features, and may require normalization.

[0062] More complex feature fusion algorithms, such as principal component analysis (PCA) and linear discriminant analysis (LDA), are used to reduce the dimension of the extracted features and fuse them. These algorithms can remove redundant information between features and improve the efficiency and accuracy of feature fusion.

[0063] Use deep learning models (such as neural networks) for feature extraction and fusion. Deep learning models can automatically learn feature representations in data and fuse features from different modalities. For example, a convolutional neural network (CNN) can be used to extract speech and text image features (such as spectrograms, text images, etc.), and then a recurrent neural network (RNN) or long short-term memory network (LSTM) can be used to perform temporal modeling and fusion of these features.

[0064] In an optional embodiment, the above-mentioned step S102 can also be implemented in the following manner: calling a preset language model to extract text features of the training text; and calling a preset speech model to extract audio features of the training speech; encoding the emotion category to obtain emotion features; and performing feature fusion on text features, audio features and emotion features to obtain fusion features.

[0065] Exemplarily, a preset language model can be selected for text feature extraction: according to the application scenario and data characteristics, a suitable pre-trained language model is selected, such as BERT, GPT series, RoBERTa, etc., the training text is input into the preset language model, and text features are extracted from the output of the language model. These features may include word embeddings, sentence embeddings, context vectors, etc., which can reflect the content and semantic information of the text.

[0066] Audio feature extraction can be performed using a pre-trained speech model: Based on the characteristics of the speech data, select an appropriate pre-trained speech model, such as WaveNet, DeepSpeech, or VGGish. Input the training speech (which may have been processed by framing, windowing, and Fourier transform) into the pre-trained speech model. Extract audio features from the output of the speech model. These features can include Mel-spectrograms, MFCCs, spectral envelopes, and fundamental frequencies, reflecting the speech's rhythm, sound quality, and content.

[0067] Emotional feature encoding can numerically encode emotional categories (such as joy, sadness, anger, etc.) using methods such as one-hot encoding, label encoding, or embedding encoding. The encoded emotional features will be used as part of the fusion feature.

[0068] Among them, the training samples can also include time information, and feature fusion of text features, audio features and emotional features is performed to obtain fused features, including: splicing text features and audio features corresponding to the same moment to obtain initial fused features; matching the initial fused features with emotional features to obtain fused features.

[0069] For example, the time information corresponding to the text features and audio features can be aligned, and the text features and audio features corresponding to each moment can be spliced. The dimension of the emotional feature after encoding in the above embodiment is consistent with the dimension of the text features and audio features. The emotional feature can be directly matched with the text features and audio features to obtain a fused feature. The initial fused feature is fused with the emotional feature. This can be achieved through a variety of methods, including but not limited to:

[0070] Direct concatenation: Simply concatenates the sentiment feature vector to the end of the initial fused features. Weighted summation: Performs a weighted sum of the initial fused features and sentiment features based on their weights. The weights can be trained to optimize the expressiveness of the fused features. Attention mechanism: Uses an attention mechanism to dynamically adjust the importance of the initial fused features and sentiment features during the fusion process.

[0071] In step S103, the fused features are input into the facial data to generate a large model, and the predicted facial parameters corresponding to the training samples are output.

[0072] For example, the facial data generation model can be a deep learning model, such as a generative adversarial network (GAN) or a variational autoencoder (VAE). The prepared fusion features are input into the facial data generation model. This usually means passing the feature vector to the input layer of the model.

[0073] The model infers the input fused features, which involves a forward propagation process within the model. The fused features are transformed through the various layers of the model, ultimately generating predicted facial parameters. The predicted facial parameters are extracted from the model's output layer. These parameters may include features such as facial shape, expression, and pose, depending on the model's training objectives and data. The predicted facial parameters undergo necessary post-processing. This may include smoothing, denoising, and normalization to ensure the accuracy and usability of the output.

[0074] In step S104, the difference between the predicted facial parameters and the training samples is compared to obtain the loss value of the large model generated by the facial data.

[0075] For example, the vector data corresponding to each frame in the predicted facial parameters corresponds to an expression, a corresponding voice, and a corresponding text. Therefore, after the predicted facial parameters are obtained, the loss value of the large model generated by the facial data can be determined based on the difference between the predicted facial parameters and the training samples.

[0076] Specifically, the loss function may include mean square error (MSE), mean absolute error (MAE), cross entropy loss, etc. A first difference is determined between the predicted text corresponding to the vector features at each moment in the predicted facial parameters and the training text at the corresponding moment; the predicted facial parameters are input into a preset emotion recognition model to obtain a preset emotion; the preset emotion is compared with the emotion category to obtain a second difference; and a loss value is calculated based on the first difference and the second difference.

[0077] For example, since the predicted facial parameters directly correspond to facial features rather than text, it is first necessary to generate the text corresponding to the facial parameters based on the predicted facial parameters, and compare the text corresponding to the predicted facial parameters with the training text to generate the first difference. Specifically, for each moment, the predicted text is compared with the training text, which can be achieved by calculating text similarity (such as cosine similarity, Jaccard similarity, etc.) or edit distance (such as Levenshtein distance). Record the text difference at each moment, which can be the difference in similarity score or the value of edit distance, and use this difference as the first difference.

[0078] The predicted facial parameters are fed into a pre-set emotion recognition model to obtain the recognized emotion category. The recognized emotion category is then compared with the emotion categories in the training sample. This can be done by calculating the similarity between the two emotion categories (if the emotion categories are continuous or quantifiable) or determining whether they are identical (if the emotion categories are discrete) as a second difference.

[0079] Specifically, the preset emotion recognition model classifies the input predicted facial parameters into emotion categories, calculates the probability distribution of the input variables over all categories through layers, and calculates the error between the emotion categories in the training sample and the emotion categories identified by the emotion recognition model through cosine similarity. It is used as the second difference. The following formula is expressed:

[0080]

[0081] emotion_class_loss=1-similarity

[0082] Among them, similarity is the second difference, ground_truth is the emotion category in the training sample, and pred is the emotion category identified by the emotion recognition model.

[0083] Specifically, we can also predict emotion intensity based on the emotion categories identified by the emotion recognition model. Using a sigmoid layer, we map the emotion features to a range between 0 and 1, outputting a continuous value representing the emotional intensity of the expression, thereby obtaining a gradation of emotional intensity. By comparing the emotional intensity levels of the input and output, we calculate the Mean Squared Error (MSE) loss as the emotional level error. The higher the correlation between the input and output emotions, the smaller the loss, and vice versa.

[0084] Calculate the loss value based on the relationship between the first difference and the second difference. Specifically, it can be a weighted sum, weighted square sum, or other combination. The weights of the loss function can be adjusted according to the task requirements and data characteristics. Apply the loss function to the first difference and the second difference to calculate the final loss value.

[0085] The above loss calculation method is based on unsupervised learning, meaning there are no accurate facial parameters to estimate the loss of the predicted facial data. In this case, the loss is determined by calculating the first difference using training text and determining the second difference using known emotion categories. This improves sample availability, expands the amount of training samples, and improves the accuracy of training large models generated from facial data.

[0086] If the training sample also includes training facial data, the corresponding process of calculating the loss value can be: comparing the vector features of the predicted facial parameters at each moment with the vector features of the training facial data at the corresponding moment to obtain a first difference; comparing the preset emotions and emotion categories to obtain a second difference; based on the first difference and the second difference, calculating the loss value.

[0087] For example, when training facial data is included, the first difference can be obtained by directly comparing the training facial data with the predicted facial parameters. The implementation of the second difference is consistent with that in the above embodiment and will not be repeated here.

[0088] In step S105, based on the relationship between the loss value and the preset standard, the parameters of the large model generated by the facial data are adjusted.

[0089] Exemplarily, the backpropagation algorithm can be used to calculate the gradient of the loss function with respect to the model parameters. The gradient indicates the rate and direction at which the loss function value changes with the model parameters. The model parameters are updated based on the calculated gradient and the update rule selected by the optimizer. The update process involves moving the parameters by a small step size (learning rate) in the opposite direction of the gradient. After each parameter update, the performance of the model on the training set or validation set is evaluated. If the model performance meets or exceeds the preset standard, the training is stopped or the next step is performed (such as saving the model, performing hyperparameter adjustment, etc.). If the model performance does not meet the preset standard, the training is continued and the parameters are adjusted.

[0090] Repeat the above steps of calculating gradients, updating parameters, and checking preset criteria until the stopping criteria are met (such as reaching the preset number of training rounds or the loss value no longer significantly decreasing). During training, monitor the changes in the loss value and the model's performance on the validation set. If the model is found to be overfitting (performing well on the training set but degrading on the validation set), consider adopting strategies such as regularization and early stopping.

[0091] like Figure 2 As shown, it is a schematic diagram of feature extraction, feature fusion, prediction and loss value determination involved in the model training process of the above embodiment. The training samples include audio, text and emotion classification labels. If there is only audio, the text in the audio can be extracted as training text. The features of the audio and text are extracted separately, and the emotion categories are encoded. Next, feature fusion is performed, and the fused features are input into the facial data to generate a large model, and 4D facial parameters are output. The output 4D facial parameters are subjected to the content discrimination module to obtain the first difference, and the emotion discrimination module to obtain the second difference. Finally, the corresponding loss value is obtained based on the first difference and the second difference. As shown Figure 3 As shown, it is a flowchart of feature extraction, feature fusion, prediction and loss value determination involved in the model training process of the above embodiment.

[0092] The embodiment of the present disclosure provides a facial expression data generation large model training method based on emotion discrimination, which obtains training samples, and the training samples include training speech, training text and emotion categories, thereby ensuring the diversity of training sample data; extracts fusion features of training speech, training text and emotion categories, inputs the fusion features into the facial data generation large model, and outputs predicted facial parameters corresponding to the training samples. By inputting the fusion features into the model, the model can use these features to predict facial parameters that match the input speech, text and emotion categories, which helps the model generate facial expressions that are consistent with the emotions of the input data and improves the authenticity and accuracy of emotional expression; on this basis, the difference between the predicted facial parameters and the training samples is compared to obtain the loss value of the facial data generation large model, wherein the parameters of the facial data generation large model are adjusted based on the relationship between the loss value and the preset standard. By adjusting the parameters of the model, the model can gradually reduce the difference between its predicted facial parameters and the training samples, thereby improving its prediction accuracy.

[0093] Corresponding to the above-mentioned facial expression data generation model training method based on emotion discrimination, the present disclosure also includes a method for generating virtual facial data using the above-mentioned trained facial expression data generation model based on emotion discrimination, specifically as follows: Figure 4 As shown, the facial expression data generation method based on emotion discrimination includes the following steps:

[0094] In step S401, generation data is acquired.

[0095] In step S402 , features of the generated data are extracted.

[0096] In step S403, the features are input into the facial data to generate a large model, and virtual facial data is output.

[0097] Exemplarily, the method for generating facial expression data based on emotion discrimination in an embodiment of the present disclosure is to generate a large model using the facial data trained in the aforementioned embodiment. The generated data obtained in the embodiment of the present disclosure can be speech and emotion categories, or speech, text, and emotion categories. If the generated data is audio data, text recognition is performed on the audio data to obtain text features; feature extraction is performed on the text features to obtain text features; and feature extraction is performed on the audio data to obtain audio features; if the generated data is audio features and text features, feature extraction is performed on the text features to obtain text features; and feature extraction is performed on the audio data to obtain audio features.

[0098] For example, if there is no text, text recognition needs to be performed based on the voice to obtain the text. The implementation method of extracting features is the same as the extraction method of model training in the above embodiment, and will not be repeated here.

[0099] The present disclosure also provides an electronic device for executing the above-mentioned data transmission method. Figure 5 , which shows a schematic diagram of an electronic device provided by some embodiments of the present disclosure. Figure 5 As shown, the electronic device includes: a processor 500, a memory 501, a bus 502 and a communication interface 503, the processor 500, the communication interface 503 and the memory 501 are connected via the bus 502; the memory 501 stores a computer program that can be run on the processor 500, and the processor 500 executes the aforementioned Figure 1 or Figure 4 The method provided.

[0100] The memory 501 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage. The system network element and at least one other network element are connected via at least one communication interface 503 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0101] The bus 502 can be an ISA bus, a PCI bus or an EISA bus. The bus can be divided into an address bus, a data bus, a control bus, etc. Among them, the memory 501 is used to store programs. After receiving the execution instruction, the processor 500 executes the program. Figure 1 or Figure 4 The disclosed method may be applied to the processor 500 or implemented by the processor 500 .

[0102] The processor 500 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method may be completed by hardware integrated logic circuits or software instructions in the processor 500. The processor 500 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It may implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present disclosure. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present disclosure may be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 501 , and the processor 500 reads the information in the memory 501 and completes the steps of the above method in combination with its hardware.

[0103] The electronic device provided by the embodiment of the present disclosure and the data transmission method provided by the embodiment of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented by them.

[0104] The present disclosure also provides a computer-readable storage medium corresponding to the data transmission method provided in the above embodiment. Figure 6 The computer-readable storage medium shown is a CD 30 on which a computer program (ie, a program product) is stored. When the computer program is run by a microprocessor, it executes the data transmission method provided by any of the aforementioned embodiments.

[0105] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0106] The computer-readable storage medium provided by the above-mentioned embodiments of the present disclosure and the data transmission method provided by the embodiments of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0107] It should be noted that:

[0108] In the description provided herein, numerous specific details are described. However, it is understood that embodiments of the present disclosure can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0109] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present disclosure, various features of the present disclosure are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed approach should not be interpreted as reflecting a schematic representation that the claimed disclosure requires more features than are expressly recited in each claim. Rather, as reflected in the claims below, inventive aspects lie in less than all of the features of the individual embodiments disclosed above. Accordingly, the claims that follow the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of the present disclosure.

[0110] Furthermore, those skilled in the art will appreciate that although some embodiments herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of this disclosure and to form different embodiments. For example, in the claims below, any of the claimed embodiments may be used in any combination.

[0111] The above are merely preferred embodiments of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A method for generating a large model training method based on facial expression data for emotion discrimination, characterized in that: The method comprises: Acquire training samples, wherein the training samples include training speech, training text, and emotion categories; Extracting fusion features of the training speech, the training text, and the emotion category; Inputting the fused features into the facial data to generate a large model, and outputting the predicted facial parameters corresponding to the training sample; Comparing the difference between the predicted facial parameters and the training sample, obtaining a loss value of the facial data for generating a large model; Based on the relationship between the loss value and the preset standard, the parameters of the facial data generated large model are adjusted.

2. The method according to claim 1, characterized in that The extracting of the fusion features of the training speech, the training text and the emotion category includes: Calling a preset language model to extract text features of the training text; and calling a preset speech model to extract audio features of the training speech; Encoding the emotion category to obtain emotion features; The text features, the audio features and the sentiment features are fused to obtain fused features.

3. The method according to claim 2, characterized in that The training sample also includes time information. The text feature, the audio feature and the emotion feature are fused to obtain the fused feature, including: Concatenate the text features and the audio features corresponding to the same moment to obtain an initial fusion feature; The initial fusion feature is matched with the emotion feature to obtain the fusion feature.

4. The method according to claim 2 or 3, characterized in that: The audio feature includes at least one of a tone feature, a speech speed feature, and a sound intensity feature.

5. The method according to claim 1, characterized in that The comparing the difference between the predicted facial parameters and the training sample to obtain the loss value of the facial data to generate a large model includes: Comparing the predicted text corresponding to the vector feature at each moment in the predicted facial parameters with the first difference of the training text at the corresponding moment; Inputting the predicted facial parameters into a preset emotion recognition model to obtain a preset emotion; Comparing the preset emotion with the emotion category to obtain a second difference; The loss value is calculated based on the first difference and the second difference.

6. The method according to claim 1, characterized in that The training sample also includes training facial data, and the comparing the difference between the predicted facial parameters and the training sample to obtain the loss value of the facial data to generate a large model includes: Comparing the vector features of the predicted facial parameters at each moment with the vector features of the training facial data at the corresponding moment to obtain a first difference; Comparing the preset emotion with the emotion category to obtain a second difference; The loss value is calculated based on the first difference and the second difference.

7. A method for generating facial expression data based on emotion discrimination, characterized in that: The method comprises: Get generated data; extracting features of the generated data; The features are input into the facial data to generate a large model, and the virtual facial data is output.

8. The method according to claim 7, characterized in that The generated data includes audio data and / or text data.

9. The method according to claim 8, characterized in that The extracting the features of the generated data comprises: If the generated data is audio data, performing text recognition on the audio data to obtain text features; performing feature extraction on the text features to obtain text features; and performing feature extraction on the audio data to obtain audio features; If the production data is audio features and text features, feature extraction is performed on the text features to obtain text features; and feature extraction is performed on the audio data to obtain audio features.

Images