Virtual character mouth shape driving method and device and storage medium

Abstract

The invention discloses a mouth shape driving method and device of a virtual character and a storage medium. The method comprises the following steps: carrying out preliminary training on a deep learning model by utilizing an original data set to obtain a preliminary model, and carrying out preliminary screening on the original data set by utilizing the preliminary model to obtain a preliminary screening data set; increasing a difference value between the similarity of a positive sample and the similarity of a negative sample in the primary screening data set based on a dynamically adjusted stepping value, retraining the primary model based on the difference value to obtain a retraining model, and rescreening the primary screening data set by using the retraining model to obtain a secondary screening data set; and performing enhanced training on the retraining model by using the secondary screening data set to obtain an enhanced model, and driving the mouth shape of the virtual character by using the enhanced model. The technical problem that in the prior art, mouth shape driving of a virtual character is not vivid is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method, device and storage medium for driving the mouth shape of a virtual character. Background Art

[0002] With the development of technologies such as virtual reality and augmented reality, the application of virtual characters is becoming more and more extensive in the fields of games, film and television production, social media, etc. In order to enhance the user's immersion and interactive experience, the performance of virtual characters, especially the precise driving of lip shape, has become a research focus. However, the existing virtual character lip shape driving technology still has certain shortcomings in terms of realism and naturalness. Traditional lip shape driving methods mostly rely on predefined lip shape animations or rule models, which cannot be flexibly adjusted according to real-time input data, resulting in insufficient naturalness and accuracy in the lip shape performance of virtual characters, especially when dealing with complex situations, such as emotional expression and lip shape synchronization of fast speech.

[0003] The introduction of deep learning technology has improved the accuracy of virtual character lip shape driving to a certain extent, but the existing deep learning model does not process the similarity difference between positive and negative samples, resulting in a small similarity difference between positive and negative samples, which affects the model's accurate driving of lip shape.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present invention provide a method, device and storage medium for driving the mouth shape of a virtual character, so as to at least solve the technical problem that the mouth shape driving of the virtual character in the prior art is not realistic.

[0006] According to one aspect of an embodiment of the present invention, a method for driving the mouth shape of a virtual character is provided, comprising: using an original data set to preliminarily train a deep learning model to obtain a preliminary model, and using the preliminary model to preliminarily screen the original data set to obtain a preliminary screening data set; based on a dynamically adjusted step value, increasing the difference between the similarity of positive samples and the similarity of negative samples in the preliminary screening data set, retraining the preliminary model based on the difference to obtain a retrained model, and using the retrained model to rescreen the preliminary screening data set to obtain a secondary screening data set; using the secondary screening data set to filter the retrained model; The model is enhanced trained to obtain an enhanced model, and the enhanced model is used to drive the mouth shape of the virtual character; wherein the similarity of the positive sample is the degree of matching between the actual mouth shape data and the predicted mouth shape data generated based on the positive sample, and the similarity of the negative sample is the degree of matching between the actual mouth shape data and the predicted mouth shape data generated based on the negative sample, wherein the similarity of the positive sample is the degree of matching between the actual mouth shape data and the predicted mouth shape data generated based on the positive sample, and the similarity of the negative sample is the degree of matching between the actual mouth shape data and the predicted mouth shape data generated based on the negative sample.

[0007] According to another aspect of an embodiment of the present invention, a mouth shape driving device for a virtual character is also provided, comprising: a primary screening module, configured to perform preliminary training on a deep learning model using an original data set to obtain a preliminary model, and perform preliminary screening on the original data set using the preliminary model to obtain a primary screening data set; a secondary screening module, configured to increase the difference between the similarity of positive samples and the similarity of negative samples in the primary screening data set based on a dynamically adjusted step value, retrain the preliminary model based on the difference to obtain a retrained model, and rescreen the primary screening data set using the retrained model to obtain a secondary screening data set; a driving module, configured to perform enhanced training on the retrained model using the secondary screening data set to obtain an enhanced model, and use the enhanced model to drive the mouth shape of the virtual character; wherein the similarity of the positive sample is the degree of matching between the actual mouth shape data and the predicted mouth shape data generated based on the positive sample, and the similarity of the negative sample is the degree of matching between the actual mouth shape data and the predicted mouth shape data generated based on the negative sample.

[0008] In an embodiment of the present invention, the original data set is used to perform preliminary training on the deep learning model to obtain a preliminary model, and the preliminary model is used to perform preliminary screening on the original data set to obtain a primary screening data set; based on the dynamically adjusted step value, the difference between the similarity of the positive sample and the similarity of the negative sample in the primary screening data set is increased, and based on the difference, the preliminary model is retrained to obtain a retrained model, and the primary screening data set is rescreened using the retrained model to obtain a secondary screening data set; the retrained model is enhanced and trained using the secondary screening data set to obtain an enhanced model, and the enhanced model is used to drive the mouth shape of the virtual character. Through the above scheme, the technical problem of unrealistic mouth shape driving of virtual characters in the prior art is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0010] Figure 1 is a flow chart of a method for driving the mouth shape of a virtual character according to an embodiment of the present invention;

[0011] Figure 2 is a flow chart of another method for driving the mouth shape of a virtual character according to an embodiment of the present invention;

[0012] Figure 3 is a flow chart of a method for performing preliminary screening of an original data set using a preliminary model according to an embodiment of the present invention;

[0013] Figure 4 is a flowchart of a method for retraining a preliminary model into a retrained model according to an embodiment of the present invention;

[0014] Figure 5 is a flowchart of a method for re-screening a primary screening data set using a retraining model according to an embodiment of the present invention;

[0015] Figure 6 is a flowchart of a method for generating an enhanced model based on a two-screening data set and a retraining model according to an embodiment of the present invention;

[0016] Figure 7 is a structural diagram of a mouth shape driving device of a virtual character according to an embodiment of the present invention;

[0017] Figure 8 A schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0019] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0020] According to an embodiment of the present invention, a method embodiment of a method for driving the mouth shape of a virtual character is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0021] Figure 1 is a method for driving the mouth shape of a virtual character according to an embodiment of the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0022] Step S102, using the original data set to perform preliminary training on the deep learning model to obtain a preliminary model, and using the preliminary model to perform preliminary screening on the original data set to obtain a preliminary screening data set.

[0023] First, the original data set is used to perform preliminary training on the deep learning model to obtain a preliminary model.

[0024] Next, the original data set is preliminarily screened using the preliminary model to obtain a preliminary screening data set. Specifically, the feature vectors of the sample data in the original data set are aligned in time series and feature correlation is calculated to obtain the alignment and correlation analysis results. The present invention can effectively improve the accuracy of data screening by performing time series alignment and feature correlation calculation on the sample data in the original data set. For example, by standardizing the timestamp, the feature vectors of the sample data in the original data set are aligned in time series using the preliminary model to obtain the feature vectors after time series alignment; the feature correlation of the feature vectors after time series alignment at different time points is calculated using the Pearson correlation coefficient. By standardizing the timestamp and performing feature correlation analysis using the Pearson correlation coefficient, the present invention realizes accurate alignment and correlation evaluation of the sample data, thereby eliminating sample data with low feature correlation and ensuring the quality of the preliminary screening data set. Afterwards, based on the alignment and correlation analysis results, the sample data in the original data set whose feature correlation is less than the preset feature correlation threshold is eliminated to obtain the preliminary screening data set. In this way, not only the training efficiency of the model is improved, but also the model's ability to distinguish between positive and negative sample features is enhanced, laying a solid foundation for subsequent training.

[0025] Step S104, increasing the difference between the similarity of positive samples and the similarity of negative samples in the primary screening data set based on the dynamically adjusted step value, retraining the preliminary model based on the difference to obtain a retrained model, and re-screening the primary screening data set using the retrained model to obtain a secondary screening data set.

[0026] First, the difference between the similarity of the positive sample and the similarity of the negative sample in the preliminary screening data set is increased based on the dynamically adjusted step value. The present invention increases the difference in similarity between the positive sample and the negative sample in the preliminary screening data set by dynamically adjusting the step value, thereby significantly improving the distinguishing ability of the model. Specifically, multi-scale feature fusion processing is performed on the positive samples and negative samples in the preliminary screening data set, and based on the results of the multi-scale feature fusion processing, the similarity between the mouth shape data based on the positive sample and the negative sample and the actual mouth shape data is calculated respectively to obtain the similarity of the positive sample and the similarity of the negative sample. Afterwards, the similarity of the positive sample is compared with the similarity of the negative sample to obtain the difference between the similarity of the positive sample and the similarity of the negative sample; according to the convergence of the preliminary model during the training process, the step value is adaptively and dynamically adjusted, and the difference is increased based on the step value. This application uses multi-scale feature fusion processing to calculate the similarity between the mouth shape data of positive samples and negative samples and the actual mouth shape data, and adaptively adjusts the step value according to these similarity differences, thereby ensuring that the model performance is continuously optimized during the model training process. The dynamic adjustment mechanism of this application not only improves the recognition accuracy of positive samples, but also reduces the interference of negative samples on the model, ultimately improving the authenticity and naturalness of the virtual character's mouth shape drive.

[0027] In the present application, the similarity of the positive sample is the degree of matching between the actual mouth shape data and the predicted mouth shape data generated based on the positive sample, and the similarity of the negative sample is the degree of matching between the actual mouth shape data and the predicted mouth shape data generated based on the negative sample.

[0028] Then, the preliminary model is retrained based on the difference to obtain a retrained model. Specifically, a multi-objective loss function is determined based on the difference, the similarity of the positive sample, the similarity of the negative sample and the corresponding adaptive weight. The present invention constructs a multi-objective loss function based on the similarity difference, the positive sample similarity, the negative sample similarity and the adaptive weight, and retrains the preliminary model, thereby effectively improving the accuracy and stability of the model. Afterwards, based on the multi-objective loss function, the actual loss values ​​of the positive samples in the preliminary screening data set and the actual loss values ​​of the negative samples in the preliminary screening data set are calculated respectively; after summing the actual loss values ​​of the positive samples in the preliminary screening data set and the actual loss values ​​of the negative samples in the preliminary screening data set, the network parameters of the preliminary model are adjusted by back propagation to retrain the preliminary model. The present application realizes dynamic optimization and convergence of the model by calculating the actual loss values ​​of the positive samples and the negative samples in the preliminary screening data set, summing them and adjusting the network parameters of the model by back propagation. This application not only enhances the model's ability to accurately predict positive samples, but also reduces the misleading impact of negative samples, thereby improving the accuracy and realism of virtual character mouth shape driving.

[0029] Finally, the retraining model is used to rescreen the primary screening data set to obtain a second screening data set. For example, based on context information, the retraining model is used to perform feature expansion on the sample data in the primary screening data set to enhance the feature representation of the sample data in the primary screening data set, and obtain an expanded sample feature set; based on the expanded sample feature set, the interactive similarities of the positive samples and negative samples of the sample data in the primary screening data set in multiple dimensions are calculated respectively to obtain a multidimensional similarity matrix; based on the multidimensional similarity matrix, the sample data whose correlation of the positive samples of the sample data in the primary screening data set is less than the correlation threshold and whose correlation of the negative samples of the sample data in the primary screening data set is greater than the correlation threshold are screened out to obtain the second screening data set. The present invention rescreens the primary screening data set by retraining the model and performs feature expansion based on context information, which significantly enhances the feature representation capability of the sample data. The multidimensional interactive similarity of positive and negative samples is calculated using the expanded sample feature set, and a multidimensional similarity matrix is ​​generated, so that the sample data whose correlation of positive samples is lower than a preset threshold and whose correlation of negative samples is higher than the threshold can be more accurately screened out to obtain an optimized second screening data set. This application effectively improves the screening accuracy, reduces the interference of noise data, and thereby improves the expressiveness and accuracy of the model in virtual character mouth shape driving.

[0030] Step S106, using the second screening data set to perform enhanced training on the retrained model to obtain an enhanced model, and using the enhanced model to drive the mouth shape of the virtual character.

[0031] Figure 2 Another method for driving the mouth shape of a virtual character according to an embodiment of the present invention is as follows: Figure 2 As shown, the method comprises the following steps:

[0032] Step S202, using the original data set to perform preliminary training on the deep learning model to obtain a preliminary model.

[0033] First, receive and load the original dataset. The original dataset contains a large amount of sample data in the form of images, text, or audio. Each sample data can be decomposed into a multi-dimensional feature vector. Due to the large number of samples and diverse feature dimensions in the original dataset, direct use for model training may lead to overfitting or noise interference, so preliminary screening is required. To this end, the deep learning model is first preliminarily trained.

[0034] During the initial training process, the architecture of the deep learning model can be adjusted according to the specific application scenario. For example, if the original data set is image data, a convolutional neural network (CNN) architecture can be used; if it is text data, a Transformer-based natural language processing model can be used. The training process of the deep learning model includes multiple rounds of forward propagation and back propagation. Specifically, in the forward propagation stage, the feature vector of the input sample passes through each layer of the model, and feature extraction and mapping are performed in turn, and the predicted value is finally output. The error between the predicted value and the actual label value will be calculated by the loss function and used to update the model parameters in the back propagation stage.

[0035] Step S204, using the preliminary model to perform preliminary screening on the original data set to obtain a preliminary screening data set.

[0036] After the preliminary model training is completed, the preliminary model is used to perform preliminary screening on the original data set. The specific screening method is as follows: Figure 3 The following steps are included:

[0037] Step S2042, aligning time series.

[0038] For time series data samples, first standardize the timestamp. For example, for data collected from different sources or devices, there may be deviations or asynchrony in the timestamps. The process of standardizing the timestamps can eliminate these time offsets. Specifically, the timestamps can be standardized using interpolation methods or smoothing algorithms so that the data features within the same time period are aligned to ensure the accuracy of subsequent analysis.

[0039] Step S2044, calculating feature correlation.

[0040] After the time series are aligned, the correlation of the feature vector of each sample data is calculated. In order to improve the calculation efficiency, this embodiment uses the Pearson correlation coefficient to measure the correlation of the feature vectors at different time points. The Pearson correlation coefficient is a statistical indicator used to measure the degree of linear correlation between two variables. Its value is between -1 and 1, 1 represents a complete positive correlation, -1 represents a complete negative correlation, and 0 represents no correlation. By calculating the feature correlation of each sample data, those sample data with low correlation can be effectively identified. These sample data may contain noise or outliers, which are not conducive to the training of the model, and therefore need to be eliminated in the initial screening stage.

[0041] If the feature correlation of the sample data is less than the preset feature correlation threshold, it is considered that the sample data has low correlation with other data and is more likely to be noise or an outlier. These sample data are removed, and the remaining data constitute the initial screening data set.

[0042] Through the above process, this embodiment can effectively reduce the noise data in the original data set, improve the accuracy of data screening, and provide a more accurate sample set for subsequent model training.

[0043] Step S206, increasing the difference between the similarity of the positive samples and the similarity of the negative samples in the primary screening data set based on the dynamically adjusted step value, and retraining the preliminary model based on the difference to obtain a retrained model.

[0044] After obtaining the initial screening data set, the system further improves the recognition ability of the model. In order to ensure the distinction between positive samples and negative samples, this embodiment uses a dynamically adjusted step value to increase the similarity difference between positive and negative samples. The specific retraining process is as follows Figure 4 As shown, including the following:

[0045] Step S2062: multi-scale feature fusion processing.

[0046] In order to better calculate the similarity between positive samples and negative samples, multi-scale feature fusion is performed on the feature vector of each sample data. Multi-scale feature fusion is to extract feature information of sample data at different scales or levels and fuse this information. Taking image data as an example, multi-scale features can come from different convolutional layers, and the features of each layer represent image features at different scales. By fusing these features, the model can better capture the details and overall information of the sample data.

[0047] Step S2064, calculating the similarity difference.

[0048] For each sample data, the difference between the positive sample and the negative sample is obtained by comparing their similarity. The larger the similarity difference, the higher the distinction between the positive sample and the negative sample, which is conducive to the correct classification of the model.

[0049] Step S2066, adjusting the adaptive step value.

[0050] In order to dynamically adjust the similarity difference between positive and negative samples, this application adopts an adaptive step value adjustment mechanism. During the training process of the model, the step value is adaptively adjusted according to the convergence of the preliminary model. The step value represents the update amplitude of the model parameters at each iteration. If the similarity difference between the positive and negative samples is found to be small during the training process, the system accelerates the update of the model parameters by increasing the step value, thereby improving the recognition accuracy of the positive samples and reducing the misjudgment of the negative samples. The dynamic adjustment of the step value ensures that the model can flexibly adapt to the characteristics of the data set during the training process and gradually improve the accuracy of the classification.

[0051] Step S2068, generating a retraining model.

[0052] After the similarity difference calculation and step value adjustment are completed, the preliminary model is retrained based on this information. The retraining process is similar to the preliminary training, but this time the system uses a more optimized multi-objective loss function. The construction of the multi-objective loss function takes into account the positive sample similarity, negative sample similarity and similarity difference, and introduces adaptive weights. The role of the adaptive weight is to adjust the weight of each loss item according to the difficulty of the sample data, ensuring that the model can balance the classification accuracy of positive and negative samples during the optimization process.

[0053] Step S2069, calculate the loss value and back propagate.

[0054] During the retraining process, the actual loss values ​​of the positive and negative samples in the initial screening data set are calculated respectively. The loss value is calculated based on the multi-objective loss function, by calculating the sum of the loss values ​​of the positive and negative samples, and adjusting the network parameters of the model using the back propagation algorithm. The back propagation algorithm calculates the gradient of the loss function relative to the model parameters, updates the model parameters layer by layer, and thus gradually converges the model.

[0055] Step S208, using the retrained model to re-screen the primary screening data set to obtain a secondary screening data set.

[0056] After the retraining model is generated, the model is used to rescreen the primary screening data set to obtain the secondary screening data set. The rescreening process expands the sample feature representation and combines the context information to perform multi-dimensional similarity analysis. The specific implementation method is as follows: Figure 5 As shown, the following steps are included:

[0057] Step S2082, feature expansion.

[0058] In order to enhance the feature representation of sample data in the initial screening data set, the feature vector of each sample data is expanded based on contextual information. The contextual information can come from adjacent time points, spatial locations, or other relevant features of the sample data. For example, for time series data, the feature information of adjacent time points can be integrated into the feature vector of the current sample to enhance the richness and completeness of the feature representation. In this way, the system can better capture the potential correlation between sample data.

[0059] Step S2084, constructing a multi-dimensional similarity matrix.

[0060] After completing the feature expansion, based on the expanded sample feature set, the interactive similarity of the positive samples and negative samples in multiple dimensions is calculated respectively. Multiple dimensions can be feature representations at different levels such as time, space, and frequency. In order to calculate the similarity of samples in multiple dimensions, this application adopts a calculation method based on inner product or cosine similarity. Inner product similarity measures their similarity by calculating the dot product of two vectors, while cosine similarity measures the similarity by calculating the cosine value of the angle between the two vectors. After obtaining the similarity values ​​of positive and negative samples in multiple dimensions, a multidimensional similarity matrix is ​​constructed. Each element in the matrix represents the similarity score of the sample in a certain dimension. The rows of the matrix represent the sample features, and the columns represent the similarity dimensions. Through the similarity matrix, the similarities and differences between samples can be analyzed more intuitively, and the sample screening process can be further optimized.

[0061] Step S2086: cluster analysis of multi-dimensional similarity.

[0062] Perform cluster analysis on sample data based on a multidimensional similarity matrix. Cluster analysis is to classify samples with high similarity into the same category, thereby determining the underlying structure and pattern between the data. Clustering algorithms can use methods such as K-means, DBSCAN, or hierarchical clustering. The K-means algorithm divides samples into K categories by iteratively optimizing the centroid position; the DBSCAN algorithm is more suitable for processing noisy sample data, which determines the clustering relationship between samples based on the distribution of sample density. Through cluster analysis, outliers or noise samples can be further eliminated to generate a purer second-screen data set.

[0063] Step S2088, sample weighting and importance screening.

[0064] Based on cluster analysis, different weights are assigned to each sample data. The weight of a sample reflects its importance to model training and is usually determined by calculating the centrality measure of the sample in the cluster. For example, samples closer to the cluster center are assigned higher weights, while those farther from the cluster center are assigned lower weights. The introduction of weights can make the model pay more attention to those representative and important samples during the training process, thereby improving the training effect.

[0065] Finally, through the above re-screening process, the sample data in the primary screening data set whose positive sample correlation is less than the correlation threshold and whose negative sample correlation is greater than the correlation threshold are screened out to obtain the secondary screening data set. Compared with the primary screening data set, the secondary screening data set removes more noise samples and outliers, while retaining more representative sample data, providing more accurate data support for subsequent model training.

[0066] Step S210, generating an enhanced model based on the second screening data set and the retraining model.

[0067] After the second screening data set is generated, an enhanced model is obtained based on the second screening data set and the retraining model. The enhanced model is the final output model of the present invention, which is used to output mouth shape data in real time. The generation process of the enhanced model is as follows: Figure 6 The steps include:

[0068] Step S2102, enhance training and model optimization.

[0069] In the process of generating the enhanced model, enhanced training is performed based on the two-screen dataset. The two-screen dataset has been screened more strictly, so enhanced training can further improve the generalization ability and accuracy of the model. During the enhanced training process, the system uses improved optimization algorithms, such as adaptive learning rate optimizer (Adam), L2 regularization and other technologies to prevent overfitting and speed up the convergence of the model.

[0070] Step S2104: model parameter adjustment and accuracy verification.

[0071] During the training process of the enhanced model, the model's hyperparameters are dynamically adjusted. Hyperparameter adjustment includes optimization of parameters such as learning rate, batch size, and number of iterations. In order to ensure the performance of the enhanced model, cross-validation is performed during the training process. Cross-validation ensures that the model performs consistently on different data sets by dividing the data set into multiple subsets, which are used as training sets and validation sets in turn. Through cross-validation, the system can effectively detect the adaptability of the model on different data and avoid overfitting.

[0072] Step S212: using the enhanced model to drive the mouth shape of the virtual character.

[0073] By analyzing and processing the input voice data, the voice features are extracted. Then, the trained enhancement model is used to map these voice features into corresponding mouth shape data. The mouth shape data contains information such as the degree of mouth opening and closing, the position change of the lips, etc. Then, the mouth movement of the virtual object is driven according to the generated mouth shape data.

[0074] Through the above method, the present application enables the mouth shape of the virtual object to be highly consistent with the voice content, achieving a natural and smooth lip synchronization effect, which is suitable for scenarios such as real-time voice interaction and animation generation.

[0075] The present application also provides a virtual character's mouth shape driving device, such as Figure 7As shown, it includes: a primary screening module 72, which is configured to use the original data set to perform preliminary training on the deep learning model to obtain a preliminary model, and use the preliminary model to perform preliminary screening on the original data set to obtain a primary screening data set; a secondary screening module 74, which is configured to increase the difference between the similarity of the positive sample and the similarity of the negative sample in the primary screening data set based on a dynamically adjusted step value, retrain the preliminary model based on the difference to obtain a retrained model, and use the retrained model to rescreen the primary screening data set to obtain a secondary screening data set; a driving module 76, which is configured to use the secondary screening data set to perform enhanced training on the retrained model to obtain an enhanced model, and use the enhanced model to drive the mouth shape of the virtual character; wherein the similarity of the positive sample is the degree of matching between the actual mouth shape data and the predicted mouth shape data generated based on the positive sample, and the similarity of the negative sample is the degree of matching between the actual mouth shape data and the predicted mouth shape data generated based on the negative sample.

[0076] It should be noted that the virtual character's mouth shape driving device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the virtual character's mouth shape driving device provided in the above embodiment and the virtual character's mouth shape driving method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0077] Figure 8 FIG. 1 shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present disclosure. It should be noted that: Figure 8 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0078] like Figure 8 As shown, the electronic device includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for system operation are also stored. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0079] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0080] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for driving the mouth shape of a virtual character, characterized in that: include: Using the original data set to preliminarily train the deep learning model to obtain a preliminary model, and using the preliminary model to preliminarily screen the original data set to obtain a preliminary screening data set; Increasing the difference between the similarity of positive samples and the similarity of negative samples in the primary screening data set based on the dynamically adjusted step value, retraining the preliminary model based on the difference to obtain a retrained model, and rescreening the primary screening data set using the retrained model to obtain a secondary screening data set; Performing enhanced training on the retrained model using the second screening data set to obtain an enhanced model, and using the enhanced model to drive the mouth shape of the virtual character; Among them, the similarity of the positive sample is the matching degree between the actual mouth shape data and the predicted mouth shape data generated based on the positive sample, and the similarity of the negative sample is the matching degree between the actual mouth shape data and the predicted mouth shape data generated based on the negative sample.

2. The method according to claim 1, characterized in that The original data set is preliminarily screened using the preliminary model to obtain a preliminary screening data set, including: Performing time series alignment and feature correlation calculation on the feature vectors of the sample data in the original data set to obtain alignment and correlation analysis results; Based on the alignment and correlation analysis results, sample data in the original data set whose feature correlation is less than a preset feature correlation threshold is eliminated to obtain the preliminary screening data set.

3. The method according to claim 2, characterized in that Performing time series alignment and feature correlation calculation on the feature vectors of the sample data in the original data set includes: By standardizing the timestamp, using the preliminary model to perform time series alignment on the feature vectors of the sample data in the original data set, to obtain the time series aligned feature vectors; The Pearson correlation coefficient is used to calculate the feature correlation of the feature vectors at different time points after the time series alignment.

4. The method according to claim 1, characterized in that: Increasing the difference between the similarity of the positive samples and the similarity of the negative samples in the primary screening data set based on the dynamically adjusted step value includes: Performing multi-scale feature fusion processing on the positive samples and negative samples in the preliminary screening data set respectively, and calculating the similarity between the mouth shape data based on the positive samples and the negative samples and the actual mouth shape data based on the result of the multi-scale feature fusion processing respectively, to obtain the similarity of the positive samples and the similarity of the negative samples; Comparing the similarity of the positive sample with the similarity of the negative sample to obtain a difference between the similarity of the positive sample and the similarity of the negative sample; According to the convergence of the preliminary model during the training process, the step value is adaptively and dynamically adjusted, and the difference is increased based on the step value.

5. The method according to claim 4, characterized in that Retraining the preliminary model based on the difference includes: Determining a multi-objective loss function based on the difference, the similarity of the positive sample, the similarity of the negative sample and corresponding adaptive weights; Based on the multi-objective loss function, respectively calculating the actual loss values ​​of the positive samples in the primary screening data set and the actual loss values ​​of the negative samples in the primary screening data set; After summing the actual loss values ​​of the positive samples in the preliminary screening data set and the actual loss values ​​of the negative samples in the preliminary screening data set, the network parameters of the preliminary model are adjusted through back propagation to retrain the preliminary model.

6. The method according to claim 1, characterized in that The primary screening data set is re-screened using the retrained model to obtain a secondary screening data set, including: Based on the context information, the retrained model is used to perform feature expansion on the sample data in the primary screening data set to enhance the feature representation of the sample data in the primary screening data set, thereby obtaining an expanded sample feature set; Based on the expanded sample feature set, respectively calculating the interactive similarities of the positive samples and the negative samples of the sample data in the primary screening data set in multiple dimensions to obtain a multidimensional similarity matrix; Based on the multidimensional similarity matrix, sample data in the primary screening data set whose positive sample correlation is less than a correlation threshold and whose negative sample correlation is greater than the correlation threshold are screened out to obtain the secondary screening data set.

7. A mouth shape driving device for a virtual character, characterized in that: include: A preliminary screening module is configured to perform preliminary training on the deep learning model using the original data set to obtain a preliminary model, and perform preliminary screening on the original data set using the preliminary model to obtain a preliminary screening data set; A second screening module is configured to increase the difference between the similarity of the positive samples and the similarity of the negative samples in the primary screening data set based on the dynamically adjusted step value, retrain the preliminary model based on the difference to obtain a retrained model, and rescreen the primary screening data set using the retrained model to obtain a second screening data set; A driving module is configured to perform enhanced training on the retrained model using the second screening data set to obtain an enhanced model, and use the enhanced model to drive the mouth shape of the virtual character; Among them, the similarity of the positive sample is the matching degree between the actual mouth shape data and the predicted mouth shape data generated based on the positive sample, and the similarity of the negative sample is the matching degree between the actual mouth shape data and the predicted mouth shape data generated based on the negative sample.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 6.

9. A computer device, characterized in that: include: Memory and processor, The memory stores a computer program; The processor is used to execute the computer program stored in the memory, and when the computer program is run, the processor is enabled to execute the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images