Method, device and equipment for generating video-driven digital human expression model

Abstract

The present application provides a method, device and equipment for generating a video-driven digital human expression model, the device comprising: obtaining sample data, the sample data including sample digital human videos and sample user videos; encoding sample digital human images and user expression features based on the video-driven digital human expression model to be trained to obtain sample encoding features; convolving the sample encoding features based on the video-driven digital human expression model to be trained to obtain sample convolution features; super-resolving the sample convolution features based on the video-driven human expression model to be trained to obtain an output digital human image, the output digital human image including digital human expressions; calculating the output digital human image and the sample digital human image to obtain a loss function value; updating the model parameters of the video-driven digital expression model to be trained based on the loss function value. The technical solution of the present application can output a high-definition digital human expression image sequence in real time.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a method, device and equipment for generating a video-driven digital human expression model. Background Art

[0002] The study of video-driven digital human expressions is an important part of the field of natural human-computer interaction. Among them, digital humans are digital human images that are close to human images and created using digital technology. These digital humans exist in digital space and have anthropomorphic or real appearance, form and personality characteristics. Video-driven digital human expressions obtain the user's facial expressions through video, drive the digital human video through the user's facial expressions, and synthesize the digital human video with the user's facial expressions.

[0003] In the related technologies, the research on video-driven digital human expressions mainly focuses on optimizing the expression effects, for example, achieving better expression transfer effects at low resolution.

[0004] However, the above-mentioned video-driven digital human expression technology has a low resolution, so the clarity of the digital human expression in the synthesized digital human video is low. Summary of the invention

[0005] In order to solve the above technical problems, the embodiments of the present application provide a method, device and equipment for generating a video-driven digital human expression model, which can improve the digital human expression in the digital human video. The technical solution is as follows:

[0006] According to a first aspect of an embodiment of the present application, a method for generating a video-driven digital human expression model is provided, the method comprising:

[0007] Acquire sample data, the sample data including sample digital human videos and sample user videos; the sample digital human videos including sample digital human images, the sample digital human images including sample digital human expressions, and the sample user videos including user expression features;

[0008] Encoding the sample digital human image and the user's expression features based on the video-driven digital human expression model to be trained to obtain sample encoding features;

[0009] Based on the video to be trained, the digital human expression model is driven to convolve the sample encoding feature to obtain the sample convolution feature;

[0010] Based on the video to be trained, the digital human expression model is driven to perform super-resolution on the sample convolution features to obtain an output digital human image, where the output digital human image includes the digital human expression;

[0011] Calculate the output digital human image and the sample digital human image to obtain a loss function value;

[0012] The model parameters of the digital human expression model driven by the video to be trained are updated based on the loss function value.

[0013] In a possible implementation manner of the first aspect, obtaining sample data includes:

[0014] Obtaining sample digital human videos and sample user videos;

[0015] The sample digital human video is split to obtain a sample digital human image; and features are extracted from the sample user video to obtain the expression features of the sample user.

[0016] In a possible implementation of the first aspect, the driving digital human expression model based on the video to be trained encodes the sample digital human image and the user expression feature to obtain the sample encoding feature, including:

[0017] Based on the video to be trained, the digital human expression model is driven to superimpose the sample digital human image and the user's expression features to obtain a sample superimposed feature;

[0018] The digital human expression model is driven based on the video to be trained to encode the sample superimposed features to obtain the sample encoding features.

[0019] In a possible implementation of the first aspect, the driving digital human expression model based on the video to be trained to convolve the sample encoding feature to obtain the sample convolution feature includes:

[0020] Based on the video to be trained, the digital human expression model is driven to convolve the sample encoding feature to obtain at least one intermediate convolution feature;

[0021] Based on the video to be trained, the digital human expression model is driven to perform interval calculation on the at least one intermediate convolution feature to obtain the sample convolution feature.

[0022] In a possible implementation manner of the first aspect, the step of driving the digital human expression model based on the video to be trained to perform interval calculation on the at least one intermediate convolution feature to obtain the sample convolution feature includes:

[0023] Based on the video to be trained, driving the digital human expression model, matching every two intermediate convolution features in the at least one intermediate convolution feature, and obtaining an intermediate convolution feature group corresponding to every two intermediate convolution features in the at least one intermediate convolution feature; wherein each intermediate convolution feature in the at least one intermediate convolution feature corresponds to a first convolution calculation channel, and sequence values ​​of the two first convolution calculation channels corresponding to the intermediate convolution feature group are not adjacent;

[0024] Based on the video to be trained, driving the digital human expression model, merging two intermediate convolution features in each intermediate convolution feature group in at least one intermediate convolution feature group, to obtain an intermediate merged feature corresponding to each intermediate convolution feature group in at least one intermediate convolution feature group;

[0025] The digital human expression model is driven based on the video to be trained to calculate each intermediate merged feature in at least one intermediate merged feature to obtain the sample convolution feature.

[0026] In a possible implementation of the first aspect, the method of driving the digital human expression model based on the video to be trained to perform super-resolution on the sample convolution features to obtain an output digital human image includes:

[0027] Based on the video to be trained, the digital human expression model is driven to convolve the sample convolution feature to obtain at least one candidate convolution feature; each of the at least one candidate convolution feature corresponds to a second convolution calculation channel;

[0028] Based on the video to be trained, driving the digital human expression model, pooling each of the at least one candidate convolution feature, to obtain a candidate pooling feature corresponding to each of the at least one candidate convolution feature;

[0029] The digital human expression model is driven based on the video to be trained to calculate each candidate convolution feature of the at least one candidate convolution feature to obtain the output digital human image.

[0030] In a possible implementation of the first aspect, the driving the digital human expression model based on the video to be trained to calculate each of the at least one candidate convolution feature to obtain the output digital human image includes:

[0031] Based on the video to be trained, the digital human expression model is driven to calculate each candidate pooling feature in at least one candidate pooling feature and a candidate convolution feature corresponding to each candidate pooling feature in the at least one pooling feature, to obtain an attention weight corresponding to each second convolution calculation channel in at least one second convolution calculation channel;

[0032] The video-driven digital human expression model to be trained calculates the sample convolution feature based on the attention weight corresponding to each second convolution calculation channel in the at least one second convolution calculation channel to obtain the output digital human image.

[0033] According to a second aspect of an embodiment of the present application, a device for generating a video-driven digital human expression model is provided, the device comprising:

[0034] An acquisition module is used to acquire sample data, the sample data including sample digital human videos and sample user videos; the sample digital human videos include sample digital human images, the sample digital human images include sample digital human expressions, and the sample user videos include user expression features;

[0035] An encoding module, used to encode the sample digital human image and the user's expression features based on the video-driven digital human expression model to be trained, to obtain sample encoding features;

[0036] A convolution module, used to drive the digital human expression model to convolve the sample encoding features based on the video to be trained to obtain sample convolution features;

[0037] A super-resolution module is used to drive the digital human expression model to perform super-resolution on the sample convolution features based on the video to be trained, so as to obtain an output digital human image, wherein the output digital human image includes the digital human expression;

[0038] A calculation module, used to calculate the output digital human image and the sample digital human image to obtain a loss function value;

[0039] An updating module is used to update the model parameters of the digital human expression model driven by the video to be trained based on the loss function value.

[0040] In a possible implementation manner of the second aspect, the acquisition module includes:

[0041] An acquisition unit, used to acquire sample digital human videos and sample user videos;

[0042] The processing unit is used to split the sample digital human video to obtain a sample digital human image; and to extract features from the sample user video to obtain the expression features of the sample user.

[0043] In a possible implementation manner of the second aspect, the encoding module includes:

[0044] A superposition unit, used for superimposing the sample digital human image and the user's expression features based on the video to be trained to drive the digital human expression model, so as to obtain a sample superposition feature;

[0045] The encoding unit is used to drive the digital human expression model to encode the sample superimposed features based on the video to be trained to obtain the sample encoding features.

[0046] In a possible implementation manner of the second aspect, the convolution module includes:

[0047] A first convolution unit is used to drive the digital human expression model to convolve the sample encoding feature based on the video to be trained to obtain at least one intermediate convolution feature;

[0048] The interval calculation unit is used to drive the digital human expression model to perform interval calculation on the at least one intermediate convolution feature based on the video to be trained to obtain the sample convolution feature.

[0049] In a possible implementation manner of the second aspect, the interval calculation unit includes:

[0050] A matching subunit is used to drive the digital human expression model to match every two intermediate convolution features in the at least one intermediate convolution feature based on the video to be trained, so as to obtain an intermediate convolution feature group corresponding to every two intermediate convolution features in the at least one intermediate convolution feature; wherein each intermediate convolution feature in the at least one intermediate convolution feature corresponds to a first convolution calculation channel, and sequence values ​​of the two first convolution calculation channels corresponding to the intermediate convolution feature group are not adjacent;

[0051] A merging subunit, used to drive the digital human expression model to merge two intermediate convolution features in each intermediate convolution feature group in at least one intermediate convolution feature group based on the video to be trained, so as to obtain an intermediate merged feature corresponding to each intermediate convolution feature group in at least one intermediate convolution feature group;

[0052] The first calculation subunit is used to drive the digital human expression model to calculate each intermediate merged feature in the at least one intermediate merged feature based on the video to be trained to obtain the sample convolution feature.

[0053] In a possible implementation manner of the first aspect, the super-resolution module includes:

[0054] A second convolution unit is used to drive the digital human expression model to calculate each candidate pooling feature in the at least one candidate pooling feature and a candidate convolution feature corresponding to each candidate pooling feature in the at least one pooling feature based on the video to be trained, and obtain an attention weight corresponding to each second convolution calculation channel in the at least one second convolution calculation channel;

[0055] A pooling unit, used to drive the digital human expression model to be trained based on the video to perform pooling on each of the at least one candidate convolutional feature, to obtain a candidate pooling feature corresponding to each of the at least one candidate convolutional feature;

[0056] The computing unit is used to drive the digital human expression model to compute each of the at least one candidate convolution feature based on the video to be trained, so as to obtain the output digital human image.

[0057] In a possible implementation manner of the first aspect, the computing unit includes:

[0058] A second calculation subunit is used to drive the digital human expression model to calculate at least one candidate pooling feature and the candidate convolution feature based on the video to be trained, and obtain an attention weight corresponding to each second convolution calculation channel in at least one second convolution calculation channel;

[0059] The second calculation subunit is used for the video to drive the digital human expression model to be trained to calculate the sample convolution feature based on the attention weight corresponding to each second convolution calculation channel in the at least one second convolution calculation channel to obtain the output digital human image.

[0060] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a processor and a memory, wherein the memory is used to store at least one program, and the at least one program is loaded by the processor and executed by the method for generating a video-driven digital human expression model as provided in the first aspect above.

[0061] According to the fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, characterized in that at least one program is stored in the computer-readable storage medium, and the at least one program is loaded and executed by a processor to implement the method for generating a video-driven digital human expression model as provided in the first aspect above.

[0062] According to a fifth aspect of an embodiment of the present application, a computer program product comprising computer instructions is provided. When the computer instructions are executed on an electronic device, the electronic device implements the method for generating a video-driven digital human expression model as provided in the first aspect above.

[0063] In this application, the above names do not limit the devices or functional modules themselves. In actual implementation, these devices or functional modules may appear with other names. As long as the functions of each device or functional module are similar to those of this application, they fall within the scope of the claims of this application and their equivalent technologies.

[0064] These and other aspects of the present application will become more apparent from the following description.

[0065] In an embodiment of the present application, a method for generating a video-driven digital human expression model is provided, sample data is obtained, and the sample data includes sample digital human videos and sample user videos; the sample digital human videos include sample digital human images, the sample digital human images include sample digital human expressions, and the sample user videos include user expression features. The video-driven digital human expression model to be trained is trained using the sample data. During the training process, the video-driven digital human expression model to be trained encodes, convolves, and super-resolutions the sample data, so that the trained video-driven digital human expression model can output a high-definition digital human expression image sequence in real time to achieve a live broadcast effect, thereby improving user satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0067] Figure 1 is a schematic diagram of an implementation environment provided according to an embodiment of the present application;

[0068] Figure 2A It is a flow chart of a method for generating a video-driven digital human expression model according to an embodiment of the present application;

[0069] Figure 2B This is a first exemplary flow chart of a method for generating a video-driven digital human expression model provided in an embodiment of the present application;

[0070] Figure 2C is a second exemplary flow chart of a method for generating a video-driven digital human expression model provided in an embodiment of the present application;

[0071] Figure 2D is a third exemplary flow chart of a method for generating a video-driven digital human expression model provided in an embodiment of the present application;

[0072] Figure 2E is a fourth exemplary flow chart of a method for generating a video-driven digital human expression model provided in an embodiment of the present application;

[0073] Figure 2F is a fifth exemplary flow chart of a method for generating a video-driven digital human expression model provided in an embodiment of the present application;

[0074] Figure 2Gis a sixth exemplary flow chart of a method for generating a video-driven digital human expression model provided in an embodiment of the present application;

[0075] Figure 3 It is a structural schematic diagram of a device for generating a video-driven digital human expression model according to an embodiment of the present application;

[0076] Figure 4 is a schematic diagram of the structure of a terminal provided according to an embodiment of the present application;

[0077] Figure 5 It is a structural diagram of a server provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0078] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0079] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0080] In this application, the terms "first", "second", etc. are used to distinguish between identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there a limitation on the quantity and execution order. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms.

[0081] These terms are only used to distinguish one element from another element. For example, without departing from the scope of various examples, a first action can be referred to as a second action, and similarly, a second action can also be referred to as a first action. Both the first action and the second action can be actions, and in some cases, can be separate and different actions.

[0082] Here, at least one means one or more than one, for example, at least one action can be one action, two actions, three actions, or any other action that is an integer greater than or equal to one. And multiple means two or more than two, for example, multiple actions can be two actions, three actions, or any other action that is an integer greater than or equal to two.

[0083] It should be noted that the data (including but not limited to training data and prediction data, such as user data, terminal side data, etc.) and signals involved in this application are authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0084] Figure 1 It is a schematic diagram of an implementation environment provided according to an embodiment of the present application, and the implementation environment may include a terminal 101 and a server 102.

[0085] In the terminal 101, an audio receiving device and a video display device are provided. The video acquisition device and the video display device may be separate devices or integrated into a hardware device that integrates the video acquisition function and the video display function, such as an LED (Light Emitting Diode) screen with a video acquisition function. The terminal 101 may be a wearable device, a personal computer, a laptop, a tablet computer, a smart TV, a vehicle-mounted terminal, etc.

[0086] The server 102 may be a single server, a server cluster consisting of multiple servers, or a cloud processing center.

[0087] The terminal 101 is connected to the server 102 via a wired or wireless network.

[0088] In some embodiments, the wireless network or wired network uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to any combination of local area network (LAN), metropolitan area network (MAN), wide area network (MAN), mobile, wired or wireless network, private network or virtual private network. In some embodiments, the data exchanged through the network is represented by technology and / or format including HyperText Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technology can also be used to replace or supplement the above data communication technology.

[0089] Figure 2A FIG. 1 is a flow chart of a method for generating a video-driven digital human expression model according to an embodiment of the present application. Figure 2A As shown, in the embodiment of the present application, the method is described by taking the application on a terminal with video acquisition and video display functions, where a video acquisition device and a video display device are provided on the terminal. The method comprises the following steps:

[0090] In step 201, the terminal obtains sample data, which includes sample digital human videos and sample user videos; the sample digital human videos include sample digital human images, the sample digital human images include sample digital human expressions, and the sample user videos include user expression features.

[0091] It should be noted that the digital human involved in the embodiments of the present application is an image displayed to a third party (a user using the digital human service). For example, the digital human may be a virtual anchor.

[0092] In some embodiments, the above step 201, such as Figure 2B As shown, the following steps are included:

[0093] In step 2011: the terminal obtains sample digital human videos and sample user videos.

[0094] In some embodiments, the terminal uses a video acquisition device to obtain sample digital human videos and sample user videos (eg, 1 to 5 minutes of sample user videos).

[0095] Optionally, the association between the user and the digital human can be understood as the user being able to customize the digital human service. For example, the user customizes a virtual anchor to perform a live broadcast event such as a performance.

[0096] In step 2012: the terminal splits the sample digital human video to obtain a sample digital human image; and extracts features from the sample user video to obtain the expression features of the sample user.

[0097] In some embodiments, after obtaining the sample digital human video, the sample digital human video is beautified using a beautification tool. After obtaining the sample digital human video that meets the preset conditions, the sample digital human video is split to obtain the sample digital human image.

[0098] In some embodiments, after obtaining a sample user video, a preset video model is used to extract features from the sample user video to obtain sample user expression features. Optionally, due to the various operating speeds of video acquisition devices and the variety of user sources, a preset video model with higher generalization is used to extract features from sample user videos to improve the efficiency of feature extraction and save computing power. In turn, the trained video-driven digital human expression model can be adapted to at least one language, at least one application device, and at least one user. For example, a preset DEAC model (Data Envelopment Analysis and Cobb-Douglas production function) can be used.

[0099] In step 202, the terminal drives the digital human expression model to encode the sample digital human image and the user expression features based on the video to be trained to obtain sample encoding features.

[0100] In some implementations, the above step 202, such as Figure 2C As shown, the following steps are included:

[0101] In step 221, the terminal drives the digital human expression model based on the video to be trained to superimpose the sample digital human image and the user expression feature to obtain a sample superimposed feature.

[0102] In some embodiments, in order to fuse the sample digital human image and the user's expression features, the terminal drives the digital human expression model based on the video to be trained, superimposes the sample digital human image and the user's expression features in the channel dimension, and obtains sample superimposed features.

[0103] In step 2022, the terminal drives the digital human expression model based on the video to be trained to encode the sample superimposed features to obtain the sample encoded features.

[0104] In some embodiments, the terminal drives the digital human expression model to perform convolution encoding on the sample superimposed features based on the video to be trained to obtain the sample encoding features.

[0105] In step 203, the terminal drives the digital human expression model to convolve the sample encoding features based on the video to be trained to obtain sample convolution features.

[0106] In some embodiments, in the above step 203, if Figure 2D As shown, the following steps are included:

[0107] In step 2031, the terminal drives the digital human expression model to convolve the sample encoding feature based on the video to be trained to obtain at least one intermediate convolution feature.

[0108] In some embodiments, when the terminal drives the digital human expression model to be trained based on the video to perform convolution on the sample encoding feature, the terminal obtains at least one convolution group, each convolution group in the at least one convolution group shares hyperparameters and includes at least one convolution layer. Each convolution group in the at least one convolution group is provided with a convolution calculation channel (for example, a first convolution calculation channel and a second convolution calculation channel), and the convolution calculation channel is used to calculate the channel of the input feature (a part of the sample encoding feature).

[0109] In step 2032, the terminal drives the digital human expression model to perform interval calculation on the at least one intermediate convolution feature based on the video to be trained to obtain the sample convolution feature.

[0110] In some embodiments, the above step 2032, such as Figure 2E As shown, the following steps are included:

[0111] In step 20321, the terminal drives the digital human expression model to be trained based on the video to match every two intermediate convolution features of the at least one intermediate convolution feature, and obtains an intermediate convolution feature group corresponding to every two intermediate convolution features of the at least one intermediate convolution feature; wherein each intermediate convolution feature of the at least one intermediate convolution feature corresponds to a first convolution calculation channel, and the sequence values ​​of the two first convolution calculation channels corresponding to the intermediate convolution feature group are not adjacent.

[0112] In some embodiments, a convolution group in at least one convolution group is used as an example for explanation. When the convolution group performs convolution on the corresponding input feature, the corresponding convolution calculation channel is calculated with the channel of the corresponding input feature to obtain the convolution result of the input feature. Optionally, a part of the sample coding feature is used as an example for explanation. When the convolution group performs convolution on a part of the sample coding feature, the first convolution calculation channel corresponding to the convolution group is calculated with the channel corresponding to the part of the sample coding feature to obtain the intermediate coding feature corresponding to the part of the sample coding feature.

[0113] Optionally, taking two convolution groups in at least one convolution group as an example for explanation, if the sequence values ​​of the first convolution calculation channels corresponding to the two convolution groups are adjacent, then when calculating the two intermediate convolution features corresponding to the two convolution groups, an overfitting problem will occur. Optionally, in order to solve the overfitting problem, an embodiment of the present application performs pairwise matching on each intermediate convolution feature in at least one intermediate convolution feature based on the sequence values ​​of the convolution calculation channels of the convolution group corresponding to each intermediate convolution feature in at least one intermediate convolution feature, to obtain at least one intermediate convolution feature group.

[0114] In some embodiments, an intermediate convolution feature group in at least one intermediate convolution feature group is taken as an example for explanation, and the intermediate convolution feature group corresponds to two convolution groups, and the two convolution groups correspond to two first convolution calculation channels. The two first convolution calculation channels are not adjacent, that is, the difference between the sequence values ​​of the two first convolution calculation channels is greater than "1". Then, when calculating the two intermediate convolution features in the intermediate convolution feature group, overfitting will not occur, thereby solving the problem of overfitting.

[0115] In some embodiments, when matching every two intermediate convolution features in the at least one intermediate convolution feature based on a preset first matching rule, the difference between the sequence values ​​of the first convolution calculation channel corresponding to every two intermediate convolution features in the at least one intermediate convolution feature is greater than "1". Optionally, two intermediate convolution feature groups in the at least one intermediate convolution feature group are used as an example for explanation, and two intermediate convolution feature groups, one of which has a sequence value of "1" for the first convolution calculation channel of the convolution group corresponding to one intermediate convolution feature and a sequence value of "6" for the first convolution calculation channel of the convolution group corresponding to the other intermediate convolution feature, are matched to obtain an intermediate convolution feature group.

[0116] In step 20322, the terminal drives the digital human expression model to be trained based on the video to merge the two intermediate convolution features in each intermediate convolution feature group in at least one intermediate convolution feature group to obtain an intermediate merged feature corresponding to each intermediate convolution feature group in at least one intermediate convolution feature group.

[0117] Through the analysis of step 20321, it can be seen that each of the at least one intermediate merged feature obtained in step 20322 does not have an overfitting problem, thereby improving the accuracy of the at least one intermediate merged feature.

[0118] In step 20323, the terminal drives the digital human expression model to calculate each intermediate merged feature in at least one intermediate merged feature based on the video to be trained to obtain the sample convolution feature.

[0119] In some embodiments, when training the video-driven digital human expression model to be trained, iterative training is required to improve the accuracy of the trained video-driven digital human expression model to be trained. Optionally, in order to make the convolution results of different convolution groups in at least one convolution group more evenly integrated, after completing step 20323, before the next round of training, every two intermediate convolution features in the at least one intermediate convolution feature can be re-matched based on a preset second matching rule to obtain an intermediate convolution feature group corresponding to every two intermediate convolution features in the at least one intermediate convolution feature.

[0120] Optionally, after completing step 20323, during the next round of training, every two intermediate convolution features in the at least one intermediate convolution feature can be re-matched based on a preset third matching rule to obtain an intermediate convolution feature group corresponding to every two intermediate convolution features in the at least one intermediate convolution feature.

[0121] Optionally, taking two intermediate convolution feature groups in at least one intermediate convolution feature group as an example for explanation, two intermediate convolution feature groups, one of which has a convolution calculation channel sequence value of "1" corresponding to one intermediate convolution feature and the other of which has a convolution calculation channel sequence value of "3" corresponding to another intermediate convolution feature, are matched to obtain an intermediate convolution feature group.

[0122] In step 20323, since each intermediate merged feature in at least one intermediate merged feature does not have overfitting, the sample convolution feature obtained based on each intermediate merged feature in at least one intermediate merged feature does not have overfitting, thereby improving the accuracy of the sample convolution feature.

[0123] In step 204, the terminal drives the digital human expression model to perform super-resolution on the sample convolution features based on the video to be trained, and obtains an output digital human image, which includes the digital human expression.

[0124] In some embodiments, in order to improve the effect and generation speed of outputting digital human images, the terminal drives the digital human expression model based on the video to be trained to perform super-resolution on the sample convolution features to obtain the output digital human image.

[0125] In some embodiments, the above step 204, such as Figure 2F As shown, including:

[0126] In step 2041, the terminal drives the digital human expression model to convolve the sample convolution feature based on the video to be trained to obtain at least one candidate convolution feature; each of the at least one candidate convolution feature corresponds to a second convolution calculation channel.

[0127] In some embodiments, the sample convolution feature is super-resolved using a preset super-resolved model. The preset super-resolved model includes a channel attention module, and the channel attention module includes a convolution layer, a pooling layer, a fully connected layer, and a mapping layer. The convolution layer includes at least one second convolution calculation channel.

[0128] Optionally, when convolving the sample convolution feature, each second convolution calculation channel in at least one second convolution calculation channel in the convolution layer corresponds to the sample convolution feature, and a candidate convolution feature corresponding to each second convolution calculation channel in at least one second convolution calculation channel is obtained, that is, at least one candidate convolution feature is obtained.

[0129] In step 2042, the terminal drives the digital human expression model to be trained based on the video to pool each of the at least one candidate convolution feature to obtain a candidate pooled feature corresponding to each of the at least one candidate convolution feature.

[0130] In some embodiments, in the process of pooling each of the at least one candidate convolutional feature based on the video to be trained to drive the digital human expression model, a mean value is calculated for each of the at least one candidate convolutional feature based on the pooling layer to obtain the mean value of each of the at least one candidate convolutional feature, that is, to obtain the candidate pooling feature corresponding to each of the at least one candidate convolutional feature. In addition, each of the at least one candidate convolutional feature is also compressed in one dimension.

[0131] In step 2043, the terminal drives the digital human expression model to calculate each candidate convolution feature of the at least one candidate convolution feature based on the video to be trained to obtain the output digital human image.

[0132] In some embodiments, the above step 2043, such as Figure 2G As shown, including:

[0133] In step 20431, the terminal drives the digital human expression model to be trained based on the video, calculates each candidate pooling feature in at least one candidate pooling feature and the candidate convolution feature corresponding to each candidate pooling feature in the at least one pooling feature, and obtains the attention weight corresponding to each second convolution calculation channel in at least one second convolution calculation channel.

[0134] In some embodiments, in the process of calculating each candidate pooling feature in at least one candidate pooling feature and the candidate convolution feature corresponding to each candidate pooling feature in the at least one pooling feature based on the video to be trained to drive the digital human expression model, the fully connected layer obtains the difference between each candidate pooling feature in at least one candidate pooling feature and the candidate convolution feature corresponding to each candidate pooling feature in the at least one pooling feature, and the fully connected layer calculates the attention weights between the two second convolution calculation channels corresponding to the differences based on these differences, and adjusts the parameters of the corresponding two second convolution calculation channels based on the attention weights to obtain two adjusted second convolution calculation channels.

[0135] In step 20432, the video-driven digital human expression model to be trained calculates the sample convolution feature based on the attention weight corresponding to each second convolution calculation channel in the at least one second convolution calculation channel to obtain the output digital human image.

[0136] In some embodiments, the corresponding channel in the sample convolution feature is calculated based on each adjusted second convolution calculation channel in the at least one adjusted second convolution calculation channel obtained in step 20431 to obtain a calculation result. Optionally, after obtaining the calculation result, the calculation result is calculated based on a nonlinear function of a mapping layer, etc. to obtain a candidate output digital human image.

[0137] Optionally, after obtaining the candidate output digital human image, the candidate output digital human image is periodically screened to obtain the output digital human image, so as to improve the resolution of the output digital human image. Optionally, after obtaining the candidate output digital human image, a partial feature map corresponding to each second convolution calculation channel in at least one second convolution calculation channel in the candidate output digital human image is obtained through a preset upsampling model, and the partial feature map corresponding to each second convolution calculation channel in at least one second convolution calculation channel is periodically screened to improve the resolution of each partial feature map in at least one partial feature map, thereby improving the resolution of the candidate output digital human image, thereby obtaining a high-resolution output digital human image.

[0138] Optionally, after obtaining the output digital human image, residual processing may be performed on the output digital human image through a preset residual module to further improve the resolution of the output digital human image and retain more detail information.

[0139] In the embodiment of the present application, by super-resolution of the sample convolution features, a 512-resolution output digital human image is obtained under the premise that the size of the sample digital human image is 256 resolution, thereby improving the clarity of the digital human expression. In addition, based on the preset super-resolution model, the amount of calculation is reduced, thereby improving the running speed of video data processing, thereby realizing high-definition and real-time video-driven digital human expression.

[0140] In step 205, the terminal calculates the output digital human image and the sample digital human image to obtain a loss function value.

[0141] In some embodiments, the terminal obtains a preset loss function, and calculates the output digital human image and the sample digital human image based on the preset loss function to obtain a loss function value. Optionally, the preset loss function may be an L1 loss function (absolute value error loss function).

[0142] In step 206, the terminal updates the model parameters of the digital human expression model driven by the video to be trained based on the loss function value.

[0143] In some embodiments, in step 201, at least one sample data may also be obtained. After obtaining at least one sample data, the above steps 202 to 206 are repeatedly executed so as to iteratively train the video-driven digital human expression model to be trained through at least one sample data, thereby improving the accuracy of the trained video-driven digital human expression model.

[0144] In some embodiments, during the iterative training of the video-driven digital human expression model to be trained, the learning rate is reduced, for example, the learning rate is set to 1 / 2 of the original value, and the number of iterations is 10.

[0145] Optionally, in order to make the output digital human image smoother, the first difference between two frames of output digital human images and the second difference between the corresponding two frames of sample digital human images are calculated, the loss function value of the first difference and the second difference is calculated, and the model parameters of the video-driven digital human expression model to be trained are updated based on the loss function value.

[0146] In some embodiments, the digital human expression model may be iteratively trained automatically based on the cloud using the video to be trained.

[0147] In some embodiments, in order to better fit the expression and details of the facial area, the weight of the loss function value of the facial area is increased to obtain a higher-definition output digital human image. For example, the weight is set to "10".

[0148] In some embodiments, before step 201, the video-driven digital human expression model to be trained is pre-trained for at least one round based on big data, and the sample user video of each user is adjusted in a targeted manner (according to actual needs) to ensure the cost and speed of training, while ensuring that the trained video-driven digital human expression model has high generalization.

[0149] In some embodiments, after obtaining a trained video-driven digital human expression model, a digital human video (for example, a speaking video of the digital human) is obtained, the digital human video is split into a digital human image sequence, and then a user video is obtained through a video acquisition device, and features are extracted from the user video to obtain user features. The user features and the digital human image sequence are input into the trained video-driven digital human expression model to obtain a target digital human image sequence, and the target digital human image sequence is displayed in real time on a video display device, so that the user can watch the digital human's performance in real time, etc., thereby improving user satisfaction.

[0150] Optionally, when obtaining user video through a video capture device, the user can make expressions or speak to the video capture device, and the corresponding expressions and lip shapes will be transferred to the digital human through the trained video-driven digital human expression model, that is, the expression-driven frames are obtained in real time, so that the digital human's expressions and lip shapes are consistent with the user, thereby achieving the effect of live broadcast.

[0151] It should be noted that the above embodiments of the present application are applied to a terminal as an example. In addition, the embodiments of the present application can also be applied to a server.

[0152] In an embodiment of the present application, sample data is obtained, and the sample data includes sample digital human videos and sample user videos; the sample digital human videos include sample digital human images, the sample digital human images include sample digital human expressions, and the sample user videos include user expression features. The video-driven digital human expression model to be trained is trained using the sample data. During the training process, the video-driven digital human expression model to be trained encodes, convolves, and super-resolutions the sample data, so that the trained video-driven digital human expression model can output a high-definition digital human expression image sequence in real time to achieve a live broadcast effect, thereby improving user satisfaction.

[0153] Figure 3 A video-driven digital human expression model generation device 300 is provided according to an embodiment of the present application, and the device includes:

[0154] The acquisition module 301 is used to acquire sample data, wherein the sample data includes sample digital human videos and sample user videos; the sample digital human videos include sample digital human images, the sample digital human images include sample digital human expressions, and the sample user videos include user expression features;

[0155] An encoding module 302 is used to encode the sample digital human image and the user's expression features based on the video-driven digital human expression model to be trained to obtain sample encoding features;

[0156] The convolution module 303 is used to drive the digital human expression model to convolve the sample encoding feature based on the video to be trained to obtain the sample convolution feature;

[0157] The super-resolution module 304 is used to drive the digital human expression model to perform super-resolution on the sample convolution features based on the video to be trained, so as to obtain an output digital human image, wherein the output digital human image includes the digital human expression;

[0158] A calculation module 305 is used to calculate the output digital human image and the sample digital human image to obtain a loss function value;

[0159] The updating module 306 is used to update the model parameters of the digital human expression model driven by the video to be trained based on the loss function value.

[0160] In a possible implementation manner of the second aspect, the obtaining module 301 includes:

[0161] An acquisition unit, used to acquire sample digital human videos and sample user videos;

[0162] The processing unit is used to split the sample digital human video to obtain a sample digital human image; and to extract features from the sample user video to obtain the expression features of the sample user.

[0163] In a possible implementation manner of the second aspect, the encoding module 302 includes:

[0164] A superposition unit, used for superimposing the sample digital human image and the user's expression features based on the video to be trained to drive the digital human expression model, so as to obtain a sample superposition feature;

[0165] The encoding unit is used to drive the digital human expression model to encode the sample superimposed features based on the video to be trained to obtain the sample encoding features.

[0166] In a possible implementation manner of the second aspect, the convolution module 303 includes:

[0167] A first convolution unit is used to drive the digital human expression model to convolve the sample encoding feature based on the video to be trained to obtain at least one intermediate convolution feature;

[0168] The interval calculation unit is used to drive the digital human expression model to perform interval calculation on the at least one intermediate convolution feature based on the video to be trained to obtain the sample convolution feature.

[0169] In a possible implementation manner of the second aspect, the interval calculation unit includes:

[0170] A matching subunit is used to drive the digital human expression model to match every two intermediate convolution features in the at least one intermediate convolution feature based on the video to be trained, so as to obtain an intermediate convolution feature group corresponding to every two intermediate convolution features in the at least one intermediate convolution feature; wherein each intermediate convolution feature in the at least one intermediate convolution feature corresponds to a first convolution calculation channel, and sequence values ​​of the two first convolution calculation channels corresponding to the intermediate convolution feature group are not adjacent;

[0171] A merging subunit, used to drive the digital human expression model to merge two intermediate convolution features in each intermediate convolution feature group in at least one intermediate convolution feature group based on the video to be trained, so as to obtain an intermediate merged feature corresponding to each intermediate convolution feature group in at least one intermediate convolution feature group;

[0172] The first calculation subunit is used to drive the digital human expression model to calculate each intermediate merged feature in the at least one intermediate merged feature based on the video to be trained to obtain the sample convolution feature.

[0173] In a possible implementation manner of the first aspect, the super-resolution module 304 includes:

[0174] A second convolution unit is used to drive the digital human expression model to calculate each candidate pooling feature in the at least one candidate pooling feature and a candidate convolution feature corresponding to each candidate pooling feature in the at least one pooling feature based on the video to be trained, and obtain an attention weight corresponding to each second convolution calculation channel in the at least one second convolution calculation channel;

[0175] A pooling unit, used to drive the digital human expression model to be trained based on the video to perform pooling on each of the at least one candidate convolutional feature, to obtain a candidate pooling feature corresponding to each of the at least one candidate convolutional feature;

[0176] The computing unit is used to drive the digital human expression model to compute each of the at least one candidate convolution feature based on the video to be trained, so as to obtain the output digital human image.

[0177] In a possible implementation manner of the first aspect, the computing unit includes:

[0178] A second calculation subunit is used to drive the digital human expression model to calculate at least one candidate pooling feature and the candidate convolution feature based on the video to be trained, and obtain an attention weight corresponding to each second convolution calculation channel in at least one second convolution calculation channel;

[0179] The second calculation subunit is used for the video to drive the digital human expression model to be trained to calculate the sample convolution feature based on the attention weight corresponding to each second convolution calculation channel in the at least one second convolution calculation channel to obtain the output digital human image.

[0180] It should be noted that: the device for generating a video-driven digital human expression model provided in the above embodiment only uses the division of the above functional modules as an example when executing the corresponding steps. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device for generating a video-driven digital human expression model provided in the above embodiment and the method for generating a video-driven digital human expression model belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0181] In an embodiment of the present application, sample data is obtained, and the sample data includes sample digital human videos and sample user videos; the sample digital human videos include sample digital human images, the sample digital human images include sample digital human expressions, and the sample user videos include user expression features. The video-driven digital human expression model to be trained is trained using the sample data. During the training process, the video-driven digital human expression model to be trained encodes, convolves, and super-resolutions the sample data, so that the trained video-driven digital human expression model can output a high-definition digital human expression image sequence in real time to achieve a live broadcast effect, thereby improving user satisfaction.

[0182] An embodiment of the present application also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the method for generating a video-driven digital human expression model as described above is implemented.

[0183] Taking electronic devices as terminals as an example, Figure 4 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application, see Figure 4The terminal 400 may be a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer or a desktop computer. The terminal 400 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0184] Typically, the terminal 400 includes a processor 401 and a memory 402 .

[0185] The processor 401 may include one or more processing cores, such as a 4-core processor, a 4-core processor, etc. The processor 401 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 401 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 401 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 401 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0186] The memory 402 may include one or more computer-readable storage media, which may be non-transitory. The memory 402 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 402 is used to store at least one program code, which is used to be executed by the processor 401 to implement the process executed by the terminal in the method for generating a video-driven digital human expression model to be trained provided in the method embodiment of the present application.

[0187] In some embodiments, the terminal 400 may further optionally include: a peripheral device interface 403 and at least one peripheral device. The processor 401, the memory 402 and the peripheral device interface 403 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 403 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 404, a display screen 405, a camera assembly 406, an audio circuit 407 and a power supply 408.

[0188] The peripheral device interface 403 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 401 and the memory 402. In some embodiments, the processor 401, the memory 402 and the peripheral device interface 403 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 401, the memory 402 and the peripheral device interface 403 can be implemented on a separate chip or circuit board, which is not limited in the embodiments of the present application.

[0189] The radio frequency circuit 404 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 404 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 404 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. In some embodiments, the radio frequency circuit 404 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 404 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 404 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0190] The display screen 405 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 405 is a touch display screen, the display screen 405 also has the ability to collect touch signals on the surface or above the surface of the display screen 405. The touch signal can be input to the processor 401 as a control signal for processing. At this time, the display screen 405 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 405 can be one, set on the front panel of the terminal 400; in other embodiments, the display screen 405 can be at least two, respectively set on different surfaces of the terminal 400 or in a folding design; in other embodiments, the display screen 405 can be a flexible display screen, set on the curved surface or folding surface of the terminal 400. Even, the display screen 405 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 405 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0191] The camera assembly 406 is used to capture images or videos. In some embodiments, the camera assembly 406 includes a front camera and a rear camera. Typically, the front camera is disposed on the front panel of the terminal, and the rear camera is disposed on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 406 may also include a flash. The flash may be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0192] The audio circuit 407 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 401 for processing, or input them into the radio frequency circuit 404 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 400. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 401 or the radio frequency circuit 404 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 407 may also include a headphone jack.

[0193] The power supply 408 is used to power various components in the terminal 400. The power supply 408 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 408 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0194] In some embodiments, the terminal 400 further includes one or more sensors 409 . The one or more sensors 409 include, but are not limited to: an acceleration sensor 410 , a gyroscope sensor 411 , a pressure sensor 412 , an optical sensor 413 , and a proximity sensor 414 .

[0195] The acceleration sensor 410 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 400. For example, the acceleration sensor 410 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 401 can control the display screen 405 to display the user page in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 410. The acceleration sensor 410 can also be used for collecting game or user motion data.

[0196] The gyro sensor 411 can detect the body direction and rotation angle of the terminal 400, and the gyro sensor 411 can cooperate with the acceleration sensor 410 to collect the user's three-dimensional actions on the terminal 400. The processor 401 can implement the following functions based on the data collected by the gyro sensor 411: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0197] The pressure sensor 412 can be set on the side frame of the terminal 400 and / or the lower layer of the display screen 405. When the pressure sensor 412 is set on the side frame of the terminal 400, it can detect the user's holding signal of the terminal 400, and the processor 401 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 412. When the pressure sensor 412 is set on the lower layer of the display screen 405, the processor 401 controls the operability controls on the UI page according to the user's pressure operation on the display screen 405. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0198] The optical sensor 413 is used to collect the ambient light intensity. In one embodiment, the processor 401 can control the display brightness of the display screen 405 according to the ambient light intensity collected by the optical sensor 413. Specifically, when the ambient light intensity is high, the display brightness of the display screen 405 is increased; when the ambient light intensity is low, the display brightness of the display screen 405 is reduced. In another embodiment, the processor 401 can also dynamically adjust the shooting parameters of the camera assembly 406 according to the ambient light intensity collected by the optical sensor 413.

[0199] The proximity sensor 414, also called a distance sensor, is usually arranged on the front panel of the terminal 400. The proximity sensor 414 is used to collect the distance between the user and the front of the terminal 400. In one embodiment, when the proximity sensor 414 detects that the distance between the user and the front of the terminal 400 is gradually decreasing, the processor 401 controls the display screen 405 to switch from the screen-on state to the screen-off state; when the proximity sensor 414 detects that the distance between the user and the front of the terminal 400 is gradually increasing, the processor 401 controls the display screen 405 to switch from the screen-off state to the screen-on state.

[0200] Those skilled in the art will understand that Figure 4 The structure shown in the figure does not constitute a limitation on the terminal 400, and the terminal 400 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0201] Take the electronic device as a server as an example. Figure 5: is a structural diagram of a server provided in an embodiment of the present application. The server 500 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 501 and one or more memories 502, wherein the one or more memories 502 store at least one computer program, and the at least one computer program is loaded and executed by the one or more processors 501 to implement the above-mentioned background removal method. Of course, the server 500 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server 500 may also include other components for implementing device functions, which will not be described in detail here.

[0202] The embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned method for generating an image processing model. Optionally, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0203] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0204] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for generating a video-driven digital human expression model, characterized in that: include: Acquire sample data, wherein the sample data includes sample digital human videos and sample user videos; The sample digital human video includes a sample digital human image, the sample digital human image includes a sample digital human expression, and the sample user video includes a user expression feature; Encoding the sample digital human image and the user expression features based on the video-driven digital human expression model to be trained to obtain sample encoding features; Convolving the sample encoding features based on the video to be trained driving the digital human expression model to obtain at least one intermediate convolution feature; Based on the video to be trained, driving the digital human expression model, matching every two intermediate convolution features in the at least one intermediate convolution feature, obtaining an intermediate convolution feature group corresponding to every two intermediate convolution features in the at least one intermediate convolution feature; wherein each intermediate convolution feature in the at least one intermediate convolution feature corresponds to a first convolution calculation channel, and sequence values ​​of the two first convolution calculation channels corresponding to the intermediate convolution feature group are not adjacent; Based on the video to be trained, driving the digital human expression model, merging two intermediate convolution features in each intermediate convolution feature group in at least one intermediate convolution feature group, to obtain an intermediate merged feature corresponding to each intermediate convolution feature group in at least one intermediate convolution feature group; Based on the video to be trained, the digital human expression model is driven to calculate each intermediate merged feature in at least one intermediate merged feature to obtain a sample convolution feature; Based on the video to be trained, the digital human expression model is driven to perform super-resolution on the sample convolution features to obtain an output digital human image, wherein the output digital human image includes the digital human expression; Calculating the output digital human image and the sample digital human image to obtain a loss function value; The model parameters of the digital human expression model driven by the video to be trained are updated based on the loss function value.

2. The method according to claim 1, characterized in that: The obtaining of sample data comprises: Obtaining sample digital human videos and sample user videos; The sample digital human video is split to obtain sample digital human images; and features are extracted from the sample user video to obtain sample user expression features.

3. The method according to claim 1, characterized in that The step of encoding the sample digital human image and the user expression features based on the video to be trained and driving the digital human expression model to obtain sample encoding features includes: Based on the video to be trained, the digital human expression model is driven to superimpose the sample digital human image and the user expression features to obtain a sample superimposed feature; The sample superimposed features are encoded based on the digital human expression model driven by the video to be trained to obtain the sample encoded features.

4. The method according to claim 1, characterized in that The step of driving the digital human expression model based on the video to be trained to super-resolve the sample convolution features to obtain an output digital human image includes: Based on the video to be trained, the digital human expression model is driven to calculate each candidate pooling feature in at least one candidate pooling feature and the candidate convolution feature corresponding to each candidate pooling feature in at least one candidate pooling feature, and obtain an attention weight corresponding to each second convolution calculation channel in at least one second convolution calculation channel; the attention weight is used to adjust the parameters of the corresponding two second convolution calculation channels to obtain two adjusted second convolution calculation channels; When convolving the sample convolution feature, calculating a channel corresponding to each second convolution calculation channel in the at least one second convolution calculation channel in the convolution layer and the sample convolution feature, to obtain a candidate convolution feature corresponding to each second convolution calculation channel in the at least one second convolution calculation channel; Based on the video to be trained, driving the digital human expression model, pooling each candidate convolution feature in at least one candidate convolution feature, to obtain a candidate pooling feature corresponding to each candidate convolution feature in the at least one candidate convolution feature; The digital human expression model is driven based on the video to be trained to calculate each candidate convolution feature of the at least one candidate convolution feature to obtain the output digital human image.

5. The method according to claim 4, characterized in that The step of driving the digital human expression model based on the video to be trained to calculate each of the at least one candidate convolution feature to obtain the output digital human image includes: Based on the video to be trained, the digital human expression model is driven to calculate at least one candidate pooling feature and the candidate convolution feature to obtain an attention weight corresponding to each second convolution calculation channel in at least one second convolution calculation channel; The video-driven digital human expression model to be trained calculates the sample convolution features based on the attention weight corresponding to each second convolution calculation channel in the at least one second convolution calculation channel to obtain the output digital human image.

6. A device for generating a video-driven digital human expression model, characterized in that: include: An acquisition module, used to acquire sample data, wherein the sample data includes sample digital human videos and sample user videos; The sample digital human video includes a sample digital human image, the sample digital human image includes a sample digital human expression, and the sample user video includes a user expression feature; An encoding module, used to encode the sample digital human image and the user expression features based on the video-driven digital human expression model to be trained, to obtain sample encoding features; A convolution module, used to drive the digital human expression model to convolve the sample encoding feature based on the video to be trained to obtain at least one intermediate convolution feature; Based on the video to be trained, driving the digital human expression model, matching every two intermediate convolution features in the at least one intermediate convolution feature, obtaining an intermediate convolution feature group corresponding to every two intermediate convolution features in the at least one intermediate convolution feature; wherein each intermediate convolution feature in the at least one intermediate convolution feature corresponds to a first convolution calculation channel, and sequence values ​​of the two first convolution calculation channels corresponding to the intermediate convolution feature group are not adjacent; Based on the video to be trained, driving the digital human expression model, merging two intermediate convolution features in each intermediate convolution feature group in at least one intermediate convolution feature group, to obtain an intermediate merged feature corresponding to each intermediate convolution feature group in at least one intermediate convolution feature group; Based on the video to be trained, the digital human expression model is driven to calculate each intermediate merged feature in at least one intermediate merged feature to obtain a sample convolution feature; A super-resolution module, used to drive the digital human expression model based on the video to be trained to perform super-resolution on the sample convolution features to obtain an output digital human image, wherein the output digital human image includes the digital human expression; A calculation module, used for calculating the output digital human image and the sample digital human image to obtain a loss function value; An updating module is used to update the model parameters of the digital human expression model driven by the video to be trained based on the loss function value.

7. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory is used to store at least one program, and the at least one program is loaded by the processor and executes the method for generating a video-driven digital human expression model according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one program, and the at least one program is loaded and executed by the processor to implement the method for generating a video-driven digital human expression model according to any one of claims 1 to 5.

Images