Expression redirection training method, device, electronic device and readable storage medium

Abstract

The present application provides an expression redirection training method, device, electronic device and readable storage medium. By collecting multiple video frames of the training user, the expression base coefficients of multiple preset expression base templates relative to the video frame are calculated for each video frame, and the control parameters for controlling the expression of the virtual image are adjusted to make the expression of the virtual image consistent with the expression of the training user in the video frame. Finally, the constructed network model is trained based on the expression base coefficients and control parameters corresponding to the video frame to obtain a redirection model. In this scheme, the different expressions of the training user are first represented by the expression base template and the expression base coefficient, and various different expressions can be represented by standardized information. By adjusting the virtual image to obtain the control parameters, a redirection model that can realize expression migration and maintain the personalized characteristics of the virtual image can be trained, so as to achieve good applicability and natural expression redirection.

Description

Technical Field

[0001] The present application relates to the field of model control technology, and in particular to an expression redirection training method, device, electronic device and readable storage medium. Background Art

[0002] Expression redirection refers to transferring the user's expression to another image. For example, in a live broadcast scenario, the host's expression can be transferred to a virtual image. When transferring the user's expression to the virtual image, the commonly used processing method is generally to hope to be able to completely and accurately transfer the user's expression to the virtual image.

[0003] However, there are many styles of virtual images, such as lolita images, animal images, hero images, etc. Different types of virtual images have personalized facial features, and the proportions between various parts of the face, the size of facial features, etc. are all different. If the user's expression information is simply copied to the virtual image, due to the differences in facial features between the virtual image and the user and the differences in facial features between different types of virtual images, the expressions transferred in this complete copying method cannot be well applied to the virtual image, making the expression of the virtual image appear unnatural. Summary of the invention

[0004] The purpose of the present application includes, for example, providing an expression redirection training method, device, electronic device and readable storage medium, which can obtain a redirection model that can achieve expression migration and maintain the personalized characteristics of a virtual image.

[0005] The embodiments of the present application can be implemented as follows:

[0006] In a first aspect, the present application provides an expression redirection training method, the method comprising:

[0007] Collecting a video stream of a training user, wherein the video stream includes multiple video frames;

[0008] For each of the video frames, calculating expression base coefficients of a plurality of preset expression base templates relative to the video frame;

[0009] Adjusting control parameters for controlling the expression of the avatar so that the expression of the avatar is consistent with the expression of the training user in the video frame;

[0010] Based on the expression base coefficients and control parameters corresponding to each of the video frames, the constructed network model is trained to obtain a redirection model.

[0011] In an optional embodiment, the step of adjusting the control parameters of the controller that controls the expression of the virtual image so that the expression of the virtual image is consistent with the expression of the training user in the video frame includes:

[0012] Extracting key video frames from the multiple video frames, wherein each of the key video frames contains a target facial feature in a set state;

[0013] The control parameters of the controller controlling the expression of the virtual image are adjusted so that the expression of the virtual image is consistent with the expression of the training user in the key video frame.

[0014] In an optional embodiment, the step of adjusting the control parameters of the controller controlling the virtual image so that the expression of the virtual image is consistent with the expression of the training user in the key video frame includes:

[0015] Determining a target virtual feature in the virtual image that corresponds to the target facial feature;

[0016] Obtaining a target controller that controls the target virtual feature;

[0017] The control parameters of the target controller are adjusted so that the state of the target virtual feature in the avatar is consistent with the state of the target facial feature.

[0018] In an optional implementation, the step of training the constructed network model to obtain the redirection model based on the expression base coefficients and control parameters corresponding to each of the video frames includes:

[0019] Obtaining a coefficient subset corresponding to the target facial feature in the expression base coefficients of each of the key video frames;

[0020] Obtaining a parameter subset corresponding to the target virtual feature among the control parameters of the virtual image;

[0021] Based on the coefficient subsets and parameter subsets corresponding to each of the key video frames, the constructed network model is trained to obtain a redirection model.

[0022] In an optional implementation, the step of extracting key video frames from the multiple video frames includes:

[0023] Obtaining feature point information of a specified facial feature contained in each of the video frames;

[0024] The feature point information of the designated facial features of each of the video frames is compared to determine a video frame containing the designated facial features that meets a set state and serves as a key video frame.

[0025] In an optional implementation, the step of extracting key video frames from the multiple video frames includes:

[0026] Extracting a coefficient subset corresponding to a specified facial feature from the expression base coefficients of each of the video frames;

[0027] The coefficient subsets of the video frames are compared to determine the video frames containing the specified facial features that meet the set state and use them as key video frames.

[0028] In an optional implementation, the step of training the constructed network model to obtain the redirection model based on the expression base coefficients and control parameters corresponding to each of the video frames includes:

[0029] Importing the expression base coefficients corresponding to each of the video frames into the constructed network model to obtain an output result;

[0030] The control parameters corresponding to each of the video frames are compared with the output results, and the model parameters of the network model are adjusted based on the comparison results, and then training is continued until a redirection model is obtained when preset conditions are met.

[0031] In an optional embodiment, the method further comprises:

[0032] Importing expression base coefficients of other video frames except the key video frame in the multiple video frames into the redirection model, and outputting control parameters corresponding to each of the other video frames;

[0033] The control parameters corresponding to each of the key videos and each of the other video frames are associated with the corresponding expressions and saved.

[0034] In an optional embodiment, the method further comprises:

[0035] Obtaining a redirected video frame of a user to be processed, and calculating redirected template coefficients of a plurality of preset expression base templates relative to the redirected video frame;

[0036] Importing the redirection template coefficients into the redirection model and outputting redirection control parameters;

[0037] A controller controlling the target virtual model is adjusted based on the redirection control parameter so that the expression of the target virtual model is consistent with the expression of the user to be processed.

[0038] In a second aspect, the present application provides an expression redirection training device, the device comprising:

[0039] A collection module, used for collecting a video stream of a training user, wherein the video stream includes multiple video frames;

[0040] A calculation module, used for calculating, for each of the video frames, expression base coefficients of a plurality of preset expression base templates relative to the video frame;

[0041] An adjustment module, used for adjusting control parameters for controlling the expression of the virtual image so that the expression of the virtual image is consistent with the expression of the training user in the video frame;

[0042] The training module is used to train the constructed network model based on the expression base coefficients and control parameters corresponding to each of the video frames to obtain a redirection model.

[0043] In a third aspect, the present application provides an electronic device comprising one or more storage media and one or more processors communicating with the storage media, wherein the one or more storage media store machine-executable instructions executable by the processor, and when the electronic device is running, the processor executes the machine-executable instructions to perform the method steps described in any one of the aforementioned embodiments.

[0044] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed, the method steps described in any one of the aforementioned embodiments are implemented.

[0045] The beneficial effects of the embodiments of the present application include, for example:

[0046] The present application provides an expression redirection training method, device, electronic device and readable storage medium. By collecting multiple video frames of the training user, the expression base coefficients of multiple preset expression base templates relative to the video frame are calculated for each video frame, and the control parameters for controlling the expression of the virtual image are adjusted to make the expression of the virtual image consistent with the expression of the training user in the video frame. Finally, the constructed network model is trained based on the expression base coefficients and control parameters corresponding to the video frame to obtain a redirection model. In this scheme, the different expressions of the training user are first represented by the expression base template and the expression base coefficient, and various different expressions can be represented by standardized information. By adjusting the control parameters of the virtual image, the control parameters that make the expression of the virtual image correspond to the different expressions of the training user can be obtained, so that a redirection model that can realize expression migration and maintain the personalized characteristics of the virtual image can be trained, and a good applicability and natural expression redirection can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 A schematic diagram of an application scenario of the expression redirection training method provided in an embodiment of the present application;

[0049] Figure 2 A flowchart of the expression redirection training method provided in an embodiment of the present application;

[0050] Figure 3 A schematic diagram of an expression base template provided in an embodiment of the present application;

[0051] Figure 4 A schematic diagram of adjusting a virtual image provided in an embodiment of the present application;

[0052] Figure 5 for Figure 2 Flow chart of sub-steps included in step S130;

[0053] Figure 6 for Figure 5 Flow chart of sub-steps included in step S131;

[0054] Figure 7 for Figure 5 Another flowchart of the sub-steps included in step S131;

[0055] Figure 8 for Figure 5 Flow chart of sub-steps included in step S132;

[0056] Fig. 9 for Figure 2 Flow chart of sub-steps included in step S140;

[0057] Fig.10 for Figure 2 Another flowchart of the sub-steps included in step S140;

[0058] Fig.11 A flowchart of an association saving method in the expression redirection training method provided in an embodiment of the present application;

[0059] Fig.12 A schematic diagram of another application scenario of the expression redirection training method provided in an embodiment of the present application;

[0060] Fig.13 A flowchart of adjusting the control method in the expression redirection training method provided in an embodiment of the present application;

[0061] Fig.14 A structural block diagram of an electronic device provided in an embodiment of the present application;

[0062] Fig.15 This is a functional module block diagram of the expression redirection training device provided in an embodiment of the present application.

[0063] Icons: 100 - server; 110 - storage medium; 120 - processor; 130 - expression redirection training device; 131 - acquisition module; 132 - calculation module; 133 - adjustment module; 134 - training module; 140 - communication interface; 200 - image acquisition device; 300 - live broadcast provider; 400 - live broadcast receiver. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0065] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0066] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0067] In the description of the present application, it should be noted that, in the absence of conflict, the features in the embodiments of the present application can be combined with each other.

[0068] See also Figure 1 , is a schematic diagram of an application scenario of the expression redirection training method provided in an embodiment of the present application, wherein the application scenario includes a server 100 and an image acquisition device 200 that is communicatively connected to the server 100. The image acquisition device 200 may be one or more, and the image acquisition device 200 may include a device for acquiring two-dimensional images, such as a camera, and may also include a device for acquiring depth images, such as a depth camera.

[0069] In this embodiment, the image acquisition device 200 may send the acquired image information or video information to the server 100, and the server 100 may analyze and process the received image information or video information.

[0070] Combination Figure 2The embodiment of the present application also provides an expression redirection training method applicable to an electronic device, and the electronic device may be the above-mentioned server 100. The method steps defined in the process related to the expression redirection training method may be implemented by the electronic device. Figure 2 The specific process shown is described in detail.

[0071] Step S110, collecting a video stream of a training user, wherein the video stream includes multiple video frames.

[0072] Step S120: For each of the video frames, calculate expression base coefficients of a plurality of preset expression base templates relative to the video frame.

[0073] Step S130, adjusting the control parameters for controlling the expression of the virtual image so that the expression of the virtual image is consistent with the expression of the training user in the video frame.

[0074] Step S140, based on the expression base coefficients and control parameters corresponding to each of the video frames, the constructed network model is trained to obtain a redirection model.

[0075] In this embodiment, the image acquisition device 200 can be used to acquire a video stream of a training user, wherein the training user can be any user, such as a model of the project, etc. The training user can make as many different expressions as possible, such as opening the mouth, pursing the mouth, blinking, sticking out the tongue, etc. In this way, the multiple video frames contained in the acquired video stream can include multiple video frames with different expressions.

[0076] In addition, the electronic device stores a plurality of preset expression base templates, each of which may be a template modified for a certain facial feature of the face, such as Figure 3 For example, suppose there is a reference template in which each facial feature is in a natural state, that is, there is no obvious expression. Each expression base template is based on the reference template and modifies one of the facial features in the template to express the expression.

[0077] For example, a certain expression base template changes the mouth feature, and the mouth feature in the expression base template is in an open mouth state, while other facial features of the expression base template except the mouth feature remain consistent with the reference template. For another example, a certain expression base template changes the eye feature, and the eye feature in the expression base template is in a closed eye state.

[0078] Since each expression base template is an expression of a certain facial feature, a combination of multiple expression base templates can be used to express a variety of different expressions.

[0079] In this embodiment, for each video frame of the training user collected, the expression of the training user in each video frame can be represented by combining multiple expression base templates. When multiple expression base templates are combined for representation, multiple expression base coefficients can be calculated based on the video frame of the training user and the multiple expression base templates. After each expression base template is combined according to its corresponding expression base coefficient, it can be consistent with the expression state in the video frame of the training user.

[0080] The expression redirection of this embodiment requires transferring the expression of the training user to a virtual image. The virtual image can be a virtual image that increases the fun of the live broadcast in a live broadcast application scenario, such as an animal image, a loli image, etc., such as a rabbit animation or a cat animation.

[0081] Since the expression base template is made based on the characteristics of the person, if the information based on the expression base template is redirected to the virtual image, the expression of the redirected virtual image is very similar to that of the real person. However, since the proportional relationship between the facial features of the virtual image is inconsistent with the proportional relationship between the facial features of the actual person, if the information of the expression base template representing the person is directly transferred to the virtual image, it cannot be fully applied to the virtual image, making the expression of the virtual image unnatural.

[0082] Therefore, in this embodiment, the virtual image can be manually adjusted to make the virtual image maintain the same expression as the training user, and then the model can be trained based on the control parameters at this time, so as to obtain a model that can characterize the mapping relationship between the expression base coefficients and the control parameters.

[0083] In detail, in this embodiment, the skeleton points of the virtual image can be bound to the controller that controls the skeleton points in advance. In this embodiment, the face of the virtual image can include multiple skeleton points, wherein one skeleton point can correspond to an area, and the migration of the pixel points of the corresponding area can be achieved by controlling a certain skeleton point. A controller can be bound to one or more skeleton points, so that the control parameters of the controller can be adjusted to achieve control of the skeleton points bound to it.

[0084] After obtaining the expression base coefficients corresponding to each video frame, the designer can manually adjust the control parameters of the controller controlling the avatar for each video frame, so that the expression of the avatar is consistent with the expression of the training user in the video frame. For example, if the training user is in a laughing expression in the video frame, the designer can adjust the control parameters of the controller controlling the avatar's expression to keep the avatar's expression consistent with the training user's laughing expression.

[0085] As a possible implementation method, there are control points on the design interface that can be directly dragged with a mouse, such as Figure 4As shown in . The designer can directly adjust the position of each control point on the design interface. The electronic device obtains the control parameters of the corresponding controller according to the designer's adjustment information and displays the virtual image under the control parameters.

[0086] As another possible implementation, there is an input box on the design interface for directly inputting control parameters, and the designer can directly input the control parameters of the controller that need to be adjusted in the input box. Based on the obtained control parameters of the controller, the expression of the virtual image can be adjusted accordingly and displayed in real time on the design interface. The designer can adjust the control parameters while observing and comparing the expression of the virtual image and the expression of the training user to see if they are consistent. When it is determined that the expression of the virtual image is consistent with the expression of the training user, the control parameters of the controller at this time can be saved.

[0087] In this embodiment, multiple sets of one-to-one corresponding expression base coefficients and control parameters can be obtained through the above method. For a video frame under a certain expression of the training user, in a set of corresponding expression base coefficients and control parameters, the expression base coefficient combined with its expression base template can represent the expression of the training user in the video frame, and under the control of the control parameters, the virtual image can be consistent with the expression of the training user in the video frame.

[0088] On this basis, the constructed network model is trained using the expression base coefficients and control parameters corresponding to each video frame, so that a redirection model representing the mapping relationship between the expression base coefficients and the control parameters can be obtained. In this embodiment, the constructed network model can be, but is not limited to, a radial basis function network (RBF network).

[0089] The expression redirection training scheme provided by this scheme first uses expression base templates and expression base coefficients to represent different expressions of training users. Various different expressions can be represented by standardized information, and by adjusting the control parameters of the virtual image, control parameters that make the expressions of the virtual image correspond to the different expressions of the training users can be obtained. In this way, a redirection model that can realize expression transfer and maintain the personalized characteristics of the virtual image can be trained, thereby achieving expression redirection with good applicability and naturalness.

[0090] In this embodiment, considering that in some of the collected video frames of the training user, the expression of the training user is not obvious, if such video frames are used as a reference to adjust the controller parameters of the virtual image, it will not be convenient for the designer to observe whether the expression is consistent, and it will also increase unnecessary processing burden.

[0091] Therefore, please refer to Figure 5 ,In this embodiment, the controller parameters of the virtual image can be adjusted in the following ,way.

[0092] Step S131, extracting key video frames from the multiple video frames, wherein each of the key video frames contains a target facial feature in a set state.

[0093] Step S132, adjusting the control parameters of the controller that controls the expression of the virtual image so that the expression of the virtual image is consistent with the expression of the training user in the key video frame.

[0094] In this embodiment, the key video frame may be a video frame containing target facial features in a set state, wherein the target facial features may be, for example, mouth features, eye features, nose features, eyebrow features, etc. The set state may be set based on demand, for example, among the mouth features, a mouth in a wide-open state may be determined as being in a set state, and a mouth in a pursed state may be determined as being in a set state. In addition, among the eye features, eyes in a closed state may be determined as being in a set state, and eyes in a glaring state may be determined as being in a set state.

[0095] In this embodiment, the extracted key video frame is a video frame in which a certain facial feature can represent a relatively obvious expression. In this way, the control parameters of the controller controlling the expression of the avatar can be adjusted based on the key video frame, and the adjustment can be made based on relatively obvious information, so that the adjustment effect is better and the processing burden can be reduced.

[0096] As a possible implementation method, when determining the key video frame, it can be determined based on the feature point information of the facial features. For details, please refer to Figure 6 , the key video frames can be determined in the following way.

[0097] Step S1311A, obtaining feature point information of designated facial features contained in each of the video frames.

[0098] Step S1312A, comparing feature point information of the designated facial features of the video frames to determine a video frame containing the designated facial features that meets a set state and uses it as a key video frame.

[0099] The state of each facial feature of the training user in the video frame is jointly determined by multiple feature points constituting the facial features. For example, the information of multiple feature points constituting the mouth features is different in the open mouth state and the pursed mouth state. Therefore, the state of the facial features can be roughly determined based on the feature point information of the facial features.

[0100] For the set specified facial features, such as mouth features, the feature point information of the mouth features in each video frame can be obtained. The feature point information of the mouth features in each video frame can be compared, for example, the position information of the feature points contained in the mouth features in each video frame can be obtained, and by comparing the position information of the feature points of the mouth features in each video frame, the video frames in which the mouth features are in a relatively obvious state can be determined, for example, video frames in a state of maximum mouth opening, video frames in a state of maximum mouth crookedness, etc. The video frames in these states can be used as video frames that meet the set state, and such video frames can be used as key video frames.

[0101] As another possible implementation, when determining the key video frame, it can also be determined based on the expression base coefficients corresponding to each video frame. Figure 7 , the key video frames can be determined in the following way.

[0102] Step S1311B, extracting a coefficient subset corresponding to the specified facial features from the expression base coefficients of each of the video frames.

[0103] Step S1312B, comparing the coefficient subsets of the video frames to determine the video frames containing the specified facial features that meet the set state and use them as key video frames.

[0104] From the above, we can know that the expression base coefficients of the expression base templates relative to each video frame can be obtained. The preset multiple expression base templates can be divided into multiple groups for different facial features, where each group of expression base templates can contain one or more expression base templates. For example, please refer to Figure 3 , which may include a group of expression base templates for the nose, a group of expression base templates for the eyebrows, a group of expression base templates for the eyes, and a group of expression base templates for the mouth, etc. That is, the expression base templates belonging to the same group are changed only for a certain facial feature.

[0105] Correspondingly, the obtained expression base coefficients include multiple groups of coefficient subsets corresponding to each group of expression base templates. For example, a group of expression base templates for nose feature changes has a group of coefficient subsets, and the coefficients in the coefficient subsets correspond one-to-one to each expression base template in the group of expression base templates.

[0106] Therefore, for a specified facial feature, such as a mouth feature, a group of expression base templates for changes in the mouth feature can be determined in the expression base templates, and a coefficient subset corresponding to the group of expression base templates can be obtained.

[0107] The coefficient subsets of the specified facial features of each video frame can be combined to represent the state of the specified facial features. Figure 3In a set of expression base templates for mouth features, there are states such as open mouth, pursed mouth, and crooked mouth. If the set state is the open mouth state, then in the coefficient subset corresponding to the video frame of the training user in the open mouth state, the coefficient of the expression base template in the open mouth state should be the largest. Therefore, by comparing the coefficient subsets of each video frame, based on the size of the coefficients corresponding to each expression base template in the coefficient subset, the video frame containing the specified facial features that meet the set state can be determined as the key video frame.

[0108] In this embodiment, through the above process, a plurality of key video frames having a certain facial feature in a relatively exaggerated expression can be determined from a plurality of video frames, and the determined key video frames can be extracted.

[0109] On the basis of obtaining the key video frames, controller parameters of the virtual image can be adjusted based on the key video frames to maintain expressions consistent with those of the trained user in the key video frames.

[0110] As can be seen from the above, the key video frame is a video frame containing the target facial features in a set state, that is, among the facial features of the key video frame, only the target facial features may be in a relatively obvious expression state. If the virtual image is adjusted based on all the facial features of the key video frame, on the one hand, some facial features may not be obvious, and the adjustment effect may not be good, and on the other hand, it may also cause some unnecessary workload.

[0111] Therefore, please refer to Figure 8 In this embodiment, when adjusting the controller parameters of the virtual image based on the key video frame, it can be achieved in the following ways:

[0112] Step S1321, determining a target virtual feature in the virtual image that corresponds to the target facial feature.

[0113] Step S1322, obtaining a target controller that controls the target virtual feature.

[0114] Step S1323, adjusting the control parameters of the target controller so that the state of the target virtual feature in the virtual image is consistent with the state of the target facial feature.

[0115] In this embodiment, the facial features of the training user correspond one-to-one to the virtual features of the virtual image. For each key video frame, the target facial features in the key video frame can be obtained, that is, the facial features in the key video frame that meet the set state, for example, the target facial features in the key video frame can be mouth features. Accordingly, the target virtual features in the virtual image corresponding to the target facial features can be determined, such as the mouth features of the virtual image.

[0116] As can be seen from the above, the skeleton points of each area of ​​the avatar's face can be controlled by different controllers, for example, the mouth area and the eye area are controlled by different controllers. Therefore, a target controller for controlling a target virtual feature can be obtained, for example, a target controller for controlling the mouth feature of the avatar can be obtained.

[0117] In this way, only the control parameters of the target controller may be adjusted to make the state of the mouth features of the avatar consistent with the state of the mouth features of the training user in the key video frame.

[0118] In this embodiment, by adjusting the parameters of only the controller of the target virtual feature corresponding to the target facial feature in the set state in the key video frame, the parts with obvious expressions are adjusted in a targeted manner and compared for consistency, which can improve the adjustment effect and reduce the workload.

[0119] In this embodiment, please refer to Fig. 9 , based on obtaining a set of corresponding expression base coefficients and control parameters, when training a redirection model based on the corresponding expression base coefficients and control parameters, it can be achieved in the following ways:

[0120] Step S141A, importing the expression base coefficients corresponding to each of the video frames into the constructed network model to obtain an output result.

[0121] Step S142A, comparing the control parameters corresponding to each of the video frames with the output results, adjusting the model parameters of the network model based on the comparison results and continuing training until a redirection model is obtained when preset conditions are met.

[0122] In this embodiment, the expression base coefficients corresponding to each video frame can be used as training samples and input into the constructed network model. Here, all video frames can be used as training samples for training, or the key video frames obtained above can be used as training samples. Through the processing of the network model, the output result of the model can be obtained.

[0123] The expression base coefficients of each video frame have corresponding control parameters. The purpose of model training is to make the output results as consistent as possible with the actual control parameters. Therefore, the control parameters corresponding to each video frame can be compared with the output results, and a loss function can be constructed based on the comparison information. The constructed loss function can be a minimization function, which is minimized by calculating the loss function value, and the model parameters of the network model are adjusted after each iteration to continue training until the redirected model is obtained when the preset conditions are met. The preset conditions can be, for example, the number of iterations reaches a preset number, the loss function reaches convergence, etc.

[0124] In a possible implementation, if the network model used in this embodiment is an RBF model, the RBF model includes a plurality of network layers connected in sequence, for example, an input layer, a hidden layer, and an output layer. Among them, the hidden layer may include an RBF function layer and a fully connected layer connected in sequence. After the input expression base coefficients are processed by each network layer of the RBF model, the output result is obtained through the output layer. The minimization of the loss function constructed as described above can be used as an evaluation index of the model to adjust the model parameters of each network layer and after multiple iterations, finally obtain a redirection model that meets the preset conditions.

[0125] In a possible implementation, it can be seen from the above that key video frames can be extracted from the video frames. The key video frames contain facial features with more obvious expression states. If the model is trained with key video frames, the accuracy of the model can be guaranteed while reducing the workload. Therefore, please refer to Fig.10 In this embodiment, the training of the redirection model can also be implemented in the following ways:

[0126] Step S141B, obtaining a coefficient subset corresponding to the target facial feature in the expression base coefficients of each of the key video frames.

[0127] Step S142B, obtaining a parameter subset corresponding to the target virtual feature in the control parameters of the virtual image.

[0128] Step S143B, based on the coefficient subsets and parameter subsets corresponding to each of the key video frames, the constructed network model is trained to obtain a redirection model.

[0129] In this embodiment, for each key video frame, assuming that the target facial feature in a key video frame is a mouth feature, a group of expression base templates in which the mouth feature is changed can be obtained, and a coefficient subset of the group of expression base templates can be obtained. Through the above step of adjusting the control parameters of the controller of the virtual image based on the key video frame, a parameter subset of the target controller of the target virtual feature corresponding to the target facial feature of the key video frame can be obtained.

[0130] In this way, the network model can be trained based on the coefficient subsets of the target facial features in a group of key video frames and the parameter subsets of the corresponding target virtual features to obtain the redirected model.

[0131] In one application scenario, the expression redirection training method provided in this embodiment can be applied to obtain information that can be used to transfer multiple expressions to a virtual image offline. Fig.11 In this case, the expression redirection training method may further include the following steps:

[0132] Step S210, importing the expression base coefficients of other video frames in the multiple video frames except the key video frame into the redirection model, and outputting the control parameters corresponding to each of the other video frames.

[0133] Step S220, associating the control parameters corresponding to each of the key videos and each of the other video frames with the corresponding expressions and saving them.

[0134] In this embodiment, the redirection model trained by the key video frame can reflect the mapping relationship between the expression base coefficient and the control parameter. Therefore, for other video frames except the key video frame, the control parameters corresponding to the expression base coefficient of other video frames can be obtained by using the trained redirection model.

[0135] In each key video frame and each other video frame, the user is trained to be in different expressions, and each set of control parameters can be bound to the corresponding expression. In this way, when the virtual image needs to show a certain expression later, it only needs to find the control parameters corresponding to the required expression based on the binding relationship information, and set the controller of the virtual image to the corresponding control parameters.

[0136] In implementation, the control parameters can be bound to the expressions by obtaining description information of the expressions, such as laughing, pursing lips, crying, etc. The description information of each expression can be bound to the control parameters.

[0137] In addition, in another application scenario, the expression redirection training method provided in this embodiment can also be applied to real-time virtual image expression control. For example, in a live broadcast scenario, the virtual image can be controlled in real time based on the host's expression to display the corresponding expression.

[0138] In this case, please refer to Fig.12 The application scenario of the expression redirection training method provided in the embodiment of the present application may also include a live broadcast provider 300 and a live broadcast receiver 400, and the above-mentioned server 100 may be a live broadcast server. The live broadcast provider 300 and the live broadcast receiver 400 may be connected to the live broadcast server in communication. Among them, the live broadcast provider 300 may be a terminal device (such as a mobile phone, a tablet computer, a computer, etc.) used by the anchor during the live broadcast, and the live broadcast receiver 400 may be a terminal device (such as a mobile phone, a tablet computer, a computer, etc.) used by the audience when watching the live broadcast.

[0139] The live broadcast provider 300 can send the live video stream to the live broadcast server, and the audience can access the live broadcast server to watch the live video through the live broadcast receiving terminal 400. The live broadcast server can also receive the information sent by the live broadcast provider 300 and control the virtual image based on the trained redirection model to generate a corresponding live stream, and push it to the live broadcast receiving terminal 400.

[0140] See also Fig.13 In this application scenario, the expression redirection training method provided in this embodiment may further include the following steps:

[0141] Step S310, obtaining a redirected video frame of the user to be processed, and calculating redirected template coefficients of a plurality of preset expression base templates relative to the redirected video frame.

[0142] Step S320: import the redirection template coefficients into the redirection model, and output redirection control parameters.

[0143] Step S330: adjusting a controller controlling the target virtual model based on the redirection control parameter so that the expression of the target virtual model is consistent with the expression of the user to be processed.

[0144] In this embodiment, the user to be processed may be the host of the live broadcast room, and the obtained redirected video frame may be the captured video frame of the host. Similarly, the redirected template coefficients of the preset multiple expression base templates relative to the host's video frame may be calculated, and the calculation method of the redirected template coefficient is the same as the calculation method of the above-mentioned expression base coefficient, which will not be repeated here.

[0145] The redirection model obtained by the above training can output the redirection control parameters corresponding to the redirection template coefficients. The redirection control parameters can be used to adjust the controller that controls the target virtual model, so that the expression of the target virtual model is consistent with that of the anchor. Among them, the target virtual model is the virtual model in the current live broadcast room, which can be set according to needs.

[0146] In this way, the redirection model can be obtained by pre-training. In actual application, the redirection model can be directly called to obtain control parameters, and then the virtual image can be controlled to display the same expression as the anchor in real time. This can enhance the anchor's enthusiasm for using the virtual image for live broadcasting and enrich the liveliness of the anchor's live broadcast.

[0147] See also Fig.14 , is a schematic diagram of an exemplary component of an electronic device provided in an embodiment of the present application, and the electronic device may be Figure 1The server 100 shown in . The electronic device may include a storage medium 110, a processor 120, an expression redirection training device 130 and a communication interface 140. In the present embodiment, the storage medium 110 and the processor 120 are both located in the electronic device and the two are separately arranged. However, it should be understood that the storage medium 110 may also be independent of the electronic device and may be accessed by the processor 120 through a bus interface. Alternatively, the storage medium 110 may also be integrated into the processor 120, for example, it may be a cache and / or a general register.

[0148] The expression redirection training device 130 can be understood as the above-mentioned electronic device, or the processor 120 of the electronic device, or can be understood as a software function module independent of the above-mentioned electronic device or processor 120 that implements the above-mentioned expression redirection training method under the control of the electronic device.

[0149] like Fig.15 As shown, the above-mentioned expression redirection training device 130 may include a collection module 131, a calculation module 132, an adjustment module 133 and a training module 134. The functions of each functional module of the expression redirection training device 130 are respectively described in detail below.

[0150] An acquisition module 131 is used to acquire a video stream of a training user, wherein the video stream includes multiple video frames;

[0151] It can be understood that the acquisition module 131 can be used to execute the above step S110. The detailed implementation of the acquisition module 131 can refer to the above content related to step S110.

[0152] A calculation module 132, configured to calculate, for each of the video frames, expression base coefficients of a plurality of preset expression base templates relative to the video frame;

[0153] It can be understood that the calculation module 132 can be used to execute the above step S120. The detailed implementation of the calculation module 132 can refer to the above content related to step S120.

[0154] An adjustment module 133, used to adjust control parameters for controlling the expression of the avatar so that the expression of the avatar is consistent with the expression of the training user in the video frame;

[0155] It can be understood that the adjustment module 133 can be used to execute the above step S130. The detailed implementation of the adjustment module 133 can refer to the above content related to step S130.

[0156] The training module 134 is used to train the constructed network model based on the expression base coefficients and control parameters corresponding to each of the video frames to obtain a redirection model.

[0157] It can be understood that the training module 134 can be used to execute the above step S140. The detailed implementation of the training module 134 can refer to the above content related to step S140.

[0158] In a possible implementation, the adjustment module 133 may be specifically used to:

[0159] Extracting key video frames from the multiple video frames, wherein each of the key video frames contains a target facial feature in a set state;

[0160] The control parameters of the controller controlling the expression of the virtual image are adjusted so that the expression of the virtual image is consistent with the expression of the training user in the key video frame.

[0161] In a possible implementation, the adjustment module 133 may be specifically used to:

[0162] Determining a target virtual feature in the virtual image that corresponds to the target facial feature;

[0163] Obtaining a target controller that controls the target virtual feature;

[0164] The control parameters of the target controller are adjusted so that the state of the target virtual feature in the avatar is consistent with the state of the target facial feature.

[0165] In a possible implementation, the training module 134 may be specifically used for:

[0166] Obtaining a coefficient subset corresponding to the target facial feature in the expression base coefficients of each of the key video frames;

[0167] Obtaining a parameter subset corresponding to the target virtual feature among the control parameters of the virtual image;

[0168] Based on the coefficient subsets and parameter subsets corresponding to each of the key video frames, the constructed network model is trained to obtain a redirection model.

[0169] In a possible implementation, the adjustment module 133 may be specifically used to:

[0170] Obtaining feature point information of a specified facial feature contained in each of the video frames;

[0171] The feature point information of the designated facial features of each of the video frames is compared to determine a video frame containing the designated facial features that meets a set state and serves as a key video frame.

[0172] In a possible implementation, the adjustment module 133 may be specifically used to:

[0173] Extracting a coefficient subset corresponding to a specified facial feature from the expression base coefficients of each of the video frames;

[0174] The coefficient subsets of the video frames are compared to determine the video frames containing the specified facial features that meet the set state and use them as key video frames.

[0175] In a possible implementation, the training module 134 may be specifically used for:

[0176] Importing the expression base coefficients corresponding to each of the video frames into the constructed network model to obtain an output result;

[0177] The control parameters corresponding to each of the video frames are compared with the output results, and the model parameters of the network model are adjusted based on the comparison results, and then training is continued until a redirection model is obtained when preset conditions are met.

[0178] In a possible implementation, the expression redirection training device 130 further includes an association saving module, which can be specifically used for:

[0179] Importing expression base coefficients of other video frames except the key video frame in the multiple video frames into the redirection model, and outputting control parameters corresponding to each of the other video frames;

[0180] The control parameters corresponding to each of the key videos and each of the other video frames are associated with the corresponding expressions and saved.

[0181] In a possible implementation, the expression redirection training device 130 further includes a control module, which can be specifically used to:

[0182] Obtaining a redirected video frame of a user to be processed, and calculating redirected template coefficients of a plurality of preset expression base templates relative to the redirected video frame;

[0183] Importing the redirection template coefficients into the redirection model and outputting redirection control parameters;

[0184] A controller controlling the target virtual model is adjusted based on the redirection control parameter so that the expression of the target virtual model is consistent with the expression of the user to be processed.

[0185] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0186] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are executed, the expression redirection training method provided in the above embodiment is implemented.

[0187] Specifically, the computer-readable storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the computer-readable storage medium is executed, the above-mentioned expression redirection training method can be executed. Regarding the processes involved when the computer-readable storage medium and its executable instructions are executed, reference can be made to the relevant descriptions in the above-mentioned method embodiments, which will not be described in detail here.

[0188] In summary, the expression redirection training method, device, electronic device and readable storage medium provided by the present application collect multiple video frames of the training user, calculate the expression base coefficients of multiple preset expression base templates relative to the video frame for each video frame, and adjust the control parameters for controlling the expression of the virtual image so that the expression of the virtual image is consistent with the expression of the training user in the video frame. Finally, the constructed network model is trained based on the expression base coefficients and control parameters corresponding to the video frames to obtain the redirection model.

[0189] In this scheme, the different expressions of the training users are firstly represented by expression base templates and expression base coefficients. Various expressions can be represented by standardized information, and by adjusting the control parameters of the virtual image, the control parameters that make the expressions of the virtual image correspond to the different expressions of the training users can be obtained. Thus, a redirection model that can realize expression transfer and maintain the personalized characteristics of the virtual image can be trained, thereby realizing applicable and natural expression redirection.

[0190] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A facial expression redirection training method, characterized in that: The method comprises: Collecting a video stream of a training user, wherein the video stream includes multiple video frames; For each of the video frames, calculating expression base coefficients of a plurality of preset expression base templates relative to the video frame; Adjusting a control parameter for controlling the expression of the virtual image so that the expression of the virtual image is consistent with the expression of the training user in the video frame, wherein the control parameter is a parameter of a controller for controlling the virtual image; Based on the expression base coefficients and control parameters corresponding to each of the video frames, the constructed network model is trained to obtain a redirection model, where the redirection model is a model that characterizes the mapping relationship between the expression base coefficients and the control parameters.

2. The expression redirection training method according to claim 1, characterized in that: The step of adjusting the control parameters of the controller controlling the expression of the virtual image so that the expression of the virtual image is consistent with the expression of the training user in the video frame includes: Extracting key video frames from the multiple video frames, wherein each of the key video frames contains a target facial feature in a set state; The control parameters of the controller controlling the expression of the virtual image are adjusted so that the expression of the virtual image is consistent with the expression of the training user in the key video frame.

3. The expression redirection training method according to claim 2, characterized in that: The step of adjusting the control parameters of the controller controlling the virtual image so that the expression of the virtual image is consistent with the expression of the training user in the key video frame includes: Determining a target virtual feature in the virtual image that corresponds to the target facial feature; Obtaining a target controller that controls the target virtual feature; The control parameters of the target controller are adjusted so that the state of the target virtual feature in the avatar is consistent with the state of the target facial feature.

4. The expression redirection training method according to claim 3, characterized in that: The step of training the constructed network model to obtain the redirection model based on the expression base coefficients and control parameters corresponding to each of the video frames comprises: Obtaining a coefficient subset corresponding to the target facial feature in the expression base coefficients of each of the key video frames; Obtaining a parameter subset corresponding to the target virtual feature among the control parameters of the virtual image; Based on the coefficient subsets and parameter subsets corresponding to each of the key video frames, the constructed network model is trained to obtain a redirection model.

5. The expression redirection training method according to claim 2, characterized in that: The step of extracting key video frames from the multiple video frames comprises: Obtaining feature point information of a specified facial feature contained in each of the video frames; The feature point information of the designated facial features of each of the video frames is compared to determine a video frame containing the designated facial features that meets a set state and serves as a key video frame.

6. The expression redirection training method according to claim 2, characterized in that: The step of extracting key video frames from the multiple video frames comprises: Extracting a coefficient subset corresponding to a specified facial feature from the expression base coefficients of each of the video frames; The coefficient subsets of the video frames are compared to determine the video frames containing the specified facial features that meet the set state and use them as key video frames.

7. The expression redirection training method according to claim 1, characterized in that: The step of training the constructed network model to obtain the redirection model based on the expression base coefficients and control parameters corresponding to each of the video frames comprises: Importing the expression base coefficients corresponding to each of the video frames into the constructed network model to obtain an output result; The control parameters corresponding to each of the video frames are compared with the output results, and the model parameters of the network model are adjusted based on the comparison results, and then training is continued until a redirection model is obtained when preset conditions are met.

8. The expression redirection training method according to claim 2, characterized in that: The method further comprises: Importing expression base coefficients of other video frames except the key video frame in the multiple video frames into the redirection model, and outputting control parameters corresponding to each of the other video frames; The control parameters corresponding to each of the key videos and each of the other video frames are associated with the corresponding expressions and saved.

9. The expression redirection training method according to claim 1, characterized in that: The method further comprises: Obtaining a redirected video frame of a user to be processed, and calculating redirected template coefficients of a plurality of preset expression base templates relative to the redirected video frame; Importing the redirection template coefficients into the redirection model and outputting redirection control parameters; A controller controlling the target virtual model is adjusted based on the redirection control parameter so that the expression of the target virtual model is consistent with the expression of the user to be processed.

10. An expression redirection training device, characterized in that: For implementing the expression redirection training method according to any one of claims 1 to 9, the device comprises: A collection module, used for collecting a video stream of a training user, wherein the video stream includes multiple video frames; A calculation module, used for calculating, for each of the video frames, expression base coefficients of a plurality of preset expression base templates relative to the video frame; An adjustment module, used for adjusting control parameters for controlling the expression of the virtual image so that the expression of the virtual image is consistent with the expression of the training user in the video frame; The training module is used to train the constructed network model based on the expression base coefficients and control parameters corresponding to each of the video frames to obtain a redirection model.

11. An electronic device, characterized in that: The electronic device comprises one or more storage media and one or more processors communicating with the storage media, wherein the one or more storage media store machine executable instructions executable by the processor, and when the electronic device is running, the processor executes the machine executable instructions to perform the method steps described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed, the method steps described in any one of claims 1 to 9 are implemented.

Images