Related methods, devices and equipment for training expression conversion models

Abstract

The present application discloses a method and a device for training an expression conversion model, wherein the method for training an expression conversion model includes: obtaining a first sample set and a second sample set; wherein the samples of the first sample set and the second sample set are expression data; using the first sample set to pre-train the initial model to obtain a first pre-trained model; performing a first preset processing on the expression data of the first sample set, and inputting the first sample set after the first preset processing into the first pre-trained model for training to obtain a second pre-trained model; performing a second preset processing on the expression data of the second sample set, and inputting the second sample set after the second preset processing into the second pre-trained model for training, and determining the trained model as the expression conversion model. The above scheme can train an expression conversion model for style conversion based on expression data, thereby greatly improving the speed and efficiency of expression conversion through the expression conversion model.

Description

Technical Field

[0001] The present application relates to the technical field of expression conversion, and in particular to related methods, devices and equipment for training expression conversion models. Background Art

[0002] Style transfer for facial animation is a very challenging task.

[0003] At present, the more mature technical route for style transformation of facial animation in the industry is to let professional actors perform in accordance with a certain required animation style, generate facial animation after facial capture, and then animators manually adjust the animation data frame by frame based on the facial capture to generate animation with a certain facial expression style. This is also the most mature solution in the current film and television industry.

[0004] Although the above scheme can give satisfactory animation results according to the experience of the animator, it consumes a huge amount of time and energy and is not suitable for large-scale or batch expression conversion. Summary of the invention

[0005] The present application provides a method and apparatus for training an expression conversion model, and solves the problem of high time and energy costs of expression conversion in the prior art.

[0006] The present application provides a method for training an expression conversion model, the method comprising: obtaining a first sample set and a second sample set; wherein the samples of the first sample set and the second sample set are expression data; using the first sample set to pre-train an initial model to obtain a first pre-trained model; performing a first preset processing on the expression data of the first sample set, and inputting the first sample set after the first preset processing into the first pre-trained model for training to obtain a second pre-trained model; performing a second preset processing on the expression data of the second sample set, and inputting the second sample set after the second preset processing into the second pre-trained model for training, and determining the trained model as the expression conversion model.

[0007] Among them, the initial model includes an initial encoding model and an initial decoding model that are cascaded to each other; the initial model is pre-trained using the first sample set, and the step of obtaining the first pre-trained model includes: inputting the first sample set into the initial encoding model for encoding, and inputting the data encoded by the initial encoding model into the initial decoding model for decoding, until the result output by the initial decoding model is the same as the first sample set, so as to obtain the first encoding model and the first decoding model; and determining the first encoding model and the first decoding model as the first pre-trained model.

[0008] Among them, the expression data of the first sample set is subjected to a first preset processing, and the first sample set after the first preset processing is input into the first pre-training model for training, and the step of obtaining the second pre-training model includes: performing the first preset processing on part of the expression data in the first sample set; inputting the first sample set after the first preset processing into the first pre-training model for training until the first decoding model outputs the same data as the first sample set before the first preset processing, so as to obtain the second decoding model; and determining the first encoding model and the second decoding model as the second pre-training model.

[0009] Among them, the steps of performing a second preset processing on the expression data of the second sample set, inputting the second sample set after the second preset processing into the second pre-trained model for training, and determining the trained model as the expression conversion model include: performing a second preset processing on part of the expression data in the second sample set; inputting the first sample set after the second preset processing into the second pre-trained model for training until the second decoding model outputs the same data as the second sample set before the second preset processing, so as to obtain a third decoding model; and determining the first encoding model and the third decoding model as the expression conversion model.

[0010] The first sample set includes a plurality of frames of first expression data; the step of performing a first preset process on the expression data of the first sample set includes: randomly adjusting the expression base coefficients corresponding to the first expression data of some frames to zero.

[0011] The second sample set includes multiple frames of second expression data; the step of performing a second preset process on the expression data of the second sample set includes: randomly adjusting the expression base coefficients corresponding to the second expression data of some frames to zero.

[0012] Among them, the steps of obtaining the first sample set and the second sample set include: collecting multiple frames of first expression data of the target object within a preset time period through a face capture tool, and determining the multiple frames of first expression data as the first sample set; and receiving manual adjustments to the basic expression to obtain multiple frames of second expression data, and determining the multiple frames of second expression data as the second sample set; wherein the expression styles of the multiple frames of first expression data in the first sample set are different from those of the multiple frames of second expression data in the second sample set.

[0013] The present application also provides an expression conversion method, including obtaining expression data to be converted; inputting the expression data to be converted into an expression conversion model for conversion, and obtaining expression data after expression style conversion; wherein the expression conversion model is trained by any of the above-mentioned expression conversion model training methods.

[0014] Among them, after the step of obtaining the expression data to be converted, part of the expression data to be converted is input into the expression conversion model for conversion, and before the step of obtaining the expression data after the complete expression style conversion includes: extracting part of the expression data to be converted from the expression data to be converted based on the characteristics of the expression data to be converted and / or the expression base coefficients.

[0015] The present application also provides a training device for an expression conversion model, and the training device for the expression conversion model includes: an acquisition module, used to obtain a first sample set and a second sample set; wherein the samples of the first sample set and the second sample set are expression data; a pre-training module, used to pre-train an initial model using the first sample set to obtain a first pre-trained model; a first training module, used to perform a first preset processing on part of the expression data of the first sample set, and input the first sample set after the first preset processing into the first pre-training model for training to obtain a second pre-training model; a second training module, used to perform a second preset processing on part of the expression data of the second sample set, and input the second sample set after the second preset processing into the second pre-training model for training, and determine the trained model as the expression conversion model.

[0016] The present application also provides an expression conversion device, the expression conversion device comprising: an acquisition module, used to obtain expression data to be converted;

[0017] A conversion module, used to input part of the expression data to be converted into the expression conversion model for conversion, so as to obtain the expression data after the complete expression style conversion;

[0018] The expression conversion model is obtained by training using any of the expression conversion model training methods described above.

[0019] The present application also provides an electronic device, comprising a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement any of the above-mentioned expression conversion model training methods or expression conversion methods.

[0020] The present application also provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implements any of the above-mentioned training methods or expression conversion methods for the expression conversion model.

[0021] In the above scheme, the present application first pre-trains the initial model based on the first sample set to obtain the first pre-trained model, then performs a first preset processing on the expression data of the first sample set, and inputs the first sample set after the first preset processing into the first pre-trained model for training to obtain the second pre-trained model, and then performs a second preset processing on the expression data of the second sample set, and inputs the second sample set after the second preset processing into the second pre-trained model for training, and determines the trained model as the expression conversion model, thereby performing multiple training on the initial model through different sample sets, so that the final expression conversion model can obtain expression data with a certain expression style based on the expression data after the preset processing, complete the expression conversion, and then greatly improve the speed and efficiency of the expression conversion through the expression conversion model. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flowchart of an embodiment of a training method for an expression conversion model of the present application;

[0023] Figure 2 It is a flowchart of another embodiment of the training method of the expression conversion model of the present application;

[0024] Figure 3 It is a flowchart of an embodiment of the expression conversion method of the present application;

[0025] Figure 4 It is a flowchart of another embodiment of the expression conversion method of the present application;

[0026] Figure 5 It is a schematic diagram of the framework of an embodiment of a training device for an expression conversion model of the present application;

[0027] Figure 6 It is a schematic diagram of the framework of an embodiment of the expression conversion device of the present application;

[0028] Figure 7 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0029] Figure 8 A schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0030] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0031] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0032] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of the associated objects. There can be three kinds of relationships. For example, A and / or B can be: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally means that the associated objects before and after are in an "or" relationship. In addition, "more" in this article means two or more than two.

[0033] See also Figure 1 , Figure 1 It is a flowchart of an embodiment of a training method for an expression conversion model of the present application.

[0034] Step S11: Acquire a first sample set and a second sample set; wherein the samples of the first sample set and the second sample set are expression data.

[0035] A first sample set and a second sample set are obtained, wherein the samples of the first sample set and the second sample set are expression data, and the expression data in the first sample set and the second sample set are different.

[0036] In a specific application scenario, the first sample set can be obtained by filming the facial expressions of professional actors, and the second sample set can be obtained by animators adjusting the basic expressions using facial expression adjustment tools.

[0037] In another specific application scenario, the first sample set may be obtained by filming the facial expressions of professional actors, and the animated facial expression data may be used as the second sample set.

[0038] In another specific application scenario, an animator may also use an expression adjustment tool to adjust the basic expressions into different styles to obtain a first sample set and a second sample set.

[0039] In this embodiment, the specific method for obtaining the first sample set and the second sample set is not limited here.

[0040] Step S12: pre-training the initial model using the first sample set to obtain a first pre-trained model.

[0041] The initial model is pre-trained using the expression data in the first sample set to obtain a first pre-trained model, wherein the initial model includes an encoder-decoder mode model, a convolutional neural network, a recurrent neural network, and other models, which are not specifically limited here.

[0042] In a specific application scenario, the expression data in the first sample set can be used as labels and inputs to pre-train an initial model of the encoder-decoder mode to obtain a first pre-trained model.

[0043] In another specific application scenario, the expression data in the first sample set can be used as input, and the preset expression data can be used as labels to pre-train the initial model to obtain a first pre-trained model. The preset expression data can be set manually or according to actual needs.

[0044] Step S13: performing a first preset processing on the expression data of the first sample set, and inputting the first sample set after the first preset processing into the first pre-training model for training to obtain a second pre-training model.

[0045] After obtaining the first pre-training model, the expression data of the first sample set is subjected to a first preset processing, and the first sample set after the first preset processing is input into the first pre-training model for training to obtain a second pre-training model.

[0046] The first preset processing includes adjusting the expression data of the first sample set. Specifically, the expression base coefficients corresponding to some expression data of the first sample set can be adjusted to 0 or the expression base coefficients corresponding to the expression data of the first sample set can be adjusted according to a certain rule. The specific setting can be based on the training purpose of the second pre-trained model.

[0047] In a specific application scenario, the first sample set after the first preset processing can be input into the first pre-trained model, and the original first sample set before the first preset processing is used as a training label to train the first pre-trained model to obtain a second pre-trained model. After this training is completed, the second pre-trained model can generate complete expression data based on expression data similar to that after the first preset processing. For example, when the first preset processing is to adjust the expression base coefficient corresponding to part of the expression data of the first sample set to 0, the second pre-trained model after training can generate a complete expression animation based on the partially valid expression base coefficients.

[0048] In another specific application scenario, the first sample set after the first preset processing can be input into the first pre-trained model, and the first pre-trained model is trained using the preset style expression data as training labels to obtain a second pre-trained model. After this training is completed, the second pre-trained model can generate preset style expression data based on expression data similar to the first preset processing.

[0049] Step S14: performing a second preset processing on the expression data of the second sample set, and inputting the second sample set after the second preset processing into a second pre-trained model for training, and determining the trained model as the expression conversion model.

[0050] The expression data of the second sample set is subjected to a second preset processing, and the second sample set after the second preset processing is input into a second pre-trained model for training, and the trained model is determined as the expression conversion model. In this step, the expression conversion model is obtained by re-training the second sample set on the basis of the second pre-trained model.

[0051] The second preset processing includes adjusting the expression data of the second sample set. Specifically, the expression base coefficients corresponding to part of the expression data of the second sample set can be adjusted to 0 or the expression base coefficients corresponding to the expression data of the second sample set can be adjusted according to a certain rule. The specific setting can be based on the training purpose of the expression conversion model. The second preset processing in this step can be the same as or different from the first preset processing, and can be specifically set based on actual needs.

[0052] In a specific application scenario, the second sample set after the second preset processing can be input into the second pre-trained model, and the second pre-trained model is trained using the original second sample set before the second preset processing as a training label to obtain an expression conversion model. After this training is completed, the expression conversion model can generate expression data with the expression style of the second sample set based on expression data similar to the expression data after the first preset processing, thereby completing the expression conversion.

[0053] In another specific application scenario, the second sample set after the second preset processing can be input into the second pre-trained model, and the preset style expression data is used as a training label to train the second pre-trained model to obtain an expression conversion model. After this training is completed, the expression conversion model can generate expression data of the preset style based on expression data similar to the first preset processing, thereby completing the expression conversion.

[0054] Through the above steps, the training method of the expression conversion model of the present embodiment first pre-trains the initial model based on the first sample set to obtain the first pre-trained model, then performs the first preset processing on the expression data of the first sample set, and inputs the first sample set after the first preset processing into the first pre-trained model for training to obtain the second pre-trained model, and then performs the second preset processing on the expression data of the second sample set, and inputs the second sample set after the second preset processing into the second pre-trained model for training, and determines the trained model as the expression conversion model, thereby performing multiple training on the initial model through different sample sets, so that the final expression conversion model can obtain expression data with a certain expression style based on the expression data after the preset processing, complete the expression conversion, and then greatly improve the speed and efficiency of the expression conversion through the expression conversion model.

[0055] See also Figure 2 , Figure 21 is a flow chart of another embodiment of the training method of the expression conversion model of the present application. This embodiment will be described by taking the initial model as an encoder-decoder model as an example.

[0056] Step S21: Collect multiple frames of first expression data of the target object within a preset time period through a face capture tool, and determine the multiple frames of first expression data as a first sample set, and receive manual adjustments to the basic expression to obtain multiple frames of second expression data, and determine the multiple frames of second expression data as a second sample set.

[0057] The target object is made to make various expressions within a preset time period, and the expressions are recorded to obtain an expression video of the preset time period. The expression data of the target object within the preset time period is collected based on the expression video by a face capture tool to obtain multiple frames of first expression data.

[0058] The target object may include living objects such as humans, cats, and dogs. The preset time period may include a longer time period, such as 2 hours, 3 hours, etc., so that the target object can make more diverse and comprehensive expressions as much as possible within the preset time period, thereby enriching the first sample set and improving the training effect. The face capture tool of this embodiment includes an ARkit face capture tool, a vtuber face capture tool, or other face capture tools, which are not specifically limited here.

[0059] Moreover, after obtaining multiple frames of first expression data, expression base coefficient analysis can be performed based on the first expression data to obtain an expression base coefficient sequence of each frame of first expression data. Each expression base coefficient value ranges from 0 to 1, 0 represents that the corresponding expression base does not work, that is, invalid or no expression base, and 1 represents that the expression base is fully activated.

[0060] In a specific application scenario, when the face capture tool is an ARkit face capture tool, the ARkit face capture tool is designed based on the ARkit standard, and after collecting a frame of expression data, it can output a corresponding 51 expression base coefficient sequence based on the frame of expression data. When there are M frames of first expression data in a preset time period, the first expression data in the first sample set is M*51 matrix data. The value of M is not specifically limited.

[0061] In another specific application scenario, expression bases may be acquired sequentially from multiple frames of first expression data collected by the face capture tool based on expression recognition software or an expression recognition model, and an expression base coefficient sequence of each frame of the first expression data may be obtained.

[0062] Receive manual adjustments to the basic expression, obtain multiple frames of second expression data, and determine the multiple frames of second expression data as a second sample set. Specifically, the style adjustment of the basic expression by the animator using expression adjustment software can be accepted to obtain multiple frames of second expression data, and the multiple frames of second expression data can be determined as a second sample set. The expression adjustment software includes MAYA software or other expression adjustment software. Moreover, after obtaining multiple frames of second expression data, expression base coefficient analysis can be performed based on the second expression data to obtain an expression base coefficient sequence of each frame of the second expression data.

[0063] The expression style of the second expression data in the second sample set is the expression style that needs to be converted. In a specific application scenario, when expression animation is applied to virtual images such as cats, dogs, and hamsters, the expressions of the virtual images are difficult to be performed by actors. Therefore, the animator obtains the second expression data that matches the expression style of the virtual image through expression adjustment software, so that the expression conversion model finally trained can convert the expression into expression data that matches the expression style of the virtual image, thereby realizing the application of the converted expression.

[0064] In a specific application scenario, when each frame of the first expression data in the first sample set includes S expression bases, each frame of the second expression data in the second sample set also includes S expression bases. The number and format of the expression bases in the second expression data in the second sample set are the same as the number and format of the expression bases in the first expression data in the first sample set.

[0065] In a specific application scenario, when the first sample set is acquired through the ARkit face capture tool, and the first expression data in the first sample set is M*51 matrix data, the second expression data in the second sample set can be N frames, and the second expression data in the second sample set is N*51 matrix data. The value of N is not specifically limited.

[0066] In this embodiment, the expression style of the first expression data in the first sample set is realistic, and the expression style of the second expression data in the second sample set is different from that of the first sample set, and can be an expression style suitable for a virtual image or other required expression styles. By training the initial model with the first sample set and the second sample set, the model can convert realistic expressions into the required expression style.

[0067] Step S22: input the first sample set into the initial encoding model for encoding, and input the data encoded by the initial encoding model into the initial decoding model for decoding, until the result output by the initial decoding model is the same as the first sample set, to obtain a first encoding model and a first decoding model, and determine the first encoding model and the first decoding model as the first pre-trained model.

[0068] The initial model of this embodiment includes an initial encoding model (encoder) and an initial decoding model (decoder) which are cascaded to each other.

[0069] First, the first sample set is input into the initial encoding model for encoding, and the data encoded by the initial encoding model is input into the initial decoding model for decoding, until the result output by the initial decoding model is the same as the first sample set, thereby obtaining a first encoding model and a first decoding model, and determining the first encoding model and the first decoding model as the first pre-training model.

[0070] In this step, the input and label used for the training of the initial model are the first sample set, until the initial model converges, so that after a set of input data is encoded by the first encoding model, the first decoding model can decode based on the encoded data, thereby obtaining data with the same format, quantity, and content as the input data, that is, the same result, thereby ensuring that the model can correctly perform expression base coefficient analysis. Further, after the initial model converges, that is, the first decoding model can decode the corresponding format based on the encoded data, thereby obtaining data with the same format as the input data, and then can be adjusted based on the encoded data, and then the adjusted data is input into the first decoding model to generate expression data matching the first sample set, thereby reducing the tedious operation of collecting expressions of the target object through the face capture tool, and can directly obtain new first expression data by modifying the encoded data, thereby improving the generation efficiency of the first expression data, and then can improve the richness of the first expression data.

[0071] Step S23: Perform a first preset processing on part of the expression data in the first sample set, input the first sample set after the first preset processing into the first pre-training model for training, until the first decoding model outputs the same data as the first sample set before the first preset processing, obtain the second decoding model, and determine the first encoding model and the second decoding model as the second pre-training model.

[0072] Perform a first preset processing on part of the expression data in the first sample set, input the first sample set after the first preset processing into the first pre-training model for training, until the first decoding model outputs the same data as the first sample set before the first preset processing, obtain the second decoding model, and determine the first encoding model and the second decoding model as the second pre-training model. In this training process, the parameters of the first encoding model remain unchanged, that is, although the first sample set after the first preset processing is input into the first pre-training model for training, only the first decoding model is trained.

[0073] Specifically, this embodiment performs a random mask operation on the matrix of the complete multi-frame first expression data to obtain a sparse matrix, that is, randomly adjusts the expression base coefficients corresponding to the first expression data of some frames to zero, and then inputs the sparse matrix into the first pre-training model for decoding, until a complete matrix of the first expression data that is the same as before the random mask operation is output, and this training is completed to obtain a second decoding model. Among them, this training is based on the training data with the expression base coefficient of some frame data being 0 to generate complete expression data, so that the second pre-training model can generate a large amount of complete expression data based on a small amount of valid expression data. Among them, the range of adjusting the expression base coefficient of the first expression data to 0 in the first preset processing is 20%-95%, which can be 20%, 23%, 30%, 35%, 42%, 46%, 50%, 58%, 63%, 70%, 75%, 80%, 90%, 95%, etc. Specifically, the more expression base coefficients are adjusted to 0 in the first preset processing, the more complete expression data can be generated based on less expression data by the trained second decoding model.

[0074] In a specific application scenario, when the training sequence length is defined as 100 frames, and the expression base contained in a single frame of expression data is 51, a random mask operation can be performed on the 100*51 matrix of the first expression data. For example, the expression base coefficients corresponding to the first expression data of 91 frames in the 100*51 matrix are all adjusted to 0, and the original expression base coefficients of 9 frames are retained to obtain a 100*51 sparse matrix in which at least 91% of the data are 0. The first expression data after the random mask operation is input into the first pre-trained model for decoding until the first decoding model outputs a complete 100*51 matrix of the first expression data that is the same as before the random mask operation, that is, the first decoding model can correctly predict the masked frames. This training is completed and the second decoding model is obtained.

[0075] After the training of this step is completed, the second pre-trained model can generate a complete expression animation based on partial expression data and has an automatic completion function.

[0076] Step S24: perform a second preset processing on part of the expression data in the second sample set, input the first sample set after the second preset processing into the second pre-training model for training, until the second decoding model outputs the same data as the second sample set before the second preset processing, obtain the third decoding model, and determine the first encoding model and the third decoding model as the expression conversion model.

[0077] The training in the previous two steps can only let the model learn how to correctly predict the conventional expression coefficients, that is, to generate complete expression animation data using the output of the face capture tool, which does not have style. This step continues to train the second pre-trained model.

[0078] A second preset processing is performed on part of the expression data in the second sample set, and the first sample set after the second preset processing is input into the second decoding model for training, until the second decoding model outputs the same data as the second sample set before the second preset processing, and a third decoding model is obtained, and the first encoding model and the third decoding model are determined as the expression conversion model. In this training process, the parameters of the first encoding model remain unchanged, that is, although the first sample set after the second preset processing is input into the second pre-training model for training, only the second decoding model is trained.

[0079] Specifically, this embodiment performs a random mask operation on the matrix of the complete multi-frame second expression data to obtain a sparse matrix, that is, randomly adjusts the expression base coefficients corresponding to the second expression data of some frames to zero, and then inputs the sparse matrix into the second pre-training model for decoding until a complete matrix of the second expression data that is the same as before the random mask operation is output. This training is completed and a third decoding model is obtained. Among them, this training is based on the training data with the expression base coefficient of some frame data being 0 to generate complete expression data, so that the expression conversion model can generate a large amount of complete expression data with the same expression style as the second sample set based on a small amount of valid expression data. Among them, the range of adjusting the expression base coefficient of the second expression data to 0 in the second preset processing is 20%-95%, which can be 20%, 23%, 30%, 35%, 42%, 46%, 50%, 58%, 63%, 70%, 75%, 80%, 90%, 95% and the like. Specifically, the more expression base coefficients are adjusted to 0 in the first preset processing, the more the trained expression conversion model can generate complete expression data with the same expression style as the second sample set based on less expression data.

[0080] The specific steps of the second preset processing of this step are similar to those of the first preset processing of the previous step. Both of them perform random mask operations on multiple frames of expression data to obtain a sparse matrix, but the object is different from the actual random result.

[0081] In this step, the second decoding model is trained by using the second sample set with the expression style, so that the trained second decoding model can generate expression data with the same expression style as the second sample set.

[0082] Based on step S23, this step converts the automatic completion function of the model into the automatic completion function of the expression data with style.

[0083] Through the above steps, the training method of the expression conversion model of the present embodiment can be based on the first sample set and the second sample set with different expression styles, and train them separately after performing random Mask operations on them respectively, so that the expression conversion model can generate complete expression data with the same expression style as the second sample set based on partial expression data, and realize the automatic completion function of expression data with style, so that complete expression style conversion can be achieved based on a small amount of expression data, reducing the workload of manual adjustment, and greatly improving the speed and efficiency of expression conversion through the expression conversion model, so as to realize large-scale or batch expression conversion.

[0084] See also Figure 3 , Figure 3 It is a flowchart of an embodiment of the expression conversion method of the present application.

[0085] Step S31: Acquire the expression data to be converted.

[0086] Obtaining the expression data to be converted. In a specific application scenario, the expression of the target object can be captured by a face capture tool to obtain the expression data to be converted. In another specific application scenario, the target object can also be photographed by a camera of a smart terminal to obtain the expression data to be converted.

[0087] The expression style of the expression data to be converted in this embodiment may be the same as that of the first sample set in the aforementioned embodiment, for example, both are realistic styles.

[0088] Step S32: inputting the expression data to be converted into the expression conversion model for conversion to obtain expression data after expression style conversion.

[0089] The expression data to be converted is input into the expression conversion model for conversion to obtain expression data after expression style conversion. The expression data after expression style conversion can be the same as the expression style of the second sample set in the above embodiment.

[0090] The expression conversion model of this embodiment includes any expression conversion model in the aforementioned embodiments.

[0091] Through the above method, the expression conversion method of this embodiment converts the expression data to be converted through the expression conversion model to obtain expression data with a certain style, thereby automatically completing the style conversion of the expression data, greatly improving the speed and efficiency of expression conversion, and realizing large-scale or batch expression conversion.

[0092] See also Figure 4 , Figure 4 It is a flowchart of another embodiment of the expression conversion method of the present application.

[0093] Step S41: Obtain expression data to be converted.

[0094] This step is the same as step S31 in the aforementioned embodiment. Please refer to the above text and will not be repeated here.

[0095] Step S42: extracting part of the expression data to be converted from the expression data to be converted based on the features of the expression data to be converted and / or the expression base coefficients.

[0096] This embodiment can extract part of the expression data to be converted from the expression data to be converted based on the features of the expression data to be converted and / or the expression base coefficients. In a specific application scenario, part of the expression data to be converted can be extracted from the expression data to be converted based on the extreme values ​​of each expression base coefficient in the expression data to be converted, thereby extracting the key expression data to be converted in the expression data to be converted, so as to facilitate the expression conversion model to perform conversion. In another specific application scenario, part of the expression data to be converted can be extracted from the expression data to be converted based on the features of the expression data to be converted. For example, expression data with obvious expressions are extracted from the expression data to be converted. Among them, obvious expressions can include obvious expressions such as laughing, aggrieved, and crying. Thereby extracting the key expression data to be converted in the expression data to be converted, so as to facilitate the expression conversion model to perform conversion.

[0097] Among them, the amount of extracted partial expression data to be converted can account for 20%-95% of the expression data to be converted, specifically 20%, 23%, 30%, 35%, 42%, 46%, 50%, 58%, 63%, 70%, 75%, 80%, 90%, 95% and the like.

[0098] In a specific application scenario, an expression animation is recorded using ARkit face capture, with 100 frames. Then, 20 frames of expression data to be converted can be extracted from the expression data to be converted based on features and / or expression base coefficients. After inputting them into the expression conversion model, 100 frames of expression data after style conversion can be obtained.

[0099] In other embodiments, part of the expression data to be converted may also be randomly extracted from the expression data to be converted.

[0100] Step S43: inputting part of the expression data to be converted into the expression conversion model for conversion to obtain expression data after expression style conversion.

[0101] The extracted part of the expression data to be converted is input into the expression conversion model for conversion to obtain the expression data after the complete expression style conversion, wherein the complete expression data has the same number of frames as the expression data to be converted before the partial extraction. The expression conversion model of this embodiment includes any expression conversion model in the foregoing embodiments.

[0102] This embodiment can perform expression style conversion of complete expression data based on part of the expression data to be converted, thereby greatly improving the efficiency of preparing expression animations, and the method can be easily migrated to more expression styles. It only needs to select expression animations of different styles as training data in the training phase, so that the results of face capture output can have results of multiple styles, thereby bringing more diversity to the expression animation.

[0103] Through the above steps, the embodiment extracts part of the expression data to be converted, and then inputs part of the expression data to be converted into the expression conversion model to convert the expression data to be converted, thereby obtaining complete expression data with a certain style, thereby automatically completing the style conversion of the expression data and the automatic completion of the expression data, greatly improving the speed and efficiency of the expression conversion, and realizing large-scale or batch expression conversion.

[0104] See also Figure 5 , Figure 5 It is a schematic diagram of the framework of an embodiment of the training device of the expression conversion model of the present application. The training device 50 of the expression conversion model includes an acquisition module 51, a pre-training module 52, a first training module 53 and a second training module 54. The acquisition module 51 is used to obtain a first sample set and a second sample set; wherein the samples of the first sample set and the second sample set are expression data; the pre-training module 52 is used to pre-train the initial model using the first sample set to obtain a first pre-training model; the first training module 53 is used to perform a first preset processing on part of the expression data of the first sample set, and input the first sample set after the first preset processing into the first pre-training model for training to obtain a second pre-training model; the second training module 54 is used to perform a second preset processing on part of the expression data of the second sample set, and input the second sample set after the second preset processing into the second pre-training model for training, and determine the trained model as the expression conversion model.

[0105] The pre-training module 52 is also used to input the first sample set into the initial encoding model for encoding, and input the data encoded by the initial encoding model into the initial decoding model for decoding, until the result output by the initial decoding model is the same as the first sample set, to obtain a first encoding model and a first decoding model; the first encoding model and the first decoding model are determined as the first pre-training model.

[0106] The first training module 53 is also used to perform a first preset processing on part of the expression data in the first sample set; input the first sample set after the first preset processing into the first pre-training model for training until the first decoding model outputs the same data as the first sample set before the first preset processing, to obtain a second decoding model; and determine the first encoding model and the second decoding model as the second pre-training model.

[0107] The second training module 54 is also used to perform a second preset processing on part of the expression data in the second sample set; input the first sample set after the second preset processing into the second decoding model for training until the second decoding model outputs the same data as the second sample set before the second preset processing, to obtain a third decoding model; and determine the first encoding model and the third decoding model as expression conversion models.

[0108] The first training module 53 is further used to randomly adjust the expression base coefficients corresponding to the first expression data of some frames to zero.

[0109] The second training module 54 is further used to randomly adjust the expression base coefficients corresponding to the second expression data of some frames to zero.

[0110] The acquisition module 51 is also used to collect multiple frames of first expression data of the target object within a preset time period through a face capture tool, and determine the multiple frames of first expression data as a first sample set; and receive manual adjustments to the basic expression to obtain multiple frames of second expression data, and determine the multiple frames of second expression data as a second sample set; wherein the expression styles of the multiple frames of first expression data in the first sample set are different from those of the multiple frames of second expression data in the second sample set.

[0111] The above scheme can train an expression conversion model for style conversion based on expression data, and then greatly improve the speed and efficiency of expression conversion through the expression conversion model.

[0112] See also Figure 6 , Figure 6 It is a schematic diagram of the framework of an embodiment of the expression conversion device of the present application.

[0113] The expression conversion device 60 includes an acquisition module 61 and a conversion module 62. The acquisition module 61 is used to acquire the expression data to be converted; the conversion module 62 is used to input part of the expression data to be converted into the expression conversion model for conversion, and obtain the expression data after the complete expression style conversion;

[0114] The conversion module 62 is further configured to extract part of the expression data to be converted from the expression data to be converted based on the features of the expression data to be converted and / or the expression base coefficients.

[0115] The above solution can greatly improve the speed and efficiency of expression conversion through the expression conversion model.

[0116] See also Figure 7 , Figure 7: is a schematic diagram of a framework of an embodiment of an electronic device of the present application. The electronic device 70 includes a memory 71 and a processor 72 coupled to each other, and the processor 72 is used to execute program instructions stored in the memory 71 to implement the training method of any of the above expression conversion models and the steps of the expression conversion method. In a specific implementation scenario, the electronic device 70 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 70 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0117] Specifically, the processor 72 is used to control itself and the memory 71 to implement the steps of the training method embodiment of any of the above-mentioned expression conversion models. The processor 72 can also be called a CPU (Central Processing Unit). The processor 72 may be an integrated circuit chip with signal processing capabilities. The processor 72 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 72 can be implemented by an integrated circuit chip.

[0118] The above solution can greatly improve the speed and efficiency of expression conversion through the expression conversion model.

[0119] See also Figure 8 , Figure 8 The computer-readable storage medium 80 stores program instructions 801 that can be executed by a processor, and the program instructions 801 are used to implement the training method of any of the above expression conversion models and the steps of the expression conversion method.

[0120] The above solution can greatly improve the speed and efficiency of expression conversion through the expression conversion model.

[0121] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0122] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0123] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0124] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A training method for an expression conversion model, characterized in that: The training method of the expression conversion model includes: A first sample set and a second sample set are obtained; wherein the samples of the first sample set and the second sample set are expression data; wherein a plurality of frames of first expression data of the target object within a preset time period are collected by a face capture tool, and the plurality of frames of first expression data are determined as the first sample set; and manual adjustment of the basic expression is received to obtain a plurality of frames of second expression data, and the plurality of frames of second expression data are determined as the second sample set; the expression styles of the plurality of frames of first expression data in the first sample set and the plurality of frames of second expression data in the second sample set are different; Pre-training an initial model using the first sample set to obtain a first pre-trained model; Performing a first preset processing on the expression data of the first sample set, and inputting the first sample set after the first preset processing into the first pre-trained model for training to obtain a second pre-trained model; the first preset processing includes adjusting the expression base coefficient of the first sample set; The expression data of the second sample set is subjected to a second preset processing, and the second sample set after the second preset processing is input into the second pre-trained model for training, and the trained model is determined as the expression conversion model; the second preset processing includes adjusting the expression base coefficient of the second sample set.

2. The training method of the expression conversion model according to claim 1, characterized in that: The initial model includes an initial encoding model and an initial decoding model that are cascaded to each other; The step of pre-training the initial model using the first sample set to obtain the first pre-trained model comprises: Inputting the first sample set into the initial encoding model for encoding, and inputting the data encoded by the initial encoding model into the initial decoding model for decoding, until a result output by the initial decoding model is the same as the first sample set, thereby obtaining a first encoding model and a first decoding model; The first encoding model and the first decoding model are determined as the first pre-trained model.

3. The training method of the expression conversion model according to claim 2, characterized in that: The step of performing a first preset processing on the expression data of the first sample set, and inputting the first sample set after the first preset processing into the first pre-trained model for training to obtain the second pre-trained model comprises: performing the first preset processing on part of the expression data in the first sample set; Inputting the first sample set after the first preset processing into the first pre-trained model for training until the first decoding model outputs the same data as the first sample set before the first preset processing, thereby obtaining a second decoding model; The first encoding model and the second decoding model are determined as the second pre-trained model.

4. The training method of the expression conversion model according to claim 3, characterized in that: The steps of performing a second preset processing on the expression data of the second sample set, inputting the second sample set after the second preset processing into the second pre-trained model for training, and determining the trained model as the expression conversion model include: Performing the second preset processing on part of the expression data in the second sample set; Inputting the first sample set after the second preset processing into the second pre-trained model for training until the second decoding model outputs the same data as the second sample set before the second preset processing, thereby obtaining a third decoding model; The first encoding model and the third decoding model are determined as the expression conversion model.

5. The training method of the expression conversion model according to any one of claims 1 to 4, characterized in that: The first sample set includes a plurality of frames of first expression data; The step of performing a first preset process on the expression data of the first sample set comprises: The expression base coefficients corresponding to the first expression data of some frames are randomly adjusted to zero.

6. The training method of the expression conversion model according to any one of claims 1 to 4, characterized in that: The second sample set includes a plurality of frames of second expression data; The step of performing a second preset process on the expression data of the second sample set comprises: The expression base coefficients corresponding to the second expression data of some frames are randomly adjusted to zero.

7. A method for converting facial expressions, characterized in that: The expression conversion method comprises: Get the expression data to be converted; Inputting the expression data to be converted into the expression conversion model for conversion, and obtaining expression data after expression style conversion; Wherein, the expression transition model is obtained by training using the expression transition model training method described in any one of claims 1-6.

8. The expression conversion method according to claim 7, characterized in that: The step of inputting the expression data to be converted into the expression conversion model for conversion to obtain the expression data after the expression style conversion comprises: Extracting part of the expression data to be converted from the expression data to be converted based on the features of the expression data to be converted and / or the expression base coefficients; Part of the expression data to be converted is input into the expression conversion model for conversion to obtain complete expression data after expression style conversion.

9. A training device for an expression conversion model, characterized in that: The training device of the expression conversion model comprises: An acquisition module is used to acquire a first sample set and a second sample set; wherein the samples of the first sample set and the second sample set are expression data; wherein a plurality of frames of first expression data of a target object within a preset time period are collected by a face capture tool, and the plurality of frames of first expression data are determined as the first sample set; and an adjustment of a basic expression by manual means is received to obtain a plurality of frames of second expression data, and the plurality of frames of second expression data are determined as the second sample set; the expression styles of the plurality of frames of first expression data in the first sample set and the plurality of frames of second expression data in the second sample set are different; A pre-training module, used to pre-train the initial model using the first sample set to obtain a first pre-training model; A first training module, configured to perform a first preset processing on part of the expression data of the first sample set, and input the first sample set after the first preset processing into the first pre-training model for training to obtain a second pre-training model; the first preset processing includes adjusting the expression base coefficient of the first sample set; The second training module is used to perform a second preset processing on part of the expression data of the second sample set, and input the second sample set after the second preset processing into the second pre-training model for training, and determine the trained model as the expression conversion model; the second preset processing includes adjusting the expression base coefficient of the second sample set.

10. An expression conversion device, characterized in that: The expression conversion device comprises: An acquisition module is used to obtain expression data to be converted; A conversion module, used to input part of the expression data to be converted into the expression conversion model for conversion, so as to obtain the expression data after the complete expression style conversion; Wherein, the expression transition model is obtained by training using the expression transition model training method described in any one of claims 1-6.

11. An electronic device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the training method of the expression conversion model as described in any one of claims 1 to 6 or the expression conversion method as described in any one of claims 7 to 8.

12. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by the processor, the training method of the expression conversion model as described in any one of claims 1 to 6 or the expression conversion method as described in any one of claims 7 to 8 is implemented.

Images