Expression driving method, device, electronic device and storage medium

Abstract

The embodiment of the present invention provides an expression driving method, device, electronic device and storage medium, the method comprising: acquiring the features of a target facial image of a target object; calculating the expression amplitude parameter corresponding to the target facial image based on the features and the image features of each preset facial image of the target object extracted in advance; calculating the expression control coefficient corresponding to the target facial image based on the expression amplitude parameter and the control coefficient corresponding to each preset facial image acquired in advance, so that the expression driving controller controls the driven object to make the same expression as the target object in the target facial image based on the expression control coefficient. By adopting this method, a small number of preset facial images can be used to drive the driven object to make the same subtle expression as the target object, which not only improves the expression driving effect, so that the expression driving can be applied to scenes requiring subtle expressions, but also reduces the workload of the expression driving and improves the efficiency of the expression driving.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an expression driving method, device, electronic equipment and storage medium. Background Art

[0002] At present, expression-driven technology is widely used not only in the fields of game and film and television CG (Computer Animation) production, but also in the emerging virtual puppet industries such as virtual idols and virtual singers. Among them, virtual puppets such as virtual idols and virtual singers are produced through painting, music, animation and CG, and perform in virtual scenes or real scenes on the Internet, but they do not exist in physical form. Expression-driven is to make the driven object (such as a virtual puppet) synchronously make the same expression as the corresponding creature.

[0003] The existing expression driving method is to capture the facial expressions of the creatures corresponding to the driven object through the mobile phone camera, and use the captured expressions to drive the driven object. For example, FaceRig (an open creation platform for making characters) and Apple ARKit (Apple AR development platform) can enable the public to control a driven object (such as a virtual puppet) through their own mobile phone or an RGB camera to synchronize the same expression as their own.

[0004] However, this expression driving method uses some general expression driving algorithms, which can only drive relatively conservative expressions, so it is difficult to apply to scenes that require detailed expressions. In addition, the expression of the driven object driven by this expression driving method is not close enough to the expression of the corresponding creature, and the expression driving effect is poor. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide an expression driving method, device, electronic device and storage medium to improve the expression driving effect.

[0006] In a first aspect of the present invention, an expression driving method is first provided, comprising:

[0007] Acquire features of a target facial image of a target object;

[0008] Based on the features and the pre-extracted image features of each preset facial image of the target object, calculating the expression amplitude parameter corresponding to the target facial image; wherein each preset facial image is a facial image of the target object when it is in a different expression state;

[0009] Based on the expression amplitude parameter and the control coefficient corresponding to each preset facial image acquired in advance, the expression control coefficient corresponding to the target facial image is calculated, so that the expression drive controller controls the driven object to make the same expression as the target object in the target facial image based on the expression control coefficient; wherein the control coefficient corresponding to each preset facial image is: the parameter of the expression drive controller when the expression drive controller controls the driven object to make the same expression as the target object in the preset facial image.

[0010] Optionally, the calculating the expression amplitude parameter corresponding to the target facial image based on the feature and the pre-extracted image feature of each preset facial image of the target object includes:

[0011] Based on the features and the pre-extracted image features of each preset facial image of the target object, the expression amplitude parameter corresponding to the target facial image is calculated using the following formula:

[0012] min||Aw i -x||,stw i ≥0

[0013] Among them, w i is the i-th expression amplitude parameter corresponding to the target facial image, 1≤i≤n; a0, a1, …, a n are image features corresponding to n+1 preset facial images respectively, a0 is the image feature of the preset facial image when the target object is in a neutral expression state, a0, a1, ..., a n The corresponding transposed matrix: x=a-a0, where a is the feature of the target facial image.

[0014] Optionally, the calculating the expression control coefficient corresponding to the target facial image based on the expression amplitude parameter and the control coefficient corresponding to each preset facial image acquired in advance includes:

[0015] Based on the expression amplitude parameter and the control coefficient corresponding to each preset facial image acquired in advance, the expression control coefficient corresponding to the target facial image is calculated using the following formula:

[0016]

[0017] Wherein, B is the expression control coefficient corresponding to the target facial image; B0 is the control coefficient corresponding to the preset facial image when the target object is in a neutral expression state, and B iis the control coefficient corresponding to the i-th preset facial image, and n is the number of preset facial images excluding the preset facial image when the target object is in a neutral expression state.

[0018] Optionally, the acquiring features of the target facial image of the target object includes:

[0019] The coordinate information of facial key points in the target facial image of the target object is obtained as the features of the target facial image.

[0020] In another aspect of the present invention, an expression driving device is provided, comprising:

[0021] A feature acquisition module, used to acquire features of a target facial image of a target object;

[0022] an amplitude parameter calculation module, used to calculate the expression amplitude parameter corresponding to the target facial image based on the feature and the image features of each preset facial image of the target object extracted in advance; wherein each preset facial image is a facial image of the target object when it is in a different expression state;

[0023] A control coefficient calculation module is used to calculate the expression control coefficient corresponding to the target facial image based on the expression amplitude parameter and the control coefficient corresponding to each preset facial image obtained in advance, so that the expression drive controller controls the driven object to make the same expression as the target object in the target facial image based on the expression control coefficient; wherein the control coefficient corresponding to each preset facial image is: the parameter of the expression drive controller when the expression drive controller controls the driven object to make the same expression as the target object in the preset facial image.

[0024] Optionally, the amplitude parameter calculation module is specifically used to calculate the expression amplitude parameter corresponding to the target facial image based on the feature and the pre-extracted image feature of each preset facial image of the target object using the following formula:

[0025] min||Aw i -x||,stw i ≥0

[0026] Among them, w i is the i-th expression amplitude parameter corresponding to the target facial image, 1≤i≤n; a0, a1, …, a n are image features corresponding to n+1 preset facial images respectively, a0 is the image feature of the preset facial image when the target object is in a neutral expression state, a0, a1, ..., a nThe corresponding transposed matrix: x=a-a0, where a is the feature of the target facial image.

[0027] Optionally, the control coefficient calculation module is specifically used to calculate the expression control coefficient corresponding to the target facial image using the following formula based on the expression amplitude parameter and the control coefficient corresponding to each preset facial image acquired in advance:

[0028]

[0029] Wherein, B is the expression control coefficient corresponding to the target facial image; B0 is the control coefficient corresponding to the preset facial image when the target object is in a neutral expression state, and B i is the control coefficient corresponding to the i-th preset facial image, and n is the number of preset facial images excluding the preset facial image when the target object is in a neutral expression state.

[0030] Optionally, the feature acquisition module is specifically used to acquire coordinate information of facial key points in a target facial image of a target object as features of the target facial image.

[0031] In another aspect of the present invention, there is provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0032] Memory, used to store computer programs;

[0033] The processor is used to implement any of the above-mentioned expression driving method steps when executing the program stored in the memory.

[0034] In another aspect of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any of the above-mentioned expression driving method steps is implemented.

[0035] In another aspect of the present invention, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned expression-driven method steps.

[0036] According to the method provided by the embodiment of the present invention, the features of the target facial image of the target object are obtained; based on the features and the image features of each preset facial image of the target object extracted in advance, the expression amplitude parameter corresponding to the target facial image is calculated; based on the expression amplitude parameter and the control coefficient corresponding to each preset facial image obtained in advance, the expression control coefficient corresponding to the target facial image is calculated, so that the expression driving controller controls the driven object to make the same expression as the target object in the target facial image based on the expression control coefficient. Since some minor expressions of the target object can be regarded as expressions obtained by changing the amplitude of the large-amplitude expression and superimposing it with the neutral expression, for example, the expression of the target object with a slightly open mouth can be regarded as the expression of the target object with a wide mouth reduced by reducing the expression amplitude (reducing the amplitude of the mouth opening) and superimposing it with the neutral expression (the facial state of the target object when it does not make any expression), and the expression amplitude change and the expression control coefficient both have a linear interpolation relationship, that is, the expression of the target object can be changed by changing the expression amplitude of the target object, and the expression of the driven object driven by the expression control coefficient can be changed by changing the expression control coefficient, so that the driven object makes the same expression as the target object. Therefore, the method of the embodiment of the present invention can use the image features of multiple preset facial images to calculate the expression amplitude parameters corresponding to the target facial image, and then use the interpolation relationship between the expression amplitude change of the target object and the expression control coefficient of the driven object, and calculate the expression control coefficient corresponding to the target facial image according to the expression amplitude parameter and the control coefficient corresponding to each preset facial image, and then use the expression control coefficient to make the expression drive controller control the driven object to make the same expression as the target object in the target facial image. Therefore, the expression control coefficient is used to make the expression made by the driven object very close to the expression of the target object in the target facial image, thereby improving the expression drive effect. The method provided by the embodiment of the present invention can drive the driven object to make the same subtle expression as the target object using a small number of preset facial images, which not only improves the expression drive effect, so that the expression drive can be applied to scenes that require delicate expressions, but also reduces the workload of the expression drive and improves the efficiency of the expression drive. In addition, the calculation of the expression amplitude parameter corresponding to the target facial image and the calculation of the expression control coefficient corresponding to the target facial image only need to solve linear problems, and the solution speed can reach milliseconds, so that the expression drive method provided by the embodiment of the present invention can be applied to real-time expression drive scenes, that is, the application scenes of expression drive are expanded.

[0037] Of course, it is not necessary to achieve all of the advantages described above at the same time to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0039] Figure 1 A flow chart of the expression driving method provided by an embodiment of the present invention;

[0040] Figure 2 It is a schematic diagram of superimposing expressions of different magnitudes;

[0041] Figure 3 A schematic diagram of optimal projection coefficients of the expression of the target object in the target facial image onto the expression of the target object in each preset facial image;

[0042] Figure 4 A schematic diagram of an expression driving system provided by an embodiment of the present invention;

[0043] Figure 5 A structural diagram of an expression driving device provided by an embodiment of the present invention

[0044] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0046] Since the existing expression driving method uses some general expression driving algorithms, it can only drive relatively conservative expressions, so it is difficult to apply to scenes that require delicate expressions. In order to improve the expression driving effect, an embodiment of the present invention provides an expression driving method, device, electronic device and storage medium. The expression driving method provided by the embodiment of the present invention can be applied to any electronic device capable of image processing, such as computers, mobile phone terminals and iPads, etc., without specific limitation here.

[0047] The expression driving method provided by the embodiment of the present invention is introduced in detail below.

[0048] Figure 1 A flow chart of the expression driving method provided by an embodiment of the present invention, such as Figure 1 As shown, the method includes:

[0049] Step 101, acquiring features of a target facial image of a target object.

[0050] Step 102: Calculate expression amplitude parameters corresponding to the target facial image based on the features and pre-extracted image features of each preset facial image of the target object.

[0051] Each preset facial image is a facial image of the target object in a different expression state.

[0052] Step 103, based on the expression amplitude parameter and the control coefficient corresponding to each preset facial image obtained in advance, the expression control coefficient corresponding to the target facial image is calculated, so that the expression drive controller controls the driven object to make the same expression as the target object in the target facial image based on the expression control coefficient.

[0053] The control coefficient corresponding to each preset facial image is: a parameter of the expression driving controller when the expression driving controller controls the driven object to make the same expression as the target object in the preset facial image.

[0054] According to the method provided by the embodiment of the present invention, the features of the target facial image of the target object are obtained; based on the features and the image features of each preset facial image of the target object extracted in advance, the expression amplitude parameter corresponding to the target facial image is calculated; based on the expression amplitude parameter and the control coefficient corresponding to each preset facial image obtained in advance, the expression control coefficient corresponding to the target facial image is calculated, so that the expression driving controller controls the driven object to make the same expression as the target object in the target facial image based on the expression control coefficient. Since some minor expressions of the target object can be regarded as expressions obtained by changing the amplitude of the large-amplitude expression and superimposing it with the neutral expression, for example, the expression of the target object with a slightly open mouth can be regarded as the expression of the target object with a wide mouth reduced by reducing the expression amplitude (reducing the amplitude of the mouth opening) and superimposing it with the neutral expression (the facial state of the target object when it does not make any expression), and the expression amplitude change and the expression control coefficient both have a linear interpolation relationship, that is, the expression of the target object can be changed by changing the expression amplitude of the target object, and the expression of the driven object driven by the expression control coefficient can be changed by changing the expression control coefficient, so that the driven object makes the same expression as the target object. Therefore, the method of the embodiment of the present invention can use the image features of multiple preset facial images to calculate the expression amplitude parameters corresponding to the target facial image, and then use the interpolation relationship between the expression amplitude change of the target object and the expression control coefficient of the driven object, and calculate the expression control coefficient corresponding to the target facial image according to the expression amplitude parameter and the control coefficient corresponding to each preset facial image, and then use the expression control coefficient to make the expression drive controller control the driven object to make the same expression as the target object in the target facial image. Therefore, the expression control coefficient is used to make the expression made by the driven object very close to the expression of the target object in the target facial image, thereby improving the expression drive effect. The method provided by the embodiment of the present invention can drive the driven object to make the same subtle expression as the target object using a small number of preset facial images, which not only improves the expression drive effect, so that the expression drive can be applied to scenes that require delicate expressions, but also reduces the workload of the expression drive and improves the efficiency of the expression drive. In addition, the calculation of the expression amplitude parameter corresponding to the target facial image and the calculation of the expression control coefficient corresponding to the target facial image only need to solve linear problems, and the solution speed can reach milliseconds, so that the expression drive method provided by the embodiment of the present invention can be applied to real-time expression drive scenes, that is, the application scenes of expression drive are expanded.

[0055] In the embodiment of the present invention, the target object may be a person, an animal, or a doll that can produce expression changes, and the driven object may be a virtual doll. The virtual doll is a character image that is produced in the form of painting, music, animation, CG, etc., and performs performing activities in a virtual scene or a real scene on the Internet, but does not exist in a physical form. The virtual doll includes but is not limited to a virtual idol and a virtual singer. The essence of expression drive is to make the driven object synchronously make the same expression as the target object. In addition to changes in expression types such as closing eyes and opening mouth, the expression of the target object also has amplitude changes. For example, the amplitude of the expression of slightly opening the corners of the mouth is different from that of the expression of widening the mouth, and the small amplitude expression can be regarded as the expression obtained by superimposing a large amplitude expression and a neutral expression. For example, the expression of slightly opening the corners of the mouth can be regarded as the expression obtained by superimposing an expression of widening the mouth and a neutral expression, wherein the neutral expression refers to the facial state of the target object when it does not make any expression.

[0056] There is a linear interpolation relationship between the change in expression amplitude and the expression control coefficient, that is, the expression of the target object can be changed by changing the expression amplitude of the target object, and the expression of the driven object driven by the expression control coefficient can be changed by changing the expression control coefficient, so that the driven object can make the same expression as the target object. Figure 2 A schematic diagram of superimposing expressions of different magnitudes, such as Figure 2 As shown, Figure 2 (1) is the facial state of the target object 201 when it does not make any expression. Figure 2 (2) in FIG. 2 is the facial state of the target object 201 when the target object 201 makes an expression of closing eyes and pursing lips. Figure 2 (3) in FIG. 2 is the facial state of the target object 201 when the target object 201 makes an expression with a wide mouth and wide eyes. Figure 2 (2) and Figure 2 (3) in the example shows that the target object 201 makes a large expression, and Figure 2 (2) and Figure 2 In (3) of FIG. 1 , the target object 201 makes a large expression, which is exactly the opposite trend. Therefore, Figure 2 As shown, it can be Figure 2 (2) and Figure 2 The expressions in (3) are superimposed to obtain Figure 2 (1) the neutral expression of the target object 201, and Figure 2 (2) and Figure 2 The expression amplitude parameters of the expressions in (3) are all 0.5.

[0057] Since small expressions can be regarded as the superposition of large expressions and neutral expressions, such as Figure 2 As shown, Figure 2 (1) can be obtained by doubling Figure 2 Neutral expression in (2) and Figure 2 It is obtained by superimposing the large-scale expression (3) in the figure.

[0058] In an embodiment of the present invention, n+1 facial images of the target object in different expression states can be collected in advance using an RGB camera as preset facial images. Specifically, n facial images of the target object in an exaggerated or extreme expression state and facial images of the target object in a neutral expression state can be selected as preset facial images. For example, facial images of the target object with a neutral expression, closed left eye, closed right eye, wide mouth, and pursed lips are collected as preset facial images. Wherein, n is a natural number greater than or equal to 1, and the specific value of n can be set according to the actual application scenario. The other expressions of the target object can be regarded as a combination of different amplitudes of these extreme and exaggerated expressions.

[0059] For example, Figure 2 For the target object 201 shown in FIG. 1 , n can be set to 2, and then two facial images of the target object 201 in an exaggerated or extreme expression state and a facial image of the target object 201 in a neutral expression state can be selected. Figure 2 (1), (2) and (3) in the figure are used as the preset facial images of the target object 201.

[0060] In the embodiment of the present invention, after collecting n+1 preset facial images of the target object, for each preset facial image, the expression drive controller can be used to control the driven object to make the same expression as the target object in the preset facial image, and then the parameters of the expression drive controller when the driven object makes the same expression as the target object in the preset facial image are determined as the control coefficient corresponding to the preset facial image. Then, the combination of the control coefficients corresponding to the n+1 preset facial images of the target object can be obtained as {B0, B1, B2, …, B n}, where B0 is the control coefficient corresponding to the preset facial image when the target object is in a neutral expression state.

[0061] In an embodiment of the present invention, the image features of each preset facial image of the target object may also be pre-extracted. Specifically, since there is a lot of information irrelevant to the expression of the target object in the collected preset facial image, such as background and clothing, in order to reduce the interference of this information on the expression of the target object, the coordinate information of the key facial points in the preset facial image or the 3DMM (3D Morphable Face Model) coefficients may be extracted for each preset facial image of the target object as the image features of the preset facial image. Among them, the key facial points may be points on the facial contour, nose, eyes, mouth, eyebrows and other parts of the target object. Alternatively, in an embodiment of the present invention, for each preset facial image of the target object, the contour information of the facial features of the target object in the preset facial image may also be extracted as the image features of the preset facial image. The set of image features of the preset facial image of the target object may be recorded as {a0, a1, a2,…, a n}, where a0 is the image feature of the preset facial image when the target object is in a neutral expression state. In the embodiment of the present invention, the image feature of the preset facial image of the target object can be stored in a specific matrix form: That is, each row can be represented by the image features of the preset facial image when the target object is in a non-neutral expression state minus the image features of the preset facial image when the target object is in a neutral expression state, forming a set of expression bases, that is, A represents a set of expression bases of the target object.

[0062] Among them, the image features of each preset facial image of the target object extracted in advance are consistent with the features of the target facial image of the acquired target object, that is, if the coordinate information of the facial key points in each preset facial image is extracted in advance as the image features of the preset facial image, then the features of the target facial image of the acquired target object should be the coordinate information of the facial key points in the target facial image; if the facial contour information of the target object in each preset facial image is extracted in advance as the image features of the preset facial image, then the features of the target facial image of the acquired target object should be the facial contour information of the target object in the target facial image.

[0063] In the embodiment of the present invention, after executing the step of acquiring the features of the target facial image of the target object, the step of extracting the image features of each preset facial image of the target object and the step of acquiring the control coefficient corresponding to each preset facial image may be executed.

[0064] In a possible implementation, the calculating the expression amplitude parameter corresponding to the target facial image based on the feature and the image feature of each preset facial image of the target object extracted in advance includes: calculating the expression amplitude parameter corresponding to the target facial image based on the feature and the image feature of each preset facial image of the target object extracted in advance using the following formula:

[0065] min||Aw i -x||,stw i ≥0

[0066] Among them, w i is the i-th expression amplitude parameter corresponding to the target facial image, 1≤i≤n; a0, a1, …, a n are the image features corresponding to the n+1 preset facial images respectively, a0 is the image feature of the preset facial image when the target object is in a neutral expression state, a0, a1, ..., a n The corresponding transposed matrix: x=a-a0, where a is the feature of the target facial image.

[0067] Figure 3 is a schematic diagram of the optimal projection coefficient of the expression of the target object in the target facial image on the expression of the target object in each preset facial image, such as Figure 3 As shown in the figure, the expression of the target object in the target facial image can be obtained by multiplying the expression of the target object in each preset facial image by the corresponding projection coefficient and then subtracting the neutral expression of the target object and then calculating the norm. When the norm reaches the minimum, each projection coefficient w1-w n These are the expression amplitude parameters corresponding to the target facial image.

[0068] In a possible implementation, the step of calculating the expression control coefficient corresponding to the target facial image based on the expression amplitude parameter and the control coefficient corresponding to each preset facial image obtained in advance includes: calculating the expression control coefficient corresponding to the target facial image using the following formula based on the expression amplitude parameter and the control coefficient corresponding to each preset facial image obtained in advance:

[0069]

[0070] Among them, B is the expression control coefficient corresponding to the target facial image; B0 is the control coefficient corresponding to the preset facial image when the target object is in a neutral expression state, and B i is the control coefficient corresponding to the i-th preset facial image, and n is the number of preset facial images excluding the preset facial image when the target object is in a neutral expression state.

[0071] In the embodiment of the present invention, the target facial image may be any facial image of a target object captured by an RGB camera or other image acquisition devices.

[0072] In the embodiment of the present invention, after obtaining the expression control coefficient corresponding to the target facial image, the parameters of the expression drive controller can be adjusted to the expression control coefficient to control the driven object to make the same expression as the target object in the target facial image. The expression drive controller can be any electronic device that can control the driven object to make the same expression as the target object, and is not specifically limited here.

[0073] For example, Figure 2 For the target object 201 shown, if you want to drive the virtual doll to make the expression on the target facial image of the target object 201 (the expression of the target object 201 closing eyes and opening the mouth), the specific operation method can be referred to as follows:

[0074] Multiple facial images of the target object 201 can be collected in advance by an RGB camera. Then n can be set to 2, and then one of the multiple facial images of the target object 201 collected can be selected. Figure 2 (1), (2) and (3) are preset facial images of the target object 201. Figure 2 In (2) and (3), the target object 201 is in an exaggerated and extreme expression state. Figure 2 (1) The target object 201 is in a neutral expression state. Then, the 3DMM coefficients of each preset facial image of the target object 201 can be extracted as the image features of the preset facial image and recorded as a set {a (201)0 , a (201)1 , a (201)2} Among them, a (201)0 is a preset facial image of the target object 201 when the target object 201 is in a neutral expression state (i.e. Figure 2 The image features of (1)), a (201)1 For target object 201 in Figure 2 The image features of the preset facial image in the expression state shown in (2) in FIG. (201)2 For target object 201 in Figure 2 The image features of the preset facial image in the expression state shown in (3).

[0075] Then, the image features of the three preset facial images of the target object 201 may be stored in a matrix form: A (201) A set of expression bases representing the target object 201, They are respectively (201)0 , a (201)1 , a (201)2Then, the expression amplitude parameter w corresponding to the target facial image of the target object 201 can be calculated using the following formula: (201)i :

[0076] min||A (201) w (201)i -x (201) ||,stw (201)i ≥0

[0077] Among them, w (201)i is the i-th expression amplitude parameter, x (201) =a (201) -a (201)0 , a (201) is the feature of the target facial image of the target object 201. Then the expression amplitude parameter w corresponding to the target facial image of the target object 201 can be calculated by this formula (201)1 and w (201)2 .

[0078] Then, the expression control coefficient B corresponding to the target facial image can be calculated using the following formula: (201) :

[0079]

[0080] Among them, B (201)0 is the control coefficient corresponding to the preset facial image of the target object 201 when the target object 201 is in a neutral expression state (i.e. Figure 2 (1) corresponding control coefficient), B (201)1 for Figure 2 (2) Corresponding control coefficient, B (201)2 for Figure 2 (3) Corresponding control coefficient.

[0081] Then, the parameters of the expression drive controller can be adjusted to the expression control coefficient B (201) , control the virtual doll to make the same expression as in the target facial image of the target object 201, that is, control the virtual doll to make an expression with eyes closed and mouth wide open.

[0082] By adopting the method provided by the embodiment of the present invention, the expression amplitude parameters corresponding to the target facial image can be calculated using the image features of multiple preset facial images, and then the expression amplitude change of the target object and the expression control coefficient of the driven object are used to calculate the expression control coefficient corresponding to the target facial image according to the expression amplitude parameters and the control coefficient corresponding to each preset facial image, and then the expression control coefficient is used to enable the expression drive controller to control the driven object to make the same expression as the target object in the target facial image. The method provided by the embodiment of the present invention can drive the driven object to make the same subtle expression as the target object by using a small number of preset facial images, thereby improving the expression driving effect, so that the expression driving can be applied to scenes that require subtle expressions. Moreover, compared with the traditional expression driving method that drives the driven object by annotating a large number of biological facial images and then capturing expressions, the method provided by the embodiment of the present invention does not need to annotate a large number of biological facial images, but only requires a small number of preset facial images of the target object making extreme expressions, and uses the amplitude change of the expression to calculate the amplitude change parameters between the target facial image and these preset facial images based on the small number of preset facial images of the target object, and then determines the expression control coefficient according to the amplitude change parameters, and then can directly drive the driven object to make the same expression as the target object through the expression control coefficient, thereby reducing the amount of annotated data, that is, reducing the workload of expression driving, and improving the efficiency of expression driving.

[0083] Moreover, calculating the expression amplitude parameter corresponding to the target facial image and calculating the expression control coefficient corresponding to the target facial image only requires solving a linear problem, and the solving speed can reach the millisecond level, so that the expression driving method provided by the embodiment of the present invention can be applied to real-time expression driving scenes, that is, the application scenes of expression driving are expanded. Moreover, since the control coefficient and the features of the facial image can be adjusted according to different application scenes, the expression driving method provided by the embodiment of the present invention is not limited to the expression driving controller or the facial image feature standard, and has strong flexibility.

[0084] In addition, the method provided by the embodiment of the present invention only requires a small number of preset facial images and control coefficients of the target object to achieve high-precision expression driving for the target object. Compared with the general plug-and-play solution, because the data is collected for a specific target object, the expression of the driven object can be more detailed and closer to the expression of the target object; on the other hand, because the control coefficient corresponding to the preset facial image of the target object can be freely provided, the expression driving method provided by the embodiment of the present invention can be customized by the target object without meeting specific specifications, and has a higher degree of freedom.

[0085] Figure 4 A schematic diagram of an expression driving system provided by an embodiment of the present invention, such as Figure 4As shown, the expression driving system may include an RGB camera 401, a parameter determination module 402, and an expression driving module 403. The RGB camera may be used to collect multiple key expressions of the target object, and the key expressions are exaggeration, extreme expression, and neutral expression of the target object. Then the parameter determination module may obtain multiple key expressions of the target object collected by the RGB camera, and drive the virtual puppet to make the same expression as the key expression based on the key expression, and obtain the parameters when the expression driving controller controls the virtual puppet to make the same expression as the key expression, as the control coefficient corresponding to the key expression. Then the parameter determination module sends the control coefficient corresponding to the key expression to the expression driving module, and the expression driving module may obtain any expression of the target object from the RGB camera in real time, determine the control coefficient corresponding to any expression of the target object using the control coefficient corresponding to the key expression, and then adjust the parameters of the expression driving controller to the control coefficient corresponding to the arbitrary expression, control the virtual puppet to make the same expression as the arbitrary expression, and realize the expression driving of the virtual puppet.

[0086] Based on the same inventive concept, according to the expression driving method provided by the above embodiment of the present invention, correspondingly, another embodiment of the present invention further provides an expression driving device, whose structural schematic diagram is shown in FIG. Figure 5 As shown, specifically including:

[0087] A feature acquisition module 501 is used to acquire features of a target facial image of a target object;

[0088] A magnitude parameter calculation module 502 is used to calculate the expression magnitude parameter corresponding to the target facial image based on the feature and the image feature of each preset facial image of the target object extracted in advance; wherein each preset facial image is a facial image of the target object when it is in a different expression state;

[0089] The control coefficient calculation module 503 is used to calculate the expression control coefficient corresponding to the target facial image based on the expression amplitude parameter and the control coefficient corresponding to each preset facial image obtained in advance, so that the expression drive controller controls the driven object to make the same expression as the target object in the target facial image based on the expression control coefficient; wherein the control coefficient corresponding to each preset facial image is: the parameter of the expression drive controller when the expression drive controller controls the driven object to make the same expression as the target object in the preset facial image.

[0090] The device provided by the embodiment of the present invention is used to obtain the features of the target facial image of the target object; based on the features and the image features of each preset facial image of the target object extracted in advance, the expression amplitude parameter corresponding to the target facial image is calculated; based on the expression amplitude parameter and the control coefficient corresponding to each preset facial image obtained in advance, the expression control coefficient corresponding to the target facial image is calculated, so that the expression driving controller controls the driven object to make the same expression as the target object in the target facial image based on the expression control coefficient. Since some minor expressions of the target object can be regarded as expressions obtained by changing the amplitude of the large-amplitude expression and superimposing it with the neutral expression, for example, the expression of the target object with a slightly open mouth can be regarded as the expression of the target object with a wide mouth reduced by reducing the expression amplitude (reducing the amplitude of the mouth opening) and superimposed with the neutral expression (the facial state of the target object when no expression is made), and the expression amplitude change and the expression control coefficient both have a linear interpolation relationship, that is, the expression of the target object can be changed by changing the expression amplitude of the target object, and the expression of the driven object driven by the expression control coefficient can be changed by changing the expression control coefficient, so that the driven object makes the same expression as the target object. Therefore, the device provided by the embodiment of the present invention can use the image features of multiple preset facial images to calculate the expression amplitude parameters corresponding to the target facial image, and then use the interpolation relationship between the expression amplitude change of the target object and the expression control coefficient of the driven object to calculate the expression control coefficient corresponding to the target facial image according to the expression amplitude parameter and the control coefficient corresponding to each preset facial image, and then use the expression control coefficient to make the expression drive controller control the driven object to make the same expression as the target object in the target facial image. The device provided by the embodiment of the present invention can drive the driven object to make the same subtle expression as the target object using a small number of preset facial images, which not only improves the expression drive effect, so that the expression drive can be applied to scenes that require delicate expressions, but also reduces the workload of the expression drive and improves the efficiency of the expression drive. In addition, the calculation of the expression amplitude parameters corresponding to the target facial image and the calculation of the expression control coefficient corresponding to the target facial image only require solving linear problems, and the solution speed can reach the millisecond level, so that the expression drive device provided by the embodiment of the present invention can be applied to real-time expression drive scenes, that is, the application scenes of expression drive are expanded.

[0091] Optionally, the amplitude parameter calculation module 502 is specifically configured to calculate the expression amplitude parameter corresponding to the target facial image using the following formula based on the feature and the pre-extracted image feature of each preset facial image of the target object:

[0092] min||Aw i -x||,stw i ≥0

[0093] Among them, wi is the i-th expression amplitude parameter corresponding to the target facial image, 1≤i≤n; a0, a1, …, a n are image features corresponding to n+1 preset facial images respectively, a0 is the image feature of the preset facial image when the target object is in a neutral expression state, a0, a1, ..., a n The corresponding transposed matrix: x=a-a0, where a is the feature of the target facial image.

[0094] Optionally, the control coefficient calculation module 503 is specifically configured to calculate the expression control coefficient corresponding to the target facial image using the following formula based on the expression amplitude parameter and the control coefficient corresponding to each preset facial image acquired in advance:

[0095]

[0096] Wherein, B is the expression control coefficient corresponding to the target facial image; B0 is the control coefficient corresponding to the preset facial image when the target object is in a neutral expression state, and B i is the control coefficient corresponding to the i-th preset facial image, and n is the number of preset facial images excluding the preset facial image when the target object is in a neutral expression state.

[0097] Optionally, the feature acquisition module 501 is specifically configured to acquire coordinate information of facial key points in a target facial image of a target object as features of the target facial image.

[0098] It can be seen that by using the device provided by the embodiment of the present invention, a small number of preset facial images can be used to drive the driven object to make the same subtle expression as the target object, which not only improves the expression driving effect, so that the expression driving can be applied to scenes that require delicate expressions, but also reduces the workload of the expression driving and improves the efficiency of the expression driving. In addition, the expression driving device provided by the embodiment of the present invention is not limited to the expression driving controller or the facial image feature standard, and has strong flexibility. Only a small number of preset facial images and control coefficients of the target object are required to achieve high-precision expression driving for the target object. Compared with the plug-and-play general solution, because data is collected for a specific target object, the expression of the driven object can be made more delicate and closer to the expression of the target object; on the other hand, because the control coefficient corresponding to the preset facial image of the target object can be freely provided, the expression driving device provided by the embodiment of the present invention can be customized by the target object without meeting specific specifications, and has a higher degree of freedom.

[0099] The embodiment of the present invention further provides an electronic device, such as Figure 6As shown, it includes a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0100] Memory 603, used for storing computer programs;

[0101] The processor 601 is used to execute the program stored in the memory 603 to implement the following steps:

[0102] Acquire features of a target facial image of a target object;

[0103] Based on the features and the pre-extracted image features of each preset facial image of the target object, calculating the expression amplitude parameter corresponding to the target facial image; wherein each preset facial image is a facial image of the target object when it is in a different expression state;

[0104] Based on the expression amplitude parameter and the control coefficient corresponding to each preset facial image acquired in advance, the expression control coefficient corresponding to the target facial image is calculated, so that the expression drive controller controls the driven object to make the same expression as the target object in the target facial image based on the expression control coefficient; wherein the control coefficient corresponding to each preset facial image is: the parameter of the expression drive controller when the expression drive controller controls the driven object to make the same expression as the target object in the preset facial image.

[0105] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0106] The communication interface is used for communication between the above electronic device and other devices.

[0107] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0108] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0109] In another embodiment of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the expression driving method described in any one of the above embodiments is implemented.

[0110] In another embodiment of the present invention, a computer program product including instructions is provided, which, when executed on a computer, enables the computer to execute the expression driving method described in any one of the above embodiments.

[0111] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0112] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0113] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for devices, electronic devices, storage media, and computer program products, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0114] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. An expression driving method, characterized in that: include: Acquire features of a target facial image of a target object; Based on the features and the pre-extracted image features of each preset facial image of the target object, calculating the expression amplitude parameter corresponding to the target facial image; wherein each preset facial image is a facial image of the target object when it is in a different expression state; Based on the expression amplitude parameter and the control coefficient corresponding to each preset facial image acquired in advance, the expression control coefficient corresponding to the target facial image is calculated, so that the expression drive controller controls the driven object to make the same expression as the target object in the target facial image based on the expression control coefficient; wherein the control coefficient corresponding to each preset facial image is: the parameter of the expression drive controller when the expression drive controller controls the driven object to make the same expression as the target object in the preset facial image; The step of calculating the expression amplitude parameter corresponding to the target facial image based on the feature and the pre-extracted image feature of each preset facial image of the target object comprises: Based on the features and the pre-extracted image features of each preset facial image of the target object, the expression amplitude parameter corresponding to the target facial image is calculated using the following formula: min||Aw i -x||,s.t.w i ≥0 Among them, w i is the i-th expression amplitude parameter corresponding to the target facial image, 1≤i≤n; a0, a1, …, a n are image features corresponding to n+1 preset facial images respectively, a0 is the image feature of the preset facial image when the target object is in a neutral expression state, are a0, a1, ..., a n The corresponding transposed matrix: x=a-a0, where a is the feature of the target facial image.

2. The method according to claim 1, characterized in that The step of calculating the expression control coefficient corresponding to the target facial image based on the expression amplitude parameter and the control coefficient corresponding to each preset facial image acquired in advance comprises: Based on the expression amplitude parameter and the control coefficient corresponding to each preset facial image acquired in advance, the expression control coefficient corresponding to the target facial image is calculated using the following formula: Wherein, B is the expression control coefficient corresponding to the target facial image; B0 is the control coefficient corresponding to the preset facial image when the target object is in a neutral expression state, and B i is the control coefficient corresponding to the i-th preset facial image, and n is the number of preset facial images excluding the preset facial image when the target object is in a neutral expression state.

3. The method according to any one of claims 1 to 2, characterized in that: The step of obtaining the features of the target facial image of the target object includes: The coordinate information of the facial key points in the target facial image of the target object is obtained as the features of the target facial image.

4. An expression driving device, characterized in that: include: A feature acquisition module, used to acquire features of a target facial image of a target object; an amplitude parameter calculation module, used to calculate the expression amplitude parameter corresponding to the target facial image based on the feature and the image features of each preset facial image of the target object extracted in advance; wherein each preset facial image is a facial image of the target object when it is in a different expression state; A control coefficient calculation module is used to calculate the expression control coefficient corresponding to the target facial image based on the expression amplitude parameter and the control coefficient corresponding to each preset facial image acquired in advance, so that the expression drive controller controls the driven object to make the same expression as the target object in the target facial image based on the expression control coefficient; wherein the control coefficient corresponding to each preset facial image is: the parameter of the expression drive controller when the expression drive controller controls the driven object to make the same expression as the target object in the preset facial image; The amplitude parameter calculation module is specifically used to calculate the expression amplitude parameter corresponding to the target facial image based on the feature and the pre-extracted image feature of each preset facial image of the target object using the following formula: min||Aw i -x||,s.t.w i ≥0 Among them, w i is the i-th expression amplitude parameter corresponding to the target facial image, 1≤i≤n; a0, a1, …, a n are image features corresponding to n+1 preset facial images respectively, a0 is the image feature of the preset facial image when the target object is in a neutral expression state, are a0, a1, ..., a n The corresponding transposed matrix: x=a-a0, where a is the feature of the target facial image.

5. The device according to claim 4, characterized in that The control coefficient calculation module is specifically used to calculate the expression control coefficient corresponding to the target facial image using the following formula based on the expression amplitude parameter and the control coefficient corresponding to each preset facial image acquired in advance: Wherein, B is the expression control coefficient corresponding to the target facial image; B0 is the control coefficient corresponding to the preset facial image when the target object is in a neutral expression state, and B i is the control coefficient corresponding to the i-th preset facial image, and n is the number of preset facial images excluding the preset facial image when the target object is in a neutral expression state.

6. The device according to any one of claims 4 to 5, characterized in that: The feature acquisition module is specifically used to acquire the coordinate information of facial key points in the target facial image of the target object as the features of the target facial image.

7. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 3 when executing a program stored in a memory.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 3 are implemented.

Images