Expression generation method, device, electronic device, storage medium and program product

Abstract

The present disclosure provides an expression generation method, device, electronic device, computer-readable storage medium and program product, which relate to the field of image processing technology. The method includes: obtaining a face image, determining a first expression parameter and a second expression parameter corresponding to the face image; the first expression parameter and the second expression parameter are obtained based on different expression extraction methods, respectively, and the expression extraction methods have corresponding expression bases; obtaining a pre-constructed state evaluation function; the state evaluation function is constructed based on the expression state associated with each expression base, and the expression state is a semantic description of the corresponding expression coefficient; based on the state evaluation function, the first expression parameter and the second expression parameter are fused to obtain a fused expression parameter, so as to drive expression generation according to the fused expression parameter. The present disclosure can fuse the expression parameters obtained by different expression extraction methods, so that the determined expression parameters have both robustness and flexibility.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an expression generation method, an expression generation device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Expression recognition is an important part of virtual object driving, and the flexibility and stability of the recognized expression parameters are important considerations for virtual object performance. At present, the industry usually uses a method based on a three-dimensional (3D) structured light camera to restore the 3D of the head and then collect expressions, such as mobile phone cameras or more professional expression capture equipment. However, the disadvantage of this type of equipment is that it is relatively expensive and cannot be too far away from the head of the person. When using a monocular camera for expression collection, because the monocular camera is cheaper, easier to popularize, and can adapt to a longer distance. However, the driving effect of a monocular camera is often inferior to that of a structured light solution in terms of flexibility and stability. Summary of the invention

[0003] The present disclosure provides an expression generation method, an expression generation device, an electronic device, a computer-readable storage medium, and a computer program product, to at least solve the problem that when a network method or a 3DMM method is used alone to predict an expression, the expression result obtained has poor robustness or poor flexibility. The technical solution of the present disclosure is as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, a method for generating an expression is provided, comprising: obtaining a face image, determining a first expression parameter and a second expression parameter corresponding to the face image; the first expression parameter and the second expression parameter are respectively obtained based on different expression extraction methods, and the expression extraction methods have corresponding expression bases; obtaining a pre-constructed state evaluation function; the state evaluation function is constructed based on expression states associated with each of the expression bases, and the expression state is a semantic description of the corresponding expression coefficient; based on the state evaluation function, the first expression parameter and the second expression parameter are fused to obtain fused expression parameters, so as to drive expression generation according to the fused expression parameters.

[0005] In an exemplary embodiment of the present disclosure, determining the first expression parameter and the second expression parameter corresponding to the facial image includes: inputting the facial image into an expression extraction network, and having the expression extraction network perform expression extraction processing on the facial image to obtain the first expression parameter; inputting the facial image into a face deformation model, and having the face deformation model perform expression extraction processing on the facial image to obtain the second expression parameter.

[0006] In an exemplary embodiment of the present disclosure, the state evaluation function includes a first evaluation function and a second evaluation function, and the first expression parameter and the second expression parameter are fused based on the state evaluation function, including: obtaining a first expression base corresponding to the first expression parameter, determining a first expression state associated with the first expression base, so as to determine a first evaluation function corresponding to each first expression state; obtaining a second expression base corresponding to the second expression parameter, determining a second expression state associated with the second expression base, so as to determine a second evaluation function corresponding to each second expression state; determining an interpolation function according to the first evaluation function and the second evaluation function, so as to determine an expression fusion coefficient according to the interpolation function; and fusion processing is performed on the first expression parameter and the second expression parameter according to the expression fusion coefficient.

[0007] In an exemplary embodiment of the present disclosure, the first expression state is included in a first state set, and determining the first evaluation function corresponding to each of the first expression states includes: determining a first probability distribution corresponding to the first expression state in the first state set; determining a first stable function, a first flexible function and a first focus factor corresponding to the first expression state; and determining the first evaluation function based on the first probability distribution, the first stable function, the first flexible function and the first focus factor.

[0008] In an exemplary embodiment of the present disclosure, the second expression state is included in a second state set, and determining the second evaluation function corresponding to each of the second expression states includes: determining the second probability distribution corresponding to the second expression state in the second state set; determining the second stable function, the second flexible function and the second focus factor corresponding to the second expression state; and determining the second evaluation function based on the second probability distribution, the second stable function, the second flexible function and the second focus factor.

[0009] In an exemplary embodiment of the present disclosure, an interpolation function is determined according to the first evaluation function and the second evaluation function, including: filtering the first evaluation function to obtain a first filtered evaluation function; filtering the second evaluation function to obtain a second filtered evaluation function; generating the interpolation function according to the first evaluation function, the first filtered evaluation function, the second evaluation function and the second filtered evaluation function.

[0010] In an exemplary embodiment of the present disclosure, expression generation is driven according to the fused expression parameters, including: acquiring an associated video related to the facial image, and determining a video frame sequence based on the associated video; generating an expression filter function according to the video frame sequence; filtering the fused expression parameters based on the expression filter function to obtain target expression parameters; and driving expression generation according to the target expression parameters and the expression base.

[0011] In an exemplary embodiment of the present disclosure, an expression filter function is generated according to the video frame sequence, including: establishing an initial stability function according to the video frame sequence, filtering the initial stability function to obtain a stability function; obtaining a pre-configured first filter parameter and a second filter parameter, and determining the expression filter function according to the stability function, the first filter parameter and the second filter parameter.

[0012] According to a second aspect of an embodiment of the present disclosure, there is provided an expression generation device, comprising: an expression parameter determination module, configured to execute acquisition of a face image, and determine a first expression parameter and a second expression parameter corresponding to the face image; the first expression parameter and the second expression parameter are respectively obtained based on different expression extraction methods, and the expression extraction methods have corresponding expression bases; a state function acquisition module, configured to execute acquisition of a pre-constructed state evaluation function; the state evaluation function is constructed based on the expression state associated with each of the expression bases, and the expression state is a semantic description of the corresponding expression coefficient; an expression generation module, configured to execute fusion processing of the first expression parameter and the second expression parameter based on the state evaluation function, and obtain fused expression parameters, so as to drive expression generation according to the fused expression parameters.

[0013] In an exemplary embodiment of the present disclosure, the expression parameter determination module includes an expression parameter determination unit, and the expression parameter determination unit is configured to execute: inputting the facial image into an expression extraction network, and the expression extraction network performs expression extraction processing on the facial image to obtain the first expression parameters; inputting the facial image into a face deformation model, and the face deformation model performs expression extraction processing on the facial image to obtain the second expression parameters.

[0014] In an exemplary embodiment of the present disclosure, the state evaluation function includes a first evaluation function and a second evaluation function, the expression generation module includes an expression fusion unit, and the expression fusion unit is configured to perform: obtaining a first expression base corresponding to the first expression parameter, determining a first expression state associated with the first expression base, so as to determine a first evaluation function corresponding to each first expression state; obtaining a second expression base corresponding to the second expression parameter, determining a second expression state associated with the second expression base, so as to determine a second evaluation function corresponding to each second expression state; determining an interpolation function according to the first evaluation function and the second evaluation function, so as to determine an expression fusion coefficient according to the interpolation function; and fusing the first expression parameter and the second expression parameter according to the expression fusion coefficient.

[0015] In an exemplary embodiment of the present disclosure, the first expression state is included in a first state set, the expression fusion unit includes a first function determination subunit, and the first function determination subunit is configured to perform: determining a first probability distribution corresponding to the first expression state in the first state set; determining a first stable function, a first flexible function and a first focus factor corresponding to the first expression state; and determining the first evaluation function based on the first probability distribution, the first stable function, the first flexible function and the first focus factor.

[0016] In an exemplary embodiment of the present disclosure, the second expression state is included in a second state set, the expression fusion unit includes a second function determination subunit, and the second function determination subunit is configured to perform: determining a second probability distribution corresponding to the second expression state in the second state set; determining a second stable function, a second flexible function and a second focus factor corresponding to the second expression state; and determining the second evaluation function based on the second probability distribution, the second stable function, the second flexible function and the second focus factor.

[0017] In an exemplary embodiment of the present disclosure, the expression fusion unit includes an interpolation function determination subunit, and the interpolation function determination subunit is configured to perform: filtering the first evaluation function to obtain a first filtering evaluation function; filtering the second evaluation function to obtain a second filtering evaluation function; generating the interpolation function according to the first evaluation function, the first filtering evaluation function, the second evaluation function and the second filtering evaluation function.

[0018] In an exemplary embodiment of the present disclosure, the expression generation module includes an expression generation unit, and the expression generation unit is configured to perform: obtaining an associated video related to the facial image, and determining a video frame sequence based on the associated video; generating an expression filter function according to the video frame sequence; filtering the fused expression parameters based on the expression filter function to obtain target expression parameters; and driving expression generation based on the target expression parameters and the expression base.

[0019] In an exemplary embodiment of the present disclosure, the expression generation unit includes a filter function subunit, and the filter function subunit is configured to execute: establishing an initial stability function according to the video frame sequence, filtering the initial stability function to obtain a stability function; obtaining a pre-configured first filter parameter and a second filter parameter, and determining an expression filter function according to the stability function, the first filter parameter and the second filter parameter.

[0020] According to a third aspect of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement any one of the expression generation methods described above.

[0021] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any one of the expression generation methods described above.

[0022] According to a fifth aspect of an embodiment of the present disclosure, there is provided a computer program product, comprising a computer program / instruction, wherein the computer program / instruction implements any one of the above-mentioned expression generation methods when executed by a processor.

[0023] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: on the one hand, the expression parameters determined by the two expression extraction methods are organically integrated through the state evaluation function, providing an expression parameter integration processing method. On the other hand, by organically integrating different expression parameters, the integrated expression recognition results are made more robust and flexible.

[0024] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0026] Figure 1 The figure is a flow chart of a method for generating an expression according to an exemplary embodiment.

[0027] Figure 2 The present invention is a flowchart showing a method of determining a target expression parameter corresponding to a facial image according to an exemplary embodiment.

[0028] Figure 3 The present invention is a flowchart showing a method of fusing a first expression parameter with a second expression parameter according to an exemplary embodiment.

[0029] Figure 4 The present invention is a flowchart showing a method for determining a first evaluation function corresponding to a first expression state according to an exemplary embodiment.

[0030] Figure 5 The present invention is a flowchart showing a method of driving expression generation according to fused expression parameters according to an exemplary embodiment.

[0031] Figure 6 The figure is a block diagram of a device for generating an expression according to an exemplary embodiment.

[0032] Figure 7 A block diagram of an electronic device according to an exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0033] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0035] Existing expression prediction schemes can often be predicted directly through the network method or the 3D face deformation statistical model (3DMM) method. The network method inputs the head image of the person into a prediction network and obtains the expression coefficient through regression. The 3DMM method solves the 3D face through constraints such as key points to obtain the expression coefficient. Traditionally, facial expressions are solved by 3DMM. The 3DMM method is more robust to large expressions, but it is not easy to reflect micro-expression movements, and it cannot reflect movements that cannot be portrayed by key points such as puffing cheeks. The network method is more flexible and more sensitive to micro-expressions, but the stability is not as good as 3DMM.

[0036] Based on this, according to the embodiments of the present disclosure, an expression generation method, an expression generation device, an electronic device, a computer-readable storage medium, and a computer program product are proposed.

[0037] Figure 1 is a flow chart of a method for generating an expression according to an exemplary embodiment. Figure 1 As shown, the expression generation method can be used in a computer device, wherein the computer device described in the present disclosure may include mobile terminal devices such as mobile phones, tablet computers, laptop computers, PDAs, and fixed terminal devices such as desktop computers. This exemplary embodiment uses the method applied to a computer device as an example, and it can be understood that the method can also be applied to a server, and can also be applied to a system including a computer device and a server, and is implemented through the interaction between the computer device and the server. Specifically, the following steps are included.

[0038] In step S110, a face image is acquired, and a first expression parameter and a second expression parameter corresponding to the face image are determined; the first expression parameter and the second expression parameter are obtained based on different expression extraction methods, and the expression extraction methods have corresponding expression bases;

[0039] In step S120, a pre-constructed state evaluation function is obtained; the state evaluation function is constructed based on the expression state associated with each expression base, and the expression state is a semantic description of the corresponding expression coefficient;

[0040] In step S130, the first expression parameter and the second expression parameter are fused based on the state evaluation function to obtain fused expression parameters, so as to drive expression generation according to the fused expression parameters.

[0041] According to the expression generation method in this example embodiment, on the one hand, the expression parameters determined by the two expression extraction methods are organically integrated through the state evaluation function, providing an expression parameter integration processing method. On the other hand, by organically integrating different expression parameters, the integrated expression recognition result is made more robust and flexible.

[0042] Next, the expression generation method in this exemplary embodiment will be further described.

[0043] In step S110, a face image is acquired, and a first expression parameter and a second expression parameter corresponding to the face image are determined; the first expression parameter and the second expression parameter are obtained based on different expression extraction methods, and the expression extraction methods have corresponding expression bases.

[0044] In some exemplary embodiments of the present disclosure, a facial image may be an image to be processed for expression extraction. The facial image may be an image captured in real time by a monocular camera device and a terminal device including a monocular camera, or may be a facial image pre-stored in a database, or may be a facial image obtained from other user terminals. The expression extraction method may be a processing method for determining corresponding expression parameters based on the facial image. The expression parameters may be expression parameters determined by using the expression extraction method. The first expression parameter and the second expression parameter are expression parameters determined by using different expression extraction methods for the facial image, respectively.

[0045] When a face image to be processed is obtained, two different expression extraction methods can be used to perform expression extraction processing on the same face image to respectively determine the first expression parameter and the second expression parameter corresponding to the face image.

[0046] In an exemplary embodiment of the present disclosure, a facial image is input into an expression extraction network, which performs expression extraction processing on the facial image to obtain a first expression parameter; the facial image is input into a face deformation model, which performs expression extraction processing on the facial image to obtain a second expression parameter.

[0047] The expression extraction network may be a deep learning network for extracting expression from a face image; for example, the expression extraction network may be a convolutional neural network (CNN) structure. The face deformation model may be a three-dimensional face statistical model for extracting expression from a face image; for example, the face deformation model may be a 3DMM model.

[0048] refer to Figure 2 , Figure 2It is a flowchart of determining the target expression parameters corresponding to a facial image according to an exemplary embodiment. In step S210, a facial image is input. For example, the facial image can be input into an expression extraction network or a facial deformation model respectively. In step S220, the network result is determined. The facial image is subjected to expression extraction processing by the expression extraction network to obtain a network result output by the expression extraction network, i.e., a first expression parameter. The first expression coefficient calculated by the expression extraction network is usually more flexible and more sensitive to micro-expressions. In step S230, a 3DMM result is determined. The facial image is subjected to expression extraction processing by the facial deformation model to obtain a 3DMM result output by the facial deformation model, i.e., a second expression coefficient. The second expression coefficient determined by the 3DMM model is more robust to large expressions. For example, X can be used to represent all possible input facial images, K can represent the number of output expression parameters, and the output first expression parameter can be N:X→R K , the second expression parameter output can be M:X→R K .

[0049] It is easy for people skilled in the art to understand that in other exemplary embodiments of the present disclosure, other expression extraction methods can also be used to perform fusion processing on the expression coefficients determined based on each expression extraction method, so that the obtained fused expression coefficient can integrate the advantages of multiple expression coefficients, and the output result is more comprehensive.

[0050] In step S120, a pre-constructed state evaluation function is obtained; the state evaluation function is constructed based on the expression state associated with each expression base, and the expression state is a semantic description of the corresponding expression coefficient.

[0051] In some exemplary embodiments of the present disclosure, the expression state may be a specific state corresponding to each expression parameter. The state evaluation function may be a function used to evaluate factors related to the expression state.

[0052] From the above content, we can know that the number of expression parameters corresponding to the face image is K, and the number of expression bases corresponding to the expression parameters can also be K. The expression bases corresponding to the expression parameters can include opening the mouth, closing the eyes, frowning, etc. For each expression base i (1≤i≤K), the set W can be used i All possible expression states of expression base i are stored, for example, mouth-related movements include expression states such as still, small mouth, and wide mouth. Among them, expression parameters are lower-level descriptions of facial expressions, and expression states can be higher-level semantic descriptions of facial expressions. In this embodiment, the filtering strategy used later can be selected according to the expression state. After determining the expression state corresponding to each expression base, a state evaluation function can be further constructed according to the expression state to determine the fusion expression parameters according to the state evaluation function.

[0053] In step S130, the first expression parameter and the second expression parameter are fused based on the state evaluation function to obtain fused expression parameters, so as to drive expression generation according to the fused expression parameters.

[0054] In some exemplary embodiments of the present disclosure, the fused expression parameter may be an expression parameter obtained by fusion processing the first expression parameter and the second expression parameter.

[0055] Continue to refer Figure 2 In step S240, after determining the first expression parameter and the second expression parameter, the first expression parameter and the second expression parameter can be fused through a state evaluation function to obtain a fusion result, namely, a fused expression parameter.

[0056] In an exemplary embodiment of the present disclosure, a first expression base corresponding to a first expression parameter is obtained, and a first expression state associated with the first expression base is determined to determine a first evaluation function corresponding to each first expression state; a second expression base corresponding to a second expression parameter is obtained, and a second expression state associated with the second expression base is determined to determine a second evaluation function corresponding to each second expression state; an interpolation function is determined according to the first evaluation function and the second evaluation function to determine an expression fusion coefficient according to the interpolation function; and the first expression parameter and the second expression parameter are fused according to the expression fusion coefficient.

[0057] Among them, the first expression base may be an expression base determined by a first expression extraction method, and the first expression base corresponds to the first expression parameter one-to-one. The first expression state may be an expression state involved in each first expression base. The first evaluation function may be a function used to evaluate the relevant factors of the first expression state. The second expression base may be an expression base determined by a second expression extraction method, and the second expression base corresponds to the second expression parameter one-to-one. The second expression state may be an expression state involved in each second expression base. The second evaluation function may be a function used to evaluate the relevant factors of the second expression state. The interpolation function may be a function used to determine the expression fusion coefficient using an interpolation processing method. The expression fusion coefficient may be a fusion coefficient used to fuse the first expression parameter and the second expression parameter.

[0058] refer to Figure 3 , Figure 3It is a flowchart of a fusion process of a first expression parameter and a second expression parameter according to an exemplary embodiment. In step S310, a first expression base corresponding to the first expression parameter is obtained, and a first expression state associated with the first expression base is determined to determine a first evaluation function corresponding to each first expression state. After obtaining the first expression parameter, since the first expression parameter has a one-to-one corresponding first expression base, all possible expression states involved in each first expression base i can be obtained. For example, taking the first expression base i as "raising eyebrows" as an example, the expression state of the expression base may include eyebrows curling inward, left eyebrow pressed down, right eyebrow pressed down, left eyebrow raised outward, right eyebrow raised outward, etc. Taking the first expression i as "eye gaze" as an example, the expression state of the expression may include left eye looking up, right eye looking up, left eye looking down, right eye looking down, left eye looking inward, right eye looking inward, left eye looking outward, right eye looking outward, left upper eyelid pressed down to complete eye closing, right upper eyelid pressed down to complete eye closing, left lower eyelid lifted up to complete squinting, right lower eyelid lifted up to complete squinting, etc. After the first expression state involved in each first expression parameter is obtained, a first evaluation function corresponding to the first expression state may be determined.

[0059] In step S320, a second expression base corresponding to the second expression parameter is obtained, and a second expression state associated with the second expression base is determined to determine a second evaluation function corresponding to each second expression state. For the second expression parameter, the same acquisition method as the first expression state can be adopted to obtain the second expression state and construct a second evaluation function corresponding to the second expression state, which will not be described in detail in this disclosure.

[0060] In step S330, an interpolation function is determined according to the first evaluation function and the second evaluation function, so as to determine the expression fusion coefficient according to the interpolation function. In the process of fusing the first expression parameter and the second expression parameter, in order to determine the expression fusion parameters of the two in the fusion process, after the first evaluation function and the second evaluation function are determined respectively, an interpolation function can be determined based on the two. Calculation is performed according to the interpolation function to determine the final expression fusion coefficient.

[0061] In step S340, the first expression parameter and the second expression parameter are fused according to the expression fusion coefficient. After the expression fusion coefficient is determined, the first expression parameter and the second expression parameter can be fused according to the expression fusion coefficient to obtain a fused expression parameter, so that the fused expression parameter can combine the advantages of different expression parameters, such as the obtained fused expression parameter can have both robustness and flexibility.

[0062] In an exemplary embodiment of the present disclosure, a first probability distribution corresponding to a first expression state in a first state set is determined; a first stable function, a first flexible function and a first focus factor corresponding to the first expression state are determined; and a first evaluation function is determined based on the first probability distribution, the first stable function, the first flexible function and the first focus factor.

[0063] Among them, the first state set can be a set consisting of all expression states involved in the first expression parameter. The first probability distribution can be the probability distribution corresponding to all first expression states in the first state set. The first stability function can be a function used to evaluate the stability of the first expression state. The first flexibility function can be a function used to evaluate the flexibility of the first expression state. The first focus factor can be a focus point corresponding to a first expression state. For example, in the small mouth state, more emphasis is placed on flexibility, and the first focus factor can be determined as flexibility.

[0064] refer to Figure 4 , Figure 4 4 is a flowchart showing a method of determining a first evaluation function corresponding to a first expression state according to an exemplary embodiment. In step S410, a first probability distribution corresponding to the first expression state in the first state set is determined. Since the first state set may contain multiple first expression states, for each of the expression states, the probability distribution of each first expression state in the first state set W can be calculated. i The probability distribution on is denoted as the first probability distribution P i .

[0065] For example, each element in the first state set corresponds to an expression state. For example, for the expression parameter of opening the mouth, the corresponding set may include different expression states such as still, talking, and slightly pursing the mouth. Moreover, these expression states are not necessarily orthogonal and may be in multiple different states at the same time. Therefore, if a certain expression happens to be in the above three states, it can be represented by a set of coefficients such as 0, 0.3, and 0.2. The above content can indicate that the current mouth is moving and not still, talking, the talking intensity is 0.3, and the pursing degree is 0.2.

[0066] In step S420, the first stable function, the first flexible function and the first focus factor corresponding to the first expression state are determined. The first stable function, the first flexible function and the first focus factor corresponding to each first expression state are determined; wherein the first stable function can be recorded as S M , the first flexible function L MSpecifically, the stability coefficient and flexibility coefficient of the first expression state can be obtained by experimental measurement. For example, in various expression states, the stability coefficient and flexibility coefficient can be obtained by manual scoring, and the corresponding probability distribution Pi can be recorded, so as to perform linear fitting according to the determined values, and use Pi to linearly express S M and L M .

[0067] In step S430, a first evaluation function is determined according to the first probability distribution, the first stable function, the first flexible function and the first concern factor. After determining the first probability distribution, the first stable function, the first flexible function and the first concern factor, a first evaluation function is constructed according to the above content to perform fusion processing on the expression parameters through the evaluation function. The specific first evaluation function is shown in Formula 1, where p∈P i .

[0068] F M,i (p) = (1-A(p))·S M (p)+A(p)·L M (p)(Formula 1)

[0069] In an exemplary embodiment of the present disclosure, a second probability distribution corresponding to a second expression state in a second state set is determined; a second stable function, a second flexible function and a second focus factor corresponding to the second expression state are determined; and a second evaluation function is determined based on the second probability distribution, the second stable function, the second flexible function and the second focus factor.

[0070] Among them, the second state set can be a set consisting of all expression states involved in the second expression parameter. The second probability distribution can be the probability distribution corresponding to all second expression states in the second state set. The second stability function can be a function used to evaluate the stability of the second expression state. The second flexibility function can be a function used to evaluate the flexibility of the second expression state. The second focus factor can be a focus point corresponding to a second expression state.

[0071] For the second expression state included in the second state set, the second evaluation function corresponding to each expression state can be determined in the same way as the first expression state, so it will not be described here. Specifically, the second stable function can be recorded as S N The second flexible function is denoted as L N , the second factor of interest is also recorded as A, so the final second evaluation function is shown in Formula 2, where p∈P i .

[0072] F N,i (p) = (1-A(p))·S N(p)+A(p)·L N (p)(Formula 2)

[0073] In an exemplary embodiment of the present disclosure, the first evaluation function is filtered to obtain a first filtered evaluation function; the second evaluation function is filtered to obtain a second filtered evaluation function; and an interpolation function is generated based on the first evaluation function, the first filtered evaluation function, the second evaluation function and the second filtered evaluation function.

[0074] The first filtering evaluation function may be a function obtained by filtering the first evaluation function, and the second filtering evaluation function may be a function obtained by filtering the second evaluation function.

[0075] After determining the first evaluation function and the second evaluation function, an interpolation function can be constructed based on the first evaluation function and the second evaluation function to determine the fusion expression parameters based on the interpolation function. When determining the interpolation function, the first evaluation function F M,i With the second evaluation function F N,i Perform filtering processing, such as time domain filtering processing, to obtain the filtered function, which is the first filtering evaluation function F' M,i (p) and the second filter evaluation function F N ' ,i (p). By filtering the first evaluation function and the second evaluation function, the smoothness of the interpolation result can be ensured. In addition, the expression is divided into state spaces, and the performance of each output in each state space is examined. These performances are weighted to construct an interpolation function. The final interpolation function is shown in Formula 3, where ρ i :X→P i .

[0076]

[0077] After the interpolation function is determined, the expression fusion coefficient i can be determined according to the interpolation function, where M i (x), N i (x) is the i-th component of M(x) and N(x).

[0078] In an exemplary embodiment of the present disclosure, an associated video related to a facial image is obtained, and a video frame sequence is determined based on the associated video; an expression filter function is generated according to the video frame sequence; based on the expression filter function, fused expression parameters are filtered to obtain target expression parameters; and expression generation is driven according to the target expression parameters and an expression base.

[0079] The associated video may be a video clip or an animated image associated with the face image to be processed. The video frame sequence may be a sequence of video frames contained in the associated video. The expression filter function may be a filter function used to filter the fused expression parameters.

[0080] Continue to refer Figure 2 In step S250, the target expression parameters are determined. In expression capture, stability and followability are also two difficult contradictions. In order to ensure that the expression parameters used in expression driving have both stability and followability, after determining the fused expression parameters, the fused expression parameters can be filtered to obtain the target expression parameters.

[0081] refer to Figure 5 , Figure 5 The flowchart of driving expression generation according to fusion expression parameters according to an exemplary embodiment is shown. In step S510, an associated video related to a face image is obtained, and a video frame sequence is determined based on the associated video. Since the expression output is usually coherent when driving the expression generation, in order to ensure the stability of the expression parameters, it can be performed based on the associated video related to the face image. The associated video usually contains multiple video frames, and these video frames can construct a corresponding video frame sequence V n =(x1,x2,x3,...,x n ).

[0082] In step S520, an expression filter function is generated according to the video frame sequence. n =(x1,x2,x3,...,x n ), the expression filter function can be generated based on the video frame sequence.

[0083] In an exemplary embodiment of the present disclosure, generating an expression filter function according to a video frame sequence can be performed through the following steps: establishing an initial stability function according to the video frame sequence, filtering the initial stability function to obtain a stability function; obtaining a pre-configured first filter parameter and a second filter parameter, and determining the expression filter function according to the stability function, the first filter parameter and the second filter parameter.

[0084] The initial stability function may be a function determined based on historical computing experience to describe the stability of the expression state. The stability function may be a function obtained by filtering the initial stability function. The first filter parameter may be a parameter corresponding to the first filtering processing method. The second filter parameter may be a parameter corresponding to the second filtering processing method.

[0085] In the process of expression generation, the mouth can be processed very finely. The mouth is a very special organ, which plays a very important role in expressing human expressions. In order to further enhance the stability while retaining the flexible movement of the mouth, the concept of steady state can be created based on this. For the video frame sequence V n =(x1,x2,x3,...,x n ), we can describe the stability of the current expression state by setting up an empirical function, that is, establishing the initial stability function: G(V n )=V→[0,1]. After the initial stability function is determined, it can be filtered to obtain a smooth steady-state function, that is, a stable function G'=o(G).

[0086] In order to ensure the stability and followability of the expression parameters at the same time, different filtering methods can be used to filter the expression parameters, such as filtering methods that make the expression stable and followable can be used at the same time. For example, in this embodiment, stage filtering T and one-euro filtering O can be used in combination. The former can ensure stability in a static state, while the latter has better followability. The first filtering parameter T corresponding to the first filtering method and the second filtering parameter O corresponding to the second filtering method are obtained respectively, so as to determine the expression filtering function according to the stability function, the first filtering parameter and the second filtering parameter. The expression filtering function is shown in Formula 4.

[0087] Z=(1-G')·T+G'·O (Formula 4)

[0088] The expression filter function finally determined can ensure the static stability of the expression while not affecting the flexible movement of the expression. For example, for mouth movements, the expression filter function obtained ensures the static stability of the mouth while not affecting the flexible movement of the mouth.

[0089] In step S530, the fused expression parameters are filtered based on the expression filter function to obtain the target expression parameters. The determined target expression parameters have both stability and followability. In step S540, expression generation is driven according to the target expression parameters and the expression base. After the target expression parameters are generated, the expression generation can be driven according to the target expression parameters and the expression base. For example, the target expression generated by the drive can be used in a virtual anchor scene, and can also be used in a virtual cartoon image.

[0090] The expression generation method disclosed in the present invention obtains a face image, determines a first expression parameter and a second expression parameter corresponding to the face image; the first expression parameter and the second expression parameter are obtained based on different expression extraction methods, respectively, and the expression extraction methods have corresponding expression bases; obtains a pre-constructed state evaluation function; the state evaluation function is constructed based on the expression state associated with each expression base, and the expression state is a semantic description of the corresponding expression coefficient; the first expression parameter and the second expression parameter are fused based on the state evaluation function to obtain a fused expression parameter, so as to drive the expression generation according to the fused expression parameter. On the one hand, the expression parameters determined by the two expression extraction methods are organically fused through the state evaluation function, and an expression parameter fusion processing method is provided. On the other hand, by organically fusing different expression parameters, the expression recognition result after fusion is made more robust and flexible. On the other hand, for some facial parts with complex expression features, the expression parameters are filtered by a two-stage filtering method, which can ensure the stability and followability of the expression at the same time, and finally obtain a comprehensive expression recognition result with sensitivity, stability, followability, etc.

[0091] Figure 6 is a block diagram of an expression generation device according to an exemplary embodiment. Figure 6 The expression generating device 600 includes an expression parameter determining module 610 , a state function acquiring module 620 and an expression generating module 630 .

[0092] The expression parameter determination module 610 is configured to execute acquiring a face image and determining a first expression parameter and a second expression parameter corresponding to the face image; the first expression parameter and the second expression parameter are obtained based on different expression extraction methods, respectively, and the expression extraction methods have corresponding expression bases;

[0093] The state function acquisition module 620 is configured to execute and acquire a pre-built state evaluation function; the state evaluation function is constructed based on the expression state associated with each expression base, and the expression state is a semantic description of the corresponding expression coefficient;

[0094] The expression generation module 630 is configured to perform a fusion process on the first expression parameter and the second expression parameter based on the state evaluation function to obtain a fused expression parameter, so as to drive expression generation according to the fused expression parameter.

[0095] In an exemplary embodiment of the present disclosure, the expression parameter determination module includes an expression parameter determination unit, which is configured to execute: inputting a facial image into an expression extraction network, and having the expression extraction network perform expression extraction processing on the facial image to obtain a first expression parameter; inputting the facial image into a face deformation model, and having the face deformation model perform expression extraction processing on the facial image to obtain a second expression parameter.

[0096] In an exemplary embodiment of the present disclosure, the state evaluation function includes a first evaluation function and a second evaluation function, the expression generation module includes an expression fusion unit, and the expression fusion unit is configured to execute: obtaining a first expression base corresponding to a first expression parameter, determining a first expression state associated with the first expression base, so as to determine a first evaluation function corresponding to each first expression state; obtaining a second expression base corresponding to a second expression parameter, determining a second expression state associated with the second expression base, so as to determine a second evaluation function corresponding to each second expression state; determining an interpolation function according to the first evaluation function and the second evaluation function, so as to determine an expression fusion coefficient according to the interpolation function; and fusing the first expression parameter with the second expression parameter according to the expression fusion coefficient.

[0097] In an exemplary embodiment of the present disclosure, a first expression state is included in a first state set, and the expression fusion unit includes a first function determination subunit, and the first function determination subunit is configured to perform: determining a first probability distribution corresponding to the first expression state in the first state set; determining a first stable function, a first flexible function and a first focus factor corresponding to the first expression state; and determining a first evaluation function based on the first probability distribution, the first stable function, the first flexible function and the first focus factor.

[0098] In an exemplary embodiment of the present disclosure, the second expression state is included in the second state set, the expression fusion unit includes a second function determination subunit, and the second function determination subunit is configured to perform: determining a second probability distribution corresponding to the second expression state in the second state set; determining a second stable function, a second flexible function and a second focus factor corresponding to the second expression state; and determining a second evaluation function based on the second probability distribution, the second stable function, the second flexible function and the second focus factor.

[0099] In an exemplary embodiment of the present disclosure, the expression fusion unit includes an interpolation function determination subunit, and the interpolation function determination subunit is configured to perform: filtering the first evaluation function to obtain a first filtered evaluation function; filtering the second evaluation function to obtain a second filtered evaluation function; generating an interpolation function based on the first evaluation function, the first filtered evaluation function, the second evaluation function and the second filtered evaluation function.

[0100] In an exemplary embodiment of the present disclosure, the expression generation module includes an expression generation unit, which is configured to perform: obtaining an associated video related to a facial image, and determining a video frame sequence based on the associated video; generating an expression filter function according to the video frame sequence; filtering the fused expression parameters based on the expression filter function to obtain target expression parameters; and driving expression generation according to the target expression parameters and an expression base.

[0101] In an exemplary embodiment of the present disclosure, the expression generation unit includes a filter function subunit, which is configured to execute: establishing an initial stability function according to a video frame sequence, filtering the initial stability function to obtain a stability function; obtaining a pre-configured first filter parameter and a second filter parameter, and determining the expression filter function according to the stability function, the first filter parameter and the second filter parameter.

[0102] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0103] Reference below Figure 7 hereinafter describes an electronic device 700 according to such an embodiment of the present disclosure. Figure 7 The electronic device 700 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0104] like Figure 7 As shown, the electronic device 700 is in the form of a general computing device. The components of the electronic device 700 may include, but are not limited to: the at least one processing unit 710, the at least one storage unit 720, a bus 730 connecting different system components (including the storage unit 720 and the processing unit 710), and a display unit 740.

[0105] The storage unit stores program codes, which can be executed by the processing unit 710, so that the processing unit 710 executes the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0106] The storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 721 and / or a cache memory unit 722 , and may further include a read-only memory unit (ROM) 723 .

[0107] The storage unit 720 may include a program / utility 724 having a set (at least one) of program modules 725, such program modules 725 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0108] Bus 730 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0109] The electronic device 700 may also communicate with one or more external devices 770 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 700, and / or communicate with any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 750. Furthermore, the electronic device 700 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 760. As shown, the network adapter 760 communicates with other modules of the electronic device 700 via a bus 730. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0110] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the above instructions can be executed by a processor of the device to complete the above expression generation method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0111] In an exemplary embodiment, a computer program product is also provided, including a computer program / instruction, wherein the computer program / instruction implements any one of the above-mentioned expression generation methods when executed by a processor.

[0112] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0113] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for generating an expression, characterized in that: include: Acquire a facial image, and determine a first expression parameter and a second expression parameter corresponding to the facial image; The first expression parameter and the second expression parameter are obtained based on different expression extraction methods, the expression extraction methods have corresponding expression bases, the first expression parameter is obtained by performing expression extraction processing on the face image by an expression extraction network, and the second expression parameter is obtained by performing expression extraction processing on the face image by a face deformation model; Get the pre-built state evaluation function; The state evaluation function is constructed based on the expression state associated with each of the expression bases, and the expression state is a semantic description of the corresponding expression coefficient; Based on the state evaluation function, the first expression parameter and the second expression parameter are fused to obtain a fused expression parameter, so as to drive expression generation according to the fused expression parameter; The state evaluation function includes a first evaluation function and a second evaluation function, and the fusing process of the first expression parameter and the second expression parameter based on the state evaluation function includes: Acquire a first expression base corresponding to the first expression parameter, determine a first expression state associated with the first expression base, and determine a first evaluation function corresponding to each of the first expression states; Acquire a second expression base corresponding to the second expression parameter, determine a second expression state associated with the second expression base, and determine a second evaluation function corresponding to each second expression state; determining an interpolation function according to the first evaluation function and the second evaluation function, so as to determine an expression fusion coefficient according to the interpolation function; The first expression parameter and the second expression parameter are fused according to the expression fusion coefficient.

2. The method according to claim 1, characterized in that The first expression states are included in a first state set, and the determining of a first evaluation function corresponding to each of the first expression states includes: Determine a first probability distribution corresponding to the first expression state in the first state set; Determine a first stable function, a first flexible function, and a first focus factor corresponding to the first expression state, wherein the first flexible function is a function for evaluating the flexibility of the first expression state, and the first focus factor is a focus point corresponding to the first expression state; The first evaluation function is determined according to the first probability distribution, the first stable function, the first flexible function and the first concern factor.

3. The method according to claim 1, characterized in that The second expression state is included in a second state set, and the second evaluation function corresponding to each of the second expression states is determined, including: Determine a second probability distribution corresponding to the second expression state in the second state set; Determining a second stable function, a second flexible function, and a second focus factor corresponding to the second expression state; The second evaluation function is determined according to the second probability distribution, the second stable function, the second flexible function and the second concern factor.

4. The method according to claim 1, characterized in that: The determining of the interpolation function according to the first evaluation function and the second evaluation function comprises: Performing filtering processing on the first evaluation function to obtain a first filtering evaluation function; Performing filtering processing on the second evaluation function to obtain a second filtering evaluation function; The interpolation function is generated according to the first evaluation function, the first filtering evaluation function, the second evaluation function, and the second filtering evaluation function.

5. The method according to claim 1, characterized in that The step of driving expression generation according to the fused expression parameters comprises: Acquire an associated video related to the face image, and determine a video frame sequence based on the associated video; Generate an expression filter function according to the video frame sequence; Performing filtering processing on the fused expression parameters based on the expression filtering function to obtain target expression parameters; The expression is generated according to the target expression parameter and the expression base.

6. The method according to claim 5, characterized in that The step of generating an expression filter function according to the video frame sequence comprises: Establishing an initial stability function according to the video frame sequence, and filtering the initial stability function to obtain a stability function; A pre-configured first filter parameter and a second filter parameter are obtained, and an expression filter function is determined according to the stabilization function, the first filter parameter, and the second filter parameter.

7. An expression generating device, characterized in that: include: An expression parameter determination module is configured to obtain a facial image and determine a first expression parameter and a second expression parameter corresponding to the facial image; The first expression parameter and the second expression parameter are obtained based on different expression extraction methods, the expression extraction methods have corresponding expression bases, the first expression parameter is obtained by performing expression extraction processing on the face image by an expression extraction network, and the second expression parameter is obtained by performing expression extraction processing on the face image by a face deformation model; A state function acquisition module is configured to execute and acquire a pre-built state evaluation function; The state evaluation function is constructed based on the expression state associated with each of the expression bases, the expression state is a semantic description of the corresponding expression coefficient, and the state evaluation function includes a first evaluation function and a second evaluation function; an expression generation module, configured to perform a fusion process on the first expression parameter and the second expression parameter based on the state evaluation function to obtain a fused expression parameter, so as to drive expression generation according to the fused expression parameter; The expression generation module includes an expression fusion unit, and the expression fusion unit is configured to perform: obtaining a first expression base corresponding to the first expression parameter, determining a first expression state associated with the first expression base, and determining a first evaluation function corresponding to each of the first expression states; Acquire a second expression base corresponding to the second expression parameter, determine a second expression state associated with the second expression base, and determine a second evaluation function corresponding to each second expression state; determining an interpolation function according to the first evaluation function and the second evaluation function, so as to determine an expression fusion coefficient according to the interpolation function; The first expression parameter and the second expression parameter are fused according to the expression fusion coefficient.

8. The device according to claim 7, characterized in that The first expression state is included in a first state set, the expression fusion unit includes a first function determination subunit, and the first function determination subunit is configured to execute: Determine a first probability distribution corresponding to the first expression state in the first state set; Determine a first stable function, a first flexible function, and a first focus factor corresponding to the first expression state, wherein the first flexible function is a function for evaluating the flexibility of the first expression state, and the first focus factor is a focus point corresponding to the first expression state; The first evaluation function is determined according to the first probability distribution, the first stable function, the first flexible function and the first concern factor.

9. The device according to claim 7, characterized in that The second expression state is included in a second state set, the expression fusion unit includes a second function determination subunit, and the second function determination subunit is configured to execute: Determine a second probability distribution corresponding to the second expression state in the second state set; Determining a second stable function, a second flexible function, and a second focus factor corresponding to the second expression state; The second evaluation function is determined according to the second probability distribution, the second stable function, the second flexible function and the second concern factor.

10. The device according to claim 7, characterized in that The expression fusion unit includes an interpolation function determination subunit, and the interpolation function determination subunit is configured to perform: Performing filtering processing on the first evaluation function to obtain a first filtering evaluation function; Performing filtering processing on the second evaluation function to obtain a second filtering evaluation function; The interpolation function is generated according to the first evaluation function, the first filtering evaluation function, the second evaluation function, and the second filtering evaluation function.

11. The device according to claim 7, characterized in that The expression generation module includes an expression generation unit, and the expression generation unit is configured to execute: Acquire an associated video related to the face image, and determine a video frame sequence based on the associated video; Generate an expression filter function according to the video frame sequence; Performing filtering processing on the fused expression parameters based on the expression filtering function to obtain target expression parameters; The expression is generated according to the target expression parameter and the expression base.

12. The device according to claim 11, characterized in that The expression generation unit includes a filter function subunit, and the filter function subunit is configured to perform: Establishing an initial stability function according to the video frame sequence, and filtering the initial stability function to obtain a stability function; A pre-configured first filter parameter and a second filter parameter are obtained, and an expression filter function is determined according to the stabilization function, the first filter parameter, and the second filter parameter.

13. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the expression generation method as described in any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an expression generating device / electronic device, the expression generating device / electronic device is enabled to execute the expression generating method as claimed in any one of claims 1 to 6.

15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the expression generation method according to any one of claims 1 to 6 is implemented.

Images