An expression driving method, apparatus, device and medium

Abstract

Embodiments of the present disclosure relate to an expression driving method, device, equipment and medium, wherein the method comprises: obtaining a target image, and identifying an expression of a target object in the target image to obtain at least one expression coefficient; extracting a to-be-processed expression coefficient in the at least one expression coefficient, and determining an initial left expression coefficient and an initial right expression coefficient in the to-be-processed expression coefficient; generating a target left expression coefficient and a target right expression coefficient based on a similarity between the initial left expression coefficient and the initial right expression coefficient; and driving a virtual image to make a corresponding expression based on the target left expression coefficient and the target right expression coefficient. Embodiments of the present disclosure not only realize the post-processing relationship constraint of the identified expression coefficient, so that the consistency and synchronization of the expressions between the parts of the face having a relationship can be improved when the expression is driven by a new expression coefficient subsequently, but also avoid possible expression failure, and improve the authenticity of expression driving.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an expression-driven method, apparatus, device, and medium. Background Technology

[0002] With the continuous development of internet and artificial intelligence technologies, in some scenarios, virtual images of target objects can be generated and driven to achieve better human-computer interaction effects.

[0003] Facial expression-driven animation can be understood as acquiring facial expression images of a target object through image acquisition devices, then using algorithms to obtain facial expression driving data (such as expression coefficients), and using this data to drive a virtual avatar of the target object (such as a 3D virtual facial model of the target object), enabling the driven virtual avatar to make expressions consistent with the target object. However, in some cases, the driven expressions exhibit inconsistencies and asynchronies between related parts of the face, resulting in unrealistic driving effects. Summary of the Invention

[0004] To address the aforementioned technical problems, this disclosure provides an expression-driven method, apparatus, device, and medium.

[0005] This disclosure provides an expression-driven method, the method comprising: Acquire a target image and identify the facial expression of the target object in the target image to obtain at least one facial expression coefficient; Extract the expression coefficient to be processed from the at least one expression coefficient, and determine the initial left expression coefficient and the initial right expression coefficient from the expression coefficient to be processed; Based on the similarity between the initial left expression coefficient and the initial right expression coefficient, target left expression coefficient and target right expression coefficient are generated; Based on the target left expression coefficient and the target right expression coefficient, the virtual avatar is driven to make corresponding expressions.

[0006] This disclosure also provides an expression-driving device, the device comprising: The recognition module is used to acquire a target image and recognize the facial expression of the target object in the target image to obtain at least one facial expression coefficient. A determining module is used to extract the expression coefficients to be processed from the at least one expression coefficient, and to determine the initial left expression coefficient and the initial right expression coefficient from the expression coefficients to be processed; The processing module is used to generate target left expression coefficients and target right expression coefficients based on the similarity between the initial left expression coefficients and the initial right expression coefficients; The driving module is used to drive the virtual image to make corresponding expressions based on the target left expression coefficient and the target right expression coefficient.

[0007] This disclosure also provides an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the expression-driven method as provided in this disclosure.

[0008] This disclosure also provides a computer-readable storage medium storing a computer program for executing the expression-driven method provided in this disclosure.

[0009] Compared with the prior art, the technical solution provided in this disclosure has the following advantages: The expression-driven scheme provided in this disclosure acquires a target image and identifies the expression of the target object in the target image to obtain at least one expression coefficient; extracts the expression coefficient to be processed from the at least one expression coefficient, and determines the initial left expression coefficient and the initial right expression coefficient in the expression coefficient to be processed; generates the target left expression coefficient and the target right expression coefficient based on the similarity between the initial left expression coefficient and the initial right expression coefficient; and drives the virtual image to make the corresponding expression based on the target left expression coefficient and the target right expression coefficient. By adopting the above technical solution, by dividing the expression coefficient to be processed in the expression coefficient into left and right parts, and then generating new left and right expression coefficients based on the similarity between the left and right parts, and driving the virtual image's expression based on the new expression coefficients, it not only realizes the post-processing relationship constraint of the identified expression coefficients, so that when the expression is driven by the new expression coefficients, the consistency and synchronization of the expressions between the related parts of the face can be improved, but also avoids possible expression failure and improves the realism of expression driving. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0011] Figure 1 A flowchart illustrating an expression-driven method provided in an embodiment of this disclosure; Figure 2 A flowchart illustrating another expression-driven method provided in this embodiment of the present disclosure; Figure 3 A schematic diagram of an expression-driven expression as provided in an embodiment of this disclosure; Figure 4A comparative schematic diagram provided for an embodiment of this disclosure; Figure 5 This is a schematic diagram of the structure of an expression driving device provided in an embodiment of the present disclosure; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0012] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0013] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0014] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0015] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0016] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0017] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0018] Research has found that, due to the lack of constraints on the relationships between facial expression coefficients across different body parts, inconsistencies and asynchronies can occur between related facial features when driving expressions in virtual avatars. This leads to unrealistic driving effects. For example, in real life, the left and right eyes are roughly the same size when the eyes are open or blinking normally. If the driven eyes are inconsistent in size, the resulting effect will significantly impact the user experience. Related technologies address this issue by directly imposing consistency constraints based on the relationships between body parts. For instance, for eye driving, the corresponding expression driving coefficients for the left and right eyes can be averaged, and then the averaged coefficients can be reassigned to the original left and right eye coefficients, ensuring that the left and right eye sizes are completely consistent. However, this method is relatively simplistic and may cause some expressions to fail, such as the single-eye closed expression, resulting in unrealistic driving effects.

[0019] To address the aforementioned issues, this disclosure provides an expression-driven method, which will be described below with reference to specific embodiments.

[0020] Figure 1 This is a flowchart illustrating an expression-driven method provided in an embodiment of the present disclosure. The method can be executed by an expression-driven device, which can be implemented in software and / or hardware, and is generally integrated into an electronic device. Figure 1 As shown, the method includes: Step 101: Obtain the target image and identify the facial expression of the target object in the target image to obtain at least one facial expression coefficient.

[0021] The target image in this embodiment of the disclosure can be a two-dimensional image including the facial region of the target object, which can be obtained by capturing the facial region of the target object using an image acquisition device.

[0022] The expression coefficient can be obtained by using a computer to extract facial features from an image. The expression coefficient can be understood as the descriptive information of facial expressions that a computer can understand. Through this expression coefficient, an intelligent human-computer interaction environment can be established.

[0023] In this embodiment of the disclosure, recognizing the facial expression of a target object in a target image to obtain at least one facial expression coefficient may include: inputting the target image into a pre-trained facial expression recognition network to obtain at least one facial expression coefficient.

[0024] The facial expression recognition network can be a pre-created and trained neural network for extracting facial expression features. The specific facial expression recognition network used in this embodiment is not limited, for example, a convolutional neural network can be used.

[0025] Specifically, the expression-driven device can acquire a target image. The acquisition method of the target image is not limited, such as real-time capture or uploading. The target image can be a single image or an image sequence extracted from a video, etc. The target image containing the target object can then be input into the expression recognition network, and the output will be the expression coefficient corresponding to the target image. The expression coefficient can include expression coefficients corresponding to multiple parts, and the specific number is not limited. For example, it can include expression coefficients for the nose and expression coefficients for the mouth, etc.

[0026] Step 102: Extract the expression coefficient to be processed from at least one expression coefficient, and determine the initial left expression coefficient and the initial right expression coefficient from the expression coefficient to be processed.

[0027] The expression coefficients to be processed can be understood as expression coefficients that require certain relational constraints. That is, in some cases, there may be inconsistencies and asynchronies when using these expression coefficients to drive expressions. The initial left expression coefficient and the initial right expression coefficient can be the initial coefficients obtained by grouping the expression coefficients to be processed into left and right groups based on their semantic information. The initial left expression coefficient corresponds to the initial expression coefficient of the left half, and the initial right expression coefficient corresponds to the initial expression coefficient of the right half.

[0028] In this embodiment of the disclosure, extracting the expression coefficient to be processed from at least one expression coefficient may include: extracting the expression coefficient for a preset part from at least one expression coefficient to obtain the expression coefficient to be processed, wherein the preset part has two symmetrical sub-parts on the left and right.

[0029] The preset part can be understood as a facial feature divided into left and right sub-parts. These two sub-parts can be symmetrical, and each sub-part can move independently with relatively little influence from the other sub-part. For example, the preset part can include the eyes, eyebrows, mouth, and nose. After obtaining at least one expression coefficient, the expression coefficient for the preset part can be extracted from the at least one expression coefficient, which is the expression coefficient to be processed. Then, the initial left expression coefficient and the initial right expression coefficient in the expression coefficient to be processed can be determined based on the semantic information of the expression coefficient to be processed.

[0030] Step 103: Based on the similarity between the initial left expression coefficient and the initial right expression coefficient, generate the target left expression coefficient and the target right expression coefficient.

[0031] Among them, the target left expression coefficient can be a new expression coefficient obtained by dynamically adjusting the initial left expression coefficient, and the target right expression coefficient can be a new expression coefficient obtained by dynamically adjusting the initial right expression coefficient.

[0032] Specifically, after dividing the expression coefficients to be processed into initial left expression coefficients and initial right expression coefficients, the expression driving device can first determine the similarity between the initial left expression coefficients and initial right expression coefficients, and then process the initial left expression coefficients and initial right expression coefficients accordingly based on the similarity to obtain the target left expression coefficients and target right expression coefficients.

[0033] In some embodiments, the initial left expression coefficient includes multiple left-dimensional data, and the initial right expression coefficient includes multiple right-dimensional data. The number of multiple left-dimensional data is the same as the number of multiple right-dimensional data, and each left-dimensional data corresponds to one right-dimensional data. The similarity between the initial left expression coefficient and the initial right expression coefficient is determined in units of dimensional data pairs. There are multiple dimensional data pairs, and each dimensional data pair includes a corresponding left-dimensional data and a right-dimensional data.

[0034] Both the initial left and right expression coefficients can include data from multiple dimensions, with the same number of dimensions. There is a one-to-one correspondence between the multiple left-dimensional and right-dimensional data; that is, each left-dimensional data point has a corresponding right-dimensional data point. For example, assuming the expression coefficient to be processed corresponds to the eyes, the initial left expression coefficient corresponds to the left eye, and the initial right expression coefficient corresponds to the right eye. The initial left expression coefficient can include multiple left-dimensional coefficients such as the left blink coefficient, the left eye looking up coefficient, and the left eye squinting coefficient. Similarly, the initial right expression coefficient can include multiple right-dimensional coefficients such as the right blink coefficient, the right eye looking up coefficient, and the right eye squinting coefficient. The left blink coefficient corresponds to the right blink coefficient, the left eye looking up coefficient corresponds to the right eye looking up coefficient, and the left eye squinting coefficient corresponds to the right eye squinting coefficient.

[0035] When determining the similarity between the initial left and right expression coefficients, the expression-driven device can first extract each left-dimensional data point in the initial left expression coefficient and its corresponding right-dimensional data point in the initial right expression coefficient, forming a single dimension data pair, resulting in multiple dimension data pairs. Then, it can sequentially determine the single similarity of each dimension data pair, obtaining multiple single similarities, which represent the similarity between the initial left and right expression coefficients. Next, when adjusting the initial left and right expression coefficients based on the similarity, adjustments can also be made on a pair-by-pair basis. That is, the left and right dimension data points included in the corresponding dimension data pair are adjusted based on each single similarity, resulting in adjusted left and right dimension data. Finally, the adjusted target left expression coefficient is obtained based on the multiple adjusted left dimension data points, and the adjusted target right expression coefficient is obtained based on the multiple adjusted right dimension data points.

[0036] The higher the similarity value, the closer the target left and right expression coefficients are to the average of the initial left and right expression coefficients; conversely, the lower the similarity value, the closer the target left expression coefficient is to the initial left expression coefficient, and the closer the target right expression coefficient is to the initial right expression coefficient. That is, when the initial left and right expression coefficients differ significantly, the output similarity is close to 0. In this case, the target left and right expression coefficients will not significantly adjust the initial left and right expression coefficients, meaning they will not change the initial expression coefficients. This prevents specific expressions from failing, such as expressions made solely by the left or right sub-parts of a preset part corresponding to the expression coefficient to be processed. In some specific examples, these specific expressions include expressions with one eye closed, expressions with a crooked smile, etc.

[0037] For example, Figure 2 This is a flowchart illustrating another expression-driven method provided in an embodiment of the present disclosure, as shown below. Figure 2 As shown, in one feasible implementation, step 103 above may include the following steps: Step 201: Use a similarity function to determine the similarity between the initial left expression coefficient and the initial right expression coefficient.

[0038] The similarity function can be understood as a function that determines the similarity between two data. In this embodiment, it is used to determine the similarity between the initial left expression coefficient and the initial right expression coefficient. The specific function used is not limited. In this embodiment, the similarity function can be a decreasing linear function or a non-linear function. For example, the linear function can be a first-order linear function, and the non-linear function can be an exponential function, a logarithmic function, etc.

[0039] In some embodiments, determining the similarity between the initial left expression coefficient and the initial right expression coefficient using a similarity function may include: determining the difference between the initial left expression coefficient and the initial right expression coefficient; modulating the difference using modulation parameters; and using the modulated difference as a variable of the similarity function to calculate the similarity between the initial left expression coefficient and the initial right expression coefficient.

[0040] The modulation parameter can be a parameter used to set the smoothing intensity of the similarity function; the larger the modulation parameter, the smaller the smoothing intensity. In this embodiment, the modulation parameter can correspond to a specific expression coefficient to be processed. That is, when the expression coefficient to be processed is different, the corresponding modulation parameter can also be different. Specifically, it can be related to a preset part corresponding to the expression coefficient to be processed, thereby enabling different degrees of dynamic adjustment of the initial expression coefficient of different parts in the subsequent process. For example, when the left and right sub-parts of a preset part usually have high consistency in movement, the modulation parameter corresponding to the expression coefficient to be processed for that part can be set to a smaller value. For example, if the part is the eyes or eyebrows, the modulation parameter can be set to 4. Conversely, when the left and right sub-parts of a part usually have low consistency in movement, that is, a large difference in movement, the modulation parameter corresponding to the expression coefficient to be processed for that part can be set to a larger value. For example, if the part is the mouth, the modulation parameter can be set to 8.

[0041] Optionally, the modulation parameters mentioned above can be preset based on experience, or they can be set as adjustable parameters by the user, for example, determined by the user through interactive means. For instance, in scenarios such as virtual anchors and virtual conferences, the modulation parameters can be determined in response to user actions, with specific values ​​determined according to user needs.

[0042] Specifically, when the expression-driven device uses a similarity function to determine the similarity between the initial left and initial right expression coefficients, it can first calculate the difference between them and use this difference as a variable in the similarity function. Further, it can modulate this difference using modulation parameters to obtain a modulated difference. For example, the modulation method could be to calculate the product of the difference and the modulation parameters, and use this product as the modulated difference (this is just an example). Then, this modulated difference can be used as a variable in the similarity function for calculation, yielding the similarity between the initial left and initial right expression coefficients. Since the specific data for the initial left and initial right expression coefficients can be multi-dimensional, the corresponding similarity can be calculated separately for each dimension.

[0043] For example, taking the initial left expression coefficient and initial right expression coefficient of the i-th dimension as an example, the initial left expression coefficient of the i-th dimension is represented as... Let the initial right expression coefficient of the i-th dimension be represented as Taking an exponential function with a base of a decreasing constant e as an example, the similarity between the initial left and initial right expression coefficients in the i-th dimension is expressed as: Where S represents the similarity, and k represents the modulation parameters corresponding to the current expression coefficient to be processed. Alternatively, this similarity function can also be a decreasing linear function, where the similarity between the initial left and initial right expression coefficients in the i-th dimension is expressed as... k can take the value 1; furthermore, the similarity function can also be a decreasing exponential function, where the similarity between the initial left expression coefficient and the initial right expression coefficient in the i-th dimension is expressed as... k can take the value 1. It is understood that the above similarity function is merely an example, not a limitation.

[0044] Step 202: Adjust the initial left expression coefficient and the initial right expression coefficient based on similarity to obtain the target left expression coefficient and the target right expression coefficient.

[0045] Specifically, after determining the similarity between the initial left expression coefficient and the initial right expression coefficient, the expression driving device can adjust and calculate the initial left expression coefficient and the initial right expression coefficient based on the similarity to obtain the corresponding target left expression coefficient and target right expression coefficient.

[0046] In some embodiments, adjusting the initial left expression coefficient and the initial right expression coefficient based on similarity to obtain the target left expression coefficient and the target right expression coefficient may include: determining the difference based on similarity; and weighting the initial left expression coefficient and the initial right expression coefficient based on similarity and difference to obtain the target left expression coefficient and the target right expression coefficient. The difference is the difference between 1 and the similarity.

[0047] Specifically, the expression-driven device can first determine the first and second products of similarity with the initial left expression coefficient and the initial right expression coefficient, respectively, and determine the third and fourth products of difference with the initial left expression coefficient and the initial right expression coefficient, respectively; the sum of the second and third products is determined as the target left expression coefficient, and the sum of the first and fourth products is determined as the target right expression coefficient.

[0048] For example, taking the target left expression coefficient and target right expression coefficient in the i-th dimension as an example, the target left expression coefficient can be expressed as:

[0049] The target right expression coefficient can be expressed as:

[0050] in This represents the target left expression coefficient in the i-th dimension. This represents the target right facial expression coefficient in the i-th dimension. Let represent the similarity between the initial left expression coefficient and the initial right expression coefficient in the i-th dimension, 1- Indicates the degree of difference. Represents the first product. Indicates the second product. Represents the third product. This represents the fourth product.

[0051] Step 104: Based on the target left expression coefficient and the target right expression coefficient, drive the virtual image to make the corresponding expression.

[0052] The expression-driving device can drive the parts of the virtual image corresponding to the coefficients to be processed to make corresponding expressions by using the target left expression coefficient and the target right expression coefficient. This not only avoids possible expression failure, that is, ensures the driving effect of specific expressions, but also achieves more consistent and synchronized expressions.

[0053] The expression-driven scheme provided in this disclosure acquires a target image and identifies the expression of a target object in the target image to obtain at least one expression coefficient; extracts the expression coefficient to be processed from the at least one expression coefficient, and determines the initial left expression coefficient and the initial right expression coefficient in the expression coefficient to be processed; generates the target left expression coefficient and the target right expression coefficient based on the similarity between the initial left expression coefficient and the initial right expression coefficient; and drives the virtual image to make the corresponding expression based on the target left expression coefficient and the target right expression coefficient. By dividing the expression coefficient to be processed into left and right parts, and then generating new left and right expression coefficients based on the similarity between the two parts, and driving the virtual image's expression based on the new expression coefficients, this scheme not only achieves post-processing relationship constraints on the identified expression coefficients, improving the consistency and synchronization of expressions between related facial parts when driving expressions with new expression coefficients, but also avoids possible expression failures and improves the realism of expression driving.

[0054] The expression-driven method of this disclosure will be further illustrated by a specific example below. For example, Figure 3 A schematic diagram of an expression-driven expression provided in an embodiment of this disclosure, such as... Figure 3 As shown in the figure, taking the user as the target object, the specific process of facial expression recognition is illustrated. It can include: the target image is processed by an facial expression recognition network to obtain facial expression coefficients. The unprocessed facial expression coefficients are extracted from the facial expression coefficients and grouped into initial left and right facial expression coefficients, i.e., Left and Right in the figure. Each dimension of the Left data dimension corresponds one-to-one with each dimension of the Right data dimension. A similarity function is used to obtain the similarity between Left and Right, i.e., S1, S2, ..., SN. The similarity values ​​S1, S2, ..., SN are used to dynamically adjust Left and Right to obtain new facial expression coefficients LeftP and RightP, which are the aforementioned target left and target right facial expression coefficients.

[0055] This scheme groups the expression coefficients that need processing into Left and Right groups. Left corresponds to the expression coefficients on the left side of the face, and Right corresponds to the expression coefficients on the right side of the face. There is a one-to-one correspondence between Left and Right; for example, if the first expression coefficient of Left drives a left-sided crooked mouth, then the first expression coefficient of Right drives a right-sided crooked mouth. Then, the similarity between each Left and its corresponding Right is calculated. This similarity is used to dynamically adjust Left and Right, resulting in new LeftP and RightP. The higher the similarity, the closer the new LeftP and RightP are to the average of the initial Left and Right values. Conversely, the lower the similarity, the closer the new LeftP and RightP are to their original values, meaning that the initial expression coefficients are essentially unchanged, thus preventing specific expressions from becoming invalid.

[0056] For example, Figure 4 A comparative schematic diagram provided for embodiments of this disclosure, such as... Figure 4 As shown in the figure, the trend of the fusion parameter in the expression-driven method and related technologies of this disclosure changes with the difference between the initial left expression coefficient and the initial right expression coefficient. In this disclosure embodiment, the fusion parameter is the similarity between the initial left expression coefficient and the initial right expression coefficient. The corresponding change curve 401 in this disclosure embodiment shows that the fusion parameter changes dynamically with the difference between the initial left expression coefficient and the initial right expression coefficient. The larger the difference, the smaller the fusion parameter, eventually approaching 0. The smaller the difference, the larger the fusion parameter, eventually approaching 1. The corresponding line 402 in the related technologies of this disclosure shows that when the constraint is increased by using a direct averaging function, the fusion parameter is the specific parameter when the averaging function is processed. The fusion parameter is fixed at 0.5 and will not change, causing specific expressions such as single eye closure to fail. Here, the specific expression can be an expression with a large difference between the initial left expression coefficient and the initial right expression coefficient.

[0057] This solution provides an expression-driven method that can perform post-processing relational constraints on the expression coefficients of image recognition, avoiding the cost of redesigning the training algorithm. It effectively improves the consistency and synchronization of driving expressions with virtually no additional time and space overhead. Furthermore, it avoids possible expression failures and enhances the realism of expression-driven processing through a similarity function.

[0058] Figure 5 This is a schematic diagram of the structure of an expression-driven device provided in an embodiment of this disclosure. The device can be implemented by software and / or hardware and is generally integrated into an electronic device. Figure 5 As shown, the device includes: The recognition module 501 is used to acquire a target image and recognize the facial expression of the target object in the target image to obtain at least one facial expression coefficient; The determining module 502 is used to extract the expression coefficients to be processed from the at least one expression coefficient, and to determine the initial left expression coefficient and the initial right expression coefficient from the expression coefficients to be processed; Processing module 503 is used to generate target left expression coefficient and target right expression coefficient based on the similarity between the initial left expression coefficient and the initial right expression coefficient; The driving module 504 is used to drive the virtual image to make corresponding expressions based on the target left expression coefficient and the target right expression coefficient.

[0059] Optionally, the identification module 501 is used for: The target image is input into a pre-trained facial expression recognition network to obtain at least one facial expression coefficient.

[0060] Optionally, the determination module 502 is used for: Extract the expression coefficients for a preset part from the at least one expression coefficient to obtain the expression coefficient to be processed, wherein the preset part has two symmetrical sub-parts on the left and right.

[0061] Optionally, the processing module 503 includes: A similarity unit is used to determine the similarity between the initial left expression coefficient and the initial right expression coefficient using a similarity function; An adjustment unit is used to adjust the initial left expression coefficient and the initial right expression coefficient based on the similarity, respectively, to obtain the target left expression coefficient and the target right expression coefficient.

[0062] Optionally, the initial left expression coefficient includes multiple left-dimensional data, and the initial right expression coefficient includes multiple right-dimensional data. The number of multiple left-dimensional data is the same as the number of multiple right-dimensional data, and each left-dimensional data corresponds to one right-dimensional data. The similarity between the initial left expression coefficient and the initial right expression coefficient is determined in units of dimensional data pairs. There are multiple dimensional data pairs, and each dimensional data pair includes a corresponding left dimensional data and a right dimensional data.

[0063] Optionally, similar units are used for: Determine the difference between the initial left expression coefficient and the initial right expression coefficient; The difference is modulated using modulation parameters, and the modulated difference is used as a variable in a similarity function to calculate the similarity between the initial left expression coefficient and the initial right expression coefficient.

[0064] Optionally, the modulation parameters correspond to the expression coefficients to be processed, and the similarity function is a decreasing linear function or a nonlinear function.

[0065] Optionally, the adjustment unit is used for: Based on the similarity, the degree of difference is determined; The initial left expression coefficient and the initial right expression coefficient are weighted based on the similarity and the difference, respectively, to obtain the target left expression coefficient and the target right expression coefficient.

[0066] The facial expression driving device provided in this disclosure can execute the facial expression driving method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0067] This disclosure also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the expression-driven method provided in any embodiment of this disclosure.

[0068] Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of the present disclosure. See below for details. Figure 6 The diagram illustrates a structural schematic suitable for implementing the electronic device 600 in the embodiments of this disclosure. The electronic device 600 in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0069] like Figure 6 As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0070] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0071] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the expression-driven method of embodiments of this disclosure.

[0072] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0073] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0074] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0075] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire a target image and identify the facial expression of a target object in the target image to obtain at least one facial expression coefficient; extract a facial expression coefficient to be processed from the at least one facial expression coefficient and determine an initial left facial expression coefficient and an initial right facial expression coefficient from the facial expression coefficient to be processed; generate a target left facial expression coefficient and a target right facial expression coefficient based on the similarity between the initial left facial expression coefficient and the initial right facial expression coefficient; and drive a virtual avatar to make a corresponding facial expression based on the target left facial expression coefficient and the target right facial expression coefficient.

[0076] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0077] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0078] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0079] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0080] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0081] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0082] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0083] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0084] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0085] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An expression-driven method, characterized in that, include: Acquire a target image and identify the facial expression of the target object in the target image to obtain at least one facial expression coefficient; Extract the expression coefficient to be processed from the at least one expression coefficient, and determine the initial left expression coefficient and the initial right expression coefficient from the expression coefficient to be processed; wherein, the initial left expression coefficient and the initial right expression coefficient are initial coefficients obtained after grouping the expression coefficient to be processed into left and right groups based on semantic information; The similarity between the initial left expression coefficient and the initial right expression coefficient is determined using a similarity function; Based on the similarity, the initial left expression coefficient and the initial right expression coefficient are adjusted respectively to obtain the target left expression coefficient and the target right expression coefficient; Based on the target left expression coefficient and the target right expression coefficient, the virtual avatar is driven to make corresponding expressions; The step of adjusting the initial left expression coefficient and the initial right expression coefficient based on the similarity to obtain the target left expression coefficient and the target right expression coefficient includes: Based on the similarity, the degree of difference is determined; The initial left expression coefficient and the initial right expression coefficient are weighted based on the similarity and the difference, respectively, to obtain the target left expression coefficient and the target right expression coefficient.

2. The method according to claim 1, characterized in that, Identify the facial expression of the target object in the target image to obtain at least one facial expression coefficient, including: The target image is input into a pre-trained facial expression recognition network to obtain at least one facial expression coefficient.

3. The method according to claim 1, characterized in that, Extracting the expression coefficient to be processed from the at least one expression coefficient includes: Extract the expression coefficients for a preset part from the at least one expression coefficient to obtain the expression coefficient to be processed, wherein the preset part has two symmetrical sub-parts on the left and right.

4. The method according to claim 1 or 3, characterized in that, The initial left expression coefficient includes multiple left-dimensional data, and the initial right expression coefficient includes multiple right-dimensional data. The number of multiple left-dimensional data is the same as the number of multiple right-dimensional data, and each left-dimensional data corresponds to one right-dimensional data. The similarity between the initial left expression coefficient and the initial right expression coefficient is determined in units of dimensional data pairs. There are multiple dimensional data pairs, and each dimensional data pair includes a corresponding left dimensional data and a right dimensional data.

5. The method according to claim 3, characterized in that, Determining the similarity between the initial left expression coefficient and the initial right expression coefficient using a similarity function includes: Determine the difference between the initial left expression coefficient and the initial right expression coefficient; The difference is modulated using modulation parameters, and the modulated difference is used as a variable in a similarity function to calculate the similarity between the initial left expression coefficient and the initial right expression coefficient.

6. The method according to claim 5, characterized in that, The modulation parameters correspond to the expression coefficients to be processed, and the similarity function is a decreasing linear function or a nonlinear function.

7. An expression-driving device, characterized in that, include: The recognition module is used to acquire a target image and recognize the facial expression of the target object in the target image to obtain at least one facial expression coefficient. The determining module is used to extract the expression coefficients to be processed from the at least one expression coefficient, and to determine the initial left expression coefficient and the initial right expression coefficient from the expression coefficients to be processed; wherein the initial left expression coefficient and the initial right expression coefficient are initial coefficients obtained after grouping the expression coefficients to be processed into left and right groups based on semantic information; A similarity determination module is used to determine the similarity between the initial left expression coefficient and the initial right expression coefficient using a similarity function; An adjustment module is used to adjust the initial left expression coefficient and the initial right expression coefficient based on the similarity, respectively, to obtain the target left expression coefficient and the target right expression coefficient; The driving module is used to drive the virtual image to make corresponding expressions based on the target left expression coefficient and the target right expression coefficient; The adjustment module is specifically used for: Based on the similarity, the degree of difference is determined; The initial left expression coefficient and the initial right expression coefficient are weighted based on the similarity and the difference, respectively, to obtain the target left expression coefficient and the target right expression coefficient.

8. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the expression-driven method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the expression-driven method according to any one of claims 1-6.

Images