Digital human expression binding method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a digital person expression binding method and device, electronic equipment and storage medium, belonging to the technical field of augmented reality, virtual reality, computer vision, deep learning and the like. The specific implementation scheme is: based on each frame expression in the collected expression sequence, configuring a plurality of morphing control units of the digital person face model to execute each frame expression; configuring a plurality of controllers for the digital person face model; based on the binding relationship between the plurality of pre-configured controllers and the plurality of morphing control units, binding the plurality of controllers and the plurality of morphing control units, so that each controller can control the bound morphing control unit, and further control the digital person face model to express each frame expression in the expression sequence. The technology of the present disclosure can bind the expressions in the expression sequence for the digital person, effectively improve the accuracy and binding efficiency of the digital person expression binding, and further enable the digital person to express the expression sequence very naturally, beautifully and vividly.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to augmented reality, virtual reality, computer vision, deep learning, and other technical fields, and can be applied to scenarios such as the metaverse and virtual digital humans. Specifically, it relates to a method, apparatus, electronic device, and storage medium for binding facial expressions to a digital human. Background Technology

[0002] A digital human (or meta human) is a digitally created human figure that closely resembles a human being.

[0003] Digital humans are a product of the integration of information science and life science, utilizing information science methods to virtually simulate the human body at different levels of form and function. Currently, a large number of digital humans have emerged in fields such as film, games, live streaming, industry, and advertising. Because digital humans simulate humans through technological means, their appearance is very similar to their corresponding real-life counterparts. Through techniques such as facial expression binding and real-time driving, digital humans can simulate human facial expressions and perform corresponding physical movements. Summary of the Invention

[0004] This disclosure provides a method, apparatus, electronic device, and storage medium for binding facial expressions to a digital human.

[0005] According to one aspect of this disclosure, a method for binding facial expressions to a digital human is provided, comprising:

[0006] Based on each frame of the captured facial expression sequence, multiple deformation control units are configured to control the digital human's facial model to execute the facial expressions in each frame;

[0007] Configure multiple controllers for the digital human's facial model;

[0008] Based on the pre-configured binding relationship between multiple controllers and multiple deformation control units, the multiple controllers and multiple deformation control units are bound together, so that each controller can control the bound deformation control unit, thereby controlling the digital human's facial model to express each frame of the expression sequence.

[0009] According to another aspect of this disclosure, a digital human facial expression binding device is provided, comprising:

[0010] The deformation control unit configuration module is used to configure multiple deformation control units for controlling the digital human's facial model to execute the expressions in each frame based on the acquired expression sequence.

[0011] The controller configuration module is used to configure multiple controllers for the digital human's facial model.

[0012] The binding module is used to bind the multiple controllers and the multiple deformation control units based on the pre-configured binding relationship between multiple controllers and multiple deformation control units, so that each controller can control the bound deformation control unit, thereby controlling the digital human's facial model to express each frame of the expression sequence.

[0013] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0014] At least one processor; and

[0015] A memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods described above and any possible implementations.

[0017] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform the methods described above and any possible implementation thereof.

[0018] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the aspects and any possible implementations described above.

[0019] According to the technology disclosed herein, it is possible to bind expressions in an expression sequence to a digital human, effectively improving the accuracy and efficiency of expression binding, thereby enabling the digital human to express expression sequences very naturally, beautifully and vividly.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0021] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0022] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure;

[0023] Figure 2 This is a schematic diagram according to the second embodiment of the present disclosure;

[0024] Figure 3This is a schematic diagram of an eyelid provided in this disclosure;

[0025] Figure 4 This is a schematic diagram of a lacrimal gland provided in this publication;

[0026] Figure 5 This is a schematic diagram according to the third embodiment of the present disclosure;

[0027] Figure 6 This is a schematic diagram according to the fourth embodiment of the present disclosure;

[0028] Figure 7 This is a block diagram of an electronic device used to implement the methods of the embodiments of this disclosure. Detailed Implementation

[0029] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0030] Obviously, the described embodiments are only some, not all, of the embodiments disclosed herein. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0031] It should be noted that the terminal devices involved in the embodiments of this disclosure may include, but are not limited to, smart devices such as mobile phones, personal digital assistants (PDAs), wireless handheld devices, and tablet computers; the display devices may include, but are not limited to, personal computers, televisions, and other devices with display functions.

[0032] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0033] In existing technologies, digital human facial expression binding technology involves capturing the last frame of a person's facial expression and binding it to the digital human model, thereby driving the digital human to make that expression in a natural state. However, this facial expression binding technology is simple, crude, and has poor accuracy, resulting in very unnatural facial expressions when the digital human is driven.

[0034] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure; as shown Figure 1 As shown in the figure, this embodiment provides a method for binding facial expressions to a digital human, which may specifically include the following steps:

[0035] S101. Based on each frame of the acquired facial expression sequence, configure multiple deformation control units to control the digital human's facial model to execute each frame of facial expression;

[0036] The execution subject of the digital human expression binding method in this embodiment can be a digital human expression binding device. The device can be a physical electronic device, or it can be an application or platform integrated with software, which can bind the expressions of the digital human, especially the expression sequence of the digital human.

[0037] The facial expression sequence collected in this embodiment includes at least one intermediate frame and one final frame. Each intermediate frame and the final frame is three-dimensional (3D). Combined with the temporal relationship of each frame in the facial expression sequence, the facial expression sequence in this embodiment can be called a 4D facial expression.

[0038] This embodiment is mainly used to implement facial expression binding for digital humans, so it mainly involves the facial expression binding of facial features in the digital human's facial model. These facial features may include eyebrow features, eye features, nose features, mouth features, and ear features, etc.

[0039] In this embodiment, the configured multiple deformation control units can control the digital human's facial model to execute expressions in each frame. Thus, for each expression sequence, the multiple deformation controllers sequentially control the digital human's facial model to execute each frame of expression, thereby enabling control of the expression sequence and allowing the digital human's facial model to express and display 4D expressions.

[0040] S102, Configure multiple controllers for the digital human's facial model;

[0041] S103. Based on the pre-configured binding relationship between multiple controllers and multiple deformation control units, multiple controllers and multiple deformation control units are bound together, so that each controller can control the bound deformation control unit, thereby controlling the digital human's facial model to express each frame of the expression sequence.

[0042] For example, in order to achieve more precise driving and control of the facial expressions of the digital human, in this embodiment, multiple controllers can be configured for the facial model of the digital human, so as to bind multiple controllers and multiple deformation control units, thereby achieving more precise control of the facial expression sequence of the digital human through multiple controllers and multiple deformation control units.

[0043] Specifically, the deformation control unit configured in this embodiment is used to control the deformation of the digital human's facial model, thereby enabling it to make facial expressions. However, the deformation control unit itself cannot be actively triggered. Therefore, in this embodiment, multiple controllers need to be pre-configured for the digital human's facial model when binding facial expressions. These controllers are used to trigger the deformation control unit, thereby enabling the deformation control unit to control the digital human's facial model to make facial expressions.

[0044] In this embodiment, the number of controllers configured is related to the number of deformation control units, so as to achieve precise control of the deformation control units by the controllers. The number of deformation control units is related to the number of expression frames in the expression sequence, so as to achieve control over each expression frame.

[0045] In this embodiment, researchers can pre-configure the binding relationships between multiple controllers and multiple deformation control units according to the desired driving effect. In this way, the digital human's expression binding device can bind multiple controllers and multiple deformation control units based on the pre-configured binding relationships, enabling each controller to control the actions of the bound deformation control units, which in turn control the digital human's facial model to make corresponding expressions.

[0046] The digital human expression binding method of this embodiment can bind expression sequences to the digital human's facial model, thereby enabling the digital human to have the ability to express expression sequences. Compared with the existing technology where digital humans can only express the last frame of an expression, this method can effectively improve the accuracy and efficiency of digital human expression binding, and thus enable the digital human to express expression sequences very naturally, beautifully and vividly.

[0047] Figure 2 This is a schematic diagram based on the second embodiment of this disclosure; the digital human expression binding method of this embodiment, in the above... Figure 1 Based on the technical solutions of the illustrated embodiments, the technical solutions of this disclosure will be described in further detail. For example... Figure 2 As shown, the digital human expression binding method in this embodiment may specifically include the following steps:

[0048] S201. Based on each frame of the collected expression sequence, each facial feature in the expression is divided into at least two regions;

[0049] S202. Configure the first deformation control unit for each region of each facial feature corresponding to each frame of the expression sequence to obtain multiple first deformation control units;

[0050] The deformation control unit in this embodiment can be a control unit generated based on blended deformation (BS), or simply a BS unit.

[0051] Steps S201-S202 are as described above. Figure 1 One implementation of step S101 in the illustrated embodiment. In this implementation, a first deformation control unit is configured for each region of each facial feature corresponding to each frame of the facial expression sequence.

[0052] This implementation method allows for the configuration of a first deformation control unit for each region of facial features corresponding to each frame of an expression sequence. This enables more precise control over the deformation of each frame of an expression, thereby further improving the accuracy and efficiency of expression binding.

[0053] Specifically, each facial feature region in each frame of the expression includes multiple points. In this implementation, the first deformation control unit is used to control the points in the region corresponding to the facial feature of each frame of the expression sequence to deform from the natural state of the facial model to the point position of the point information identifier of the corresponding region of the facial feature in the expression.

[0054] In this embodiment, each frame of the expression sequence can correspond to at least one facial feature. For example, some frames may only include mouth movements; while others may include both mouth and nose movements; and still others may include mouth, nose, and eye movements. Regardless of the type of expression frame, in this embodiment, each facial feature in each frame needs to be divided into at least two regions. Then, a first deformation control unit is configured for each region of the facial feature corresponding to each frame of the expression sequence. This allows each first deformation control unit to control the points in the corresponding region of the facial feature in the digital human's facial model to deform from the natural state of the facial model to the point position identified by the point information of the corresponding region of the facial feature in the expression.

[0055] The natural state refers to the standard state of the digital human's facial model when it is not making any expressions. Based on the above principle, the deformation control unit of this embodiment controls the deformation of points in each region of each facial feature in the expression, which can be called the process of controlling the corresponding expression from 0 to 1.

[0056] Each expression in the expression sequence of this embodiment can be identified by the point information of deformation in the facial model when expressing the expression, compared with the natural state of the facial model.

[0057] For example, step S201 in this embodiment may include the following situations when it is specifically implemented:

[0058] Scenario 1: For each frame of an expression sequence, if the facial features in the expression include mouth features, and the movement of the mouth in the expression is an active movement, the mouth features in the expression are divided into four regions: upper, lower, left, and right.

[0059] Scenario 2: For each frame of an expression sequence, if the facial features in the expression include mouth features, and the movement of the mouth in the expression is a passive movement, the mouth features in the expression are split into two regions, left and right.

[0060] For example, active mouth movements can be the movement of the lips in a certain direction. Due to this movement, the deformation of the mouth is usually quite large. In order to achieve more precise control over this expression, the mouth features in the expression can be divided into four regions: upper, lower, left, and right.

[0061] Passive movement of the mouth can refer to the movement of the mouth caused by the movement of other facial features, such as the movement of the nose causing the movement of the mouth. In this case, the mouth features in the expression can be divided into two areas, left and right.

[0062] Scenario 3: For each frame of an expression sequence, if the facial features in the expression include eye features, ear features, nose features, or eyebrow features, split the corresponding eye features, ear features, nose features, or eyebrow features in the expression into two regions, left and right.

[0063] By adopting the above method, the facial features of each expression in the expression sequence can be more finely divided into regions, so as to configure the deformation control unit more accurately and achieve more precise expression binding.

[0064] Further optionally, before step S201, the method may include: registering the expression model containing the expression with the digital human's facial model.

[0065] Furthermore, before registering the facial expression model with the digital human's facial model, the process includes: acquiring facial expression sequences. In this embodiment, the acquired facial expression sequences are 4D expressions. According to pre-configured specifications, the person performing the facial expression can be specified to perform each frame of the sequence, thereby acquiring the corresponding expressions.

[0066] In this embodiment, each frame of the 4D expression sequence is a 3D expression. The expression model of the 3D expression should be registered with the digital human's facial model to achieve the same coordinate system direction and size.

[0067] Even if the expression model and the digital human's facial model have the same coordinate system and size at each frame of the captured expression sequence, the person performing the expression is a moving human and will inevitably move, leading to some errors. To further improve the accuracy of the captured expression binding, this embodiment also requires registering the expression model of each frame of the expression sequence with the digital human's facial model.

[0068] In this embodiment, each frame of expression acquired during the expression sequence acquisition stage is in point cloud format, containing a large number of points. To match the digital human's facial model, low-dimensional topological modeling is required for each frame of the acquired expression sequence, resulting in the expression represented within the digital human's facial model. Each frame of the resulting expression sequence can then be referred to as an expression model. That is, the expression model identifies the points within the deformed facial features compared to the natural state of the facial model when expressing that expression, along with the positional information of each point.

[0069] Further, optionally, in one embodiment of this disclosure, step S202 may include the following steps:

[0070] (1) Obtain the point information of each region of each facial feature corresponding to each frame of the expression sequence;

[0071] For example, if the facial features corresponding to each frame of an expression sequence include nose features, eyebrow features, or mouth features, the identifiers, coordinates, and weight information of points in each region of the facial features corresponding to the expression are obtained; the weight information of the points is used to smooth the deformation.

[0072] In other words, for facial features with significant differences in deformation between the left and right regions, it is necessary to obtain the weight information of points in each region in order to express expressions more accurately. The point weight information can be configured to gradually decrease from the boundary line of the region outwards. For example, the weight of points on the boundary line of each region can be configured to 0.5, that is, each adjacent region accounts for half. For the weight information of other points within a region, it can be gradually reduced as the distance from the boundary line increases. The point weight information is used to adjust the deformation of the point position information, making the feature curves on the facial features of each frame of expression smoother.

[0073] If the facial features corresponding to each frame of an expression sequence include eye features or ear features, obtain the identifiers and coordinates of points in each region of the facial features corresponding to the expression. For these two facial features, the deformation difference between the left and right partitions is usually not significant, so the weight information of the points can be ignored. Alternatively, in this case, weight information can also be configured in the point information of each region. In this case, the weight information of all points corresponding to the eye features or ear features is the same, for example, all can be 1.

[0074] (2) Based on the point information of each region of each facial feature corresponding to each frame of the expression sequence, configure the first deformation control unit of each region of each facial feature corresponding to each frame of the expression sequence to obtain multiple first deformation control units.

[0075] In other words, for each region of each facial feature corresponding to each frame of the expression sequence, a corresponding first deformation control unit is configured to control the point of that region of the facial feature corresponding to that frame of the expression sequence to deform from its natural state to the position of the point information identifier.

[0076] In practical applications, for the acquired 4D multiple expression sequences, the first deformation control unit for each region of each facial feature corresponding to each frame of each expression sequence can be configured in the manner described above, resulting in a total of multiple first deformation control units.

[0077] Optionally, in one embodiment of this disclosure, the expression sequence may include at least one intermediate frame expression and one final frame expression that have a sequential relationship. In this case, after step S202, the following steps may also be included:

[0078] (a1) If the expression in the last frame is adjusted, obtain the difference adjustment information between the expression in the nearest neighboring middle frame and the expression in the last frame relative to the expression in the last frame;

[0079] The nearest neighbor intermediate frame expression of the last frame's expression refers to the intermediate frame expression in the expression sequence that is the closest neighbor to the last frame's expression, preceding it. For example, an expression sequence might include five frames: frame 1, frame 2, frame 3, frame 4, and frame 5. In this case, frame 5 is the last frame's expression, frame 1 can be a natural expression, and frames 2, 3, and 4 can be intermediate frame expressions.

[0080] In practical applications, if developers find the expression in the last frame insufficient, they can adjust it. Specifically, they can adjust the position of some points in the last frame's expression. Based on this adjustment, they can obtain the first adjustment information of the last frame's expression before adjustment, relative to the nearest neighboring middle frame expression in the expression sequence (the second-to-last frame). They can then obtain the second adjustment information of the adjusted last frame's expression, relative to its nearest neighboring middle frame expression. Based on these first and second adjustment information, they can determine the adjustment difference, representing the difference in adjustment information between the last frame's expression and its nearest neighboring middle frame expression. In this embodiment, the difference in adjustment information includes not only the magnitude of the adjustment displacement but also the direction of the adjustment.

[0081] (b1) Based on the difference adjustment information of the last frame expression, according to the preset adjustment constraints, the difference adjustment information of each intermediate frame expression relative to the previous nearest neighbor expression is obtained sequentially from the back to the front in the expression sequence; the preset adjustment constraints require that the difference adjustment magnitude of each expression in the expression sequence decreases from the back to the front.

[0082] For example, following step (a1) above, the difference adjustment information of frame 5 relative to frame 4 in the example above can be obtained. Then, referring to the preset adjustment constraints that limit the decreasing difference adjustment magnitude of each expression in the expression sequence from back to front, the difference adjustment information of each expression can be determined according to the order from back to front in the expression sequence. Assuming the adjustment magnitude of the difference adjustment information of frame 5 relative to frame 4 is configured to 1, the position of point marker 1 in frame 5 relative to frame 4 is adjusted from increasing by 1 unit to increasing by 2 units, meaning the position of point 1 in the difference adjustment information of point marker 1 increases by 1 unit. According to the preset adjustment constraints, the adjustment magnitude of the difference adjustment information of frame 4 relative to frame 3 can be configured to 0.8. Again, taking point marker 1 as an example, in the adjusted frame 4 relative to frame 3, the adjustment magnitude of point marker 1 increases by 0.8 units based on the original adjustment information. By analogy, the difference adjustment information of all intermediate frames relative to the previous nearest neighbor expression can be obtained.

[0083] (c1) Based on the difference adjustment information corresponding to the expression in the last frame and the difference adjustment information corresponding to the expressions in each intermediate frame, update the first deformation control unit corresponding to the expression in the last frame and the first deformation control unit corresponding to the expressions in each intermediate frame, respectively.

[0084] The difference adjustment information corresponding to each expression in the last frame and each intermediate frame can identify the position of each point in the facial features corresponding to the expression when the expression is executed. At this time, the adjustment information can be followed to control the accurate expression of the adjusted expression in a timely manner. Based on the difference adjustment information corresponding to the last frame expression and the difference adjustment information corresponding to each intermediate frame expression, the first deformation control unit corresponding to the last frame expression and the first deformation control unit corresponding to each intermediate frame expression can be updated respectively.

[0085] Of course, the above example of an expression sequence with 5 frames is just an example. In actual applications, an expression sequence can include at least one intermediate frame, depending on the complexity of the expression. For example, some very simple expression sequences may only include one intermediate frame, while some very complex expression sequences may include more than 10 intermediate frames. The specific setting depends on the actual situation and is not limited here.

[0086] By using the above adjustment method, if the final frame expression in an expression sequence is adjusted, each intermediate frame can be non-linearly adjusted according to steps (a1)-(c1). This ensures that the adjustment range of intermediate frames closer to the final frame expression is closer to the adjustment range of the final frame expression, while the adjustment range of intermediate frames farther from the final frame expression is smaller. This makes the adjustment of the expression sequence more natural, and the expression of the adjusted expression sequence smoother and more aesthetically pleasing. Moreover, in this embodiment, after adjusting the expression sequence according to the above method, the difference adjustment information between the final frame expression and each intermediate frame expression can be obtained. Based on the difference adjustment information between the final frame expression and each intermediate frame expression, the first deformation control unit corresponding to the final frame expression and the first deformation control unit corresponding to each intermediate frame expression can be updated respectively, without reconfiguring the corresponding deformation control unit, which can effectively improve the efficiency of expression binding.

[0087] S203. For each frame of the expression sequence, if the facial features of the expression include eye features, based on the first preset rule, configure the second deformation control unit of the eyelid in the digital human's facial model to control the eyelid to follow the expression.

[0088] In this embodiment, the eye shell is an attachment template in the digital human's facial model, also known as an AO template. When the digital human expresses facial expressions, if the facial features of the expression include eye features, such as blinking or closing the eyes, the position of the points of the eye features changes. This inevitably leads to a change in the positional relationship between the points in the eye features and the eye shell template, for example, causing problems such as clipping. Therefore, in this embodiment, a second deformation control unit for the eye shell in the digital human's facial model needs to be configured accordingly to control the eye shell to follow the expression, making the positional relationship between the eye shell and the eye features more reasonable, natural, and aesthetically pleasing.

[0089] For example, in the specific implementation of step S203, for each frame of expression in the expression sequence, if the facial features of the expression include eye features, based on the first preset rule, the second deformation control unit of the eye shell in the digital human's facial model can be configured so that the second deformation control unit of the eye shell can control the deformation of the points on the two outer ring feature curves of the eye shell to be consistent with the deformation of the nearest neighbor point on the feature curve where the eyelid is located, and the deformation of the points on the other feature curves decreases towards the center, and the points on the other feature curves expand outward of the eyeball according to the preset expansion rule.

[0090] For example, Figure 3 This is a schematic diagram of an eyelid provided in this disclosure. For example... Figure 3 As shown, the black area shaped like an eye represents the eyelid shell, an accessory template that forms a shell on the surface of the eyes in a digital human's facial model, making the eyes appear more natural and aesthetically pleasing. This eyelid shell enlarges as the eyes open and gradually shrinks as the eyes close, always covering the eye surface. Figure 3 As shown, the eyelid shell includes two outer characteristic curves at the seam where it meets the eyelid. The interior includes four characteristic curves.

[0091] In this embodiment, based on the first preset rule, the second deformation control unit of the eyelid in the configured digital human facial model can control the deformation of points on the two outer ring feature curves of the eyelid to be consistent with the deformation of the nearest neighbor points on the feature curve where the eyelid is located. This ensures that the eyelid and eyelid can fit together in real time regardless of the facial expression. Moreover, to ensure the smoothness of the deformation, the deformation of points on the other feature curves, such as the four middle feature curves, decreases towards the center. That is, the deformation of points on the two outer ring feature curves is the largest, and then the deformation of points on the four middle feature curves gradually decreases as the distance from the outer ring feature curves increases, making the deformation more reasonable and natural.

[0092] In addition, the second deformation control unit can also control the points on the four characteristic curves in the middle to expand outwards from the eyeball according to a preset expansion rule, so that the eyelid shell maintains a distance from the eyeball and avoids mold insertion. For example, the preset expansion rule can require that the expansion coefficient of the points in the central region of the eyelid shell is greater than the expansion coefficient of the points far from the central region.

[0093] For example, the second deformation control unit can control the points on the two central feature curves among the four central feature curves to expand outwards from the eyeball according to a first expansion coefficient; the points on the remaining two central feature curves expand outwards from the eyeball according to a second expansion coefficient, where the first expansion coefficient is greater than the second expansion coefficient. This expansion method allows the second deformation control unit to maintain a consistent distance between the eyelid and the eyeball, avoiding mold insertion and improving the accuracy and effectiveness of the digital human's facial expressions.

[0094] By adopting the above method, the second deformation control unit of the eyelid can flexibly and efficiently control the deformation of the eyelid, so that the eyelid can follow the facial expressions of the eye features in real time.

[0095] S204. For each frame of the expression sequence, if the facial features of the expression include eye features, based on the second preset rule, configure the third deformation control unit of the lacrimal gland in the digital human's facial model to control the lacrimal gland to follow the expression.

[0096] Similar to the implementation principle of step S203 above, the tear gland in this step is also an accessory template set in the digital human's facial model, which can present the digital human more naturally and aesthetically. For example, Figure 4 This is a schematic diagram of a lacrimal gland provided in this publication. (For example...) Figure 4 As shown, the black area within the eye-shaped region represents the lacrimal gland area. The lacrimal gland includes two outer characteristic curves and one inner characteristic curve, as well as the corner characteristic curve located within the inner characteristic curve. This lacrimal gland template also needs to move in accordance with eye expressions.

[0097] For example, when this step S204 is specifically implemented, it may include: for each frame of expression in the expression sequence, if the facial features of the expression include eye features, based on the second preset rule, configuring the third deformation control unit of the lacrimal gland in the digital human's facial model, so that the third deformation control unit of the lacrimal gland can control the deformation of the points on the two outer ring feature curves of the lacrimal gland to be consistent with the deformation of the nearest neighbor points on the feature curve where the eyelid is located, the deformation of the points on the inner ring feature curve and the points on the corner of the eye feature curve decreases towards the center, and the points on the corner of the eye feature curve and the points on the inner ring feature curve expand outward of the eyeball according to the preset expansion rule.

[0098] In this embodiment, based on the second preset rule, the third deformation control unit of the lacrimal gland in the configured digital human facial model can control the deformation of points on the two outer characteristic curves of the lacrimal gland to be consistent with the deformation of the nearest neighbor points on the characteristic curve where the eyelid is located. This ensures that the lacrimal gland and eyelid can fit together in real time regardless of the facial expression. Moreover, to ensure the smoothness of the deformation, the deformation of points on the inner characteristic curve and the corner of the eye characteristic curve decreases towards the center. That is, the deformation of points on the inner characteristic curve is less than the deformation of points on the two outer characteristic curves, and the deformation of points on the corner of the eye characteristic curve is less than the deformation of points on the inner characteristic curve. This allows the lacrimal gland to not only accurately deform with facial expressions, but also to deform more reasonably and naturally.

[0099] Furthermore, the third deformation control unit can also control the points on the corner of the eye feature curve and the inner circle feature curve to expand outwards from the eyeball according to a preset expansion rule, ensuring that the eyelid maintains a distance from the eyeball and avoids insertion. For example, the preset expansion rule can require that the expansion coefficient of the points on the corner of the eye feature curve located in the central region is greater than the expansion coefficient of the points on the inner circle feature curve. Through this expansion method, the third deformation control unit can ensure that the lacrimal gland and the eyeball always maintain a distance, avoiding insertion and improving the accuracy and effectiveness of the digital human's facial expressions.

[0100] By adopting the above method, the third deformation control unit of the lacrimal gland can flexibly and efficiently control the deformation of the lacrimal gland, so that the lacrimal gland can follow the facial expressions of the eye features in real time.

[0101] Steps S203 and S204 above can also be used as described above. Figure 1 In the two implementations of step S101 in the illustrated embodiment, although the deformation of the eyelid and the lacrimal gland is configured in these two implementations respectively, it is also to enable the eyelid and the lacrimal gland to follow the eye feature expressions included in the expression sequence. Therefore, it also belongs to the deformation control unit configured to control the facial model of the digital human to perform expressions based on expressions.

[0102] S205. For a pre-configured combination of at least two expressions of the same facial feature, based on the at least two expressions, the original deformation control unit corresponding to each of the at least two expressions, and the combination expression, configure a deformation control unit for the combination expression and a deformation control unit for the difference expression, wherein the difference expression is used to identify the difference between the superposition of the at least two expressions and the combination expression.

[0103] It should be noted that in practical applications, the expressions in the collected facial expression sequences are limited. To further enrich the expressions, this embodiment can obtain combined expressions by combining at least two expressions with the same facial feature. Furthermore, a deformation control unit for the combined expressions is configured so that the combined expressions can also be controlled and executed. The at least two expressions can be the expressions in each frame of the facial expression sequence in the above embodiment, and the original deformation control unit corresponding to each expression can be the first deformation control unit in the above embodiment.

[0104] However, in the process of generating combined expressions, for example, researchers determine that combining expression 1 and expression 2 can produce expression 3. However, directly superimposing the deformation generated by the original deformation control unit corresponding to expression 1 and the deformation generated by the original control unit corresponding to expression 2 will not produce expression 3. At this time, there is a difference between the two. Based on step S205 of this embodiment, deformation control units for combined expressions and deformation control units for different expressions can be configured to facilitate the binding of more expressions, so that the digital human's facial model can express more expressions.

[0105] For example, step S205 may include the following steps in its specific implementation:

[0106] (a2) For a pre-configured combination of at least two expressions of the same facial feature, configure a difference expression based on the at least two expressions, the original deformation control unit corresponding to each of the at least two expressions, and the combination expression;

[0107] (b2) Configure a fifth deformation control unit for differential facial expressions;

[0108] (c2) Based on the original deformation control unit corresponding to each of the at least two expressions and the fifth deformation control unit of the difference expressions, configure the fourth deformation control unit of the combined expressions.

[0109] For example, the deformation of the original deformation control unit of expression 1 can be superimposed with that of expression 2, and the difference between the superimposed result and expression 3 can be calculated as the difference expression. Then, based on the difference expression and the natural facial model of the digital human, a fifth deformation control unit can be configured for the difference expression. Next, the deformation of the original deformation control unit of expression 1 can be superimposed with that of expression 2, and then the deformation of the difference expression controlled by the fifth deformation control unit can be subtracted to obtain the deformation of the combined expression, and a corresponding fourth deformation control unit for the combined expression can be configured.

[0110] In this embodiment, the deformation control unit for combined facial expressions and the deformation control unit for differential facial expressions configured in step S205 are as described above. Figure 1In a supplementary way to step S101 of the illustrated embodiment, a larger number of deformation control units can be configured for the digital human's facial model to achieve more expression binding.

[0111] S206. Configure multiple controllers for the facial model of the digital human;

[0112] S207. Based on the binding relationship between the pre-configured multiple controllers and the configured multiple deformation control units, bind at least one deformation control unit in each control direction of each controller, so that each controller can control the bound deformation control unit, thereby controlling the digital human's facial model to express each frame of expression in the expression sequence.

[0113] The multiple deformation control units configured in this embodiment include not only the first deformation control units for each region of facial features corresponding to each frame of expression configured in step S202, but also at least one of the second deformation control unit for the eyelid configured in step S203, the third deformation control unit for the lacrimal gland configured in step S204, and the deformation control unit for combined expressions and the deformation control unit for differential expressions configured in step S205. This allows for the effective configuration of deformation control units for various expression controls, thereby enabling the binding of various rich expressions to the digital human and providing effective support for driving various rich expressions. This embodiment takes steps S202 to S205 as an example. In practical applications, only some steps may be executed as needed.

[0114] In this embodiment, the pre-configured binding relationship is exemplified by binding at least one deformation control unit to one control direction of each controller. That is, one control direction of a controller can be bound to at least one deformation control unit to simultaneously control the action of at least one deformation control unit. Each of the at least one deformation control unit can, based on the configured information, control the deformation of a point in a region of a facial feature of the facial model to produce a corresponding expression. Furthermore, based on multiple deformation controllers corresponding to each expression in a series of consecutive frames of expressions, the expression of each frame of expressions can be controlled sequentially, thereby realizing the expression of a digital human's expression sequence.

[0115] In practical applications, in the pre-configured binding relationship between multiple controllers and multiple deformation control units, each controller can also control one or more deformation control units, or other control methods. There are no restrictions here, as long as the controller can control the deformation control unit and thus control the digital human's facial model to express expressions.

[0116] The digital human expression binding method in this embodiment can not only configure the first deformation control unit for each region of each facial feature in each frame of the expression sequence, but also configure the second deformation control unit for the eyelids and the third deformation control unit for the lacrimal glands in the digital human's facial model. This allows the corresponding attachment template to move along with the expression. Furthermore, it can configure deformation control units for combined expressions and for different expressions, effectively configuring deformation control units for various expression controls. Further, based on the pre-configured binding relationship between multiple controllers and multiple configured deformation control units, at least one deformation control unit can be bound to each controller in each control direction. This enables accurate and effective binding of the digital human's expression sequence, allowing for very fine control of the digital human's expressions, enabling the digital human to express expression sequences very naturally, beautifully, and vividly.

[0117] Figure 5 This is a schematic diagram based on the third embodiment of this disclosure; as shown Figure 5 As shown, this embodiment provides a digital human facial expression binding device 500, including:

[0118] The deformation control unit configuration module 501 is used to configure multiple deformation control units for controlling the digital human's facial model to execute the expressions in each frame based on the acquired expression sequence.

[0119] The controller configuration module 502 is used to configure multiple controllers for a digital human's facial model.

[0120] The binding module 503 is used to bind the multiple controllers and the multiple deformation control units based on the pre-configured binding relationship between the multiple controllers and the multiple deformation control units, so that each controller can control the bound deformation control unit, thereby controlling the digital human's facial model to express each frame of the expression sequence.

[0121] The digital human expression binding device 500 of this embodiment achieves the same implementation principle and technical effect as the above-mentioned related method embodiments by using the above-mentioned modules. For details, please refer to the description of the above-mentioned related method embodiments, which will not be repeated here.

[0122] Figure 6 This is a schematic diagram according to the fourth embodiment of the present disclosure; this embodiment provides a digital human expression binding device 600, in the above... Figure 5 Based on the technical solutions of the illustrated embodiments, the technical solutions of this disclosure will be described in further detail. For example... Figure 6 As shown, the digital human expression binding device 600 of this embodiment includes... Figure 5The modules with the same name and function shown are: deformation control unit configuration module 601, controller configuration module 602, and binding module 603.

[0123] The deformation control unit configuration module 601 in this embodiment is used for:

[0124] For each frame of the captured expression sequence, each facial feature in the expression is divided into at least two regions;

[0125] Configure the first deformation control unit for each region of each facial feature corresponding to each frame of the expression sequence to obtain a plurality of first deformation control units.

[0126] Further optionally, in one embodiment of this disclosure, the deformation control unit configuration module 601 is configured to:

[0127] For each frame of the expression sequence, if the facial features in the expression include mouth features, and the movement of the mouth in the expression is an active movement, the mouth features in the expression are divided into four regions: upper, lower, left, and right.

[0128] For each frame of the expression sequence, if the facial features in the expression include mouth features, and the movement of the mouth in the expression is a passive movement, the mouth features in the expression are split into two regions, left and right.

[0129] Further optionally, in one embodiment of this disclosure, the deformation control unit configuration module 601 is configured to:

[0130] For each frame of the expression sequence, if the facial features in the expression include eye features, ear features, nose features, or eyebrow features, the corresponding eye features, ear features, nose features, or eyebrow features in the expression are split into left and right regions.

[0131] Further optional, such as Figure 6 As shown, the digital human expression binding device 600 of this embodiment further includes:

[0132] The registration module 604 is used to register the expression model containing the expression with the digital human's facial model.

[0133] Further optionally, in one embodiment of this disclosure, the deformation control unit configuration module 601 is configured to:

[0134] Obtain point information of each region of each facial feature corresponding to each expression in each frame of the expression sequence;

[0135] Based on the point information of each region of each facial feature corresponding to each frame of the expression sequence, a first deformation control unit is configured for each region of each facial feature corresponding to each frame of the expression sequence, thereby obtaining a plurality of first deformation control units.

[0136] Further optionally, in one embodiment of this disclosure, the deformation control unit configuration module 601 is configured to:

[0137] If the facial features corresponding to the expression in each frame of the expression sequence include nose features, eyebrow features, or mouth features, the identifier, coordinates, and weight information of each point in the facial features corresponding to the expression are obtained; the weight information of the points is used to smooth the deformation.

[0138] If the facial features corresponding to each frame of the expression sequence include eye features or ear features, obtain the identifiers and coordinates of points in each region of the facial features corresponding to the expression.

[0139] Further optionally, in one embodiment of this disclosure, the expression sequence includes at least one middle frame expression and a last frame expression having a sequential relationship.

[0140] Further optional, such as Figure 6 As shown, the digital human expression binding device 600 of this embodiment further includes:

[0141] Update module 605, used for:

[0142] If the last frame expression is adjusted, obtain the nearest neighbor intermediate frame expression relative to the last frame expression and the difference adjustment information of the last frame expression;

[0143] Based on the difference adjustment information of the last frame expression, and in accordance with the preset adjustment constraints, the difference adjustment information of each intermediate frame expression relative to the previous nearest neighbor expression is obtained sequentially from the end to the beginning of the expression sequence; the preset adjustment constraints require that the difference adjustment magnitude of each expression in the expression sequence decreases from the end to the beginning.

[0144] Based on the difference adjustment information corresponding to the last frame expression and the difference adjustment information corresponding to each of the intermediate frame expressions, the first deformation control unit corresponding to the last frame expression and the first deformation control unit corresponding to each of the intermediate frame expressions are updated respectively.

[0145] Further optionally, in one embodiment of this disclosure, the deformation control unit configuration module 601 is further configured to:

[0146] For each frame of the expression sequence, if the facial features of the expression include eye features, a second deformation control unit for the eyelids in the digital human's facial model is configured based on a first preset rule to control the eyelids to follow the expression.

[0147] Further optionally, in one embodiment of this disclosure, the deformation control unit configuration module 601 is configured to:

[0148] For each frame of the expression sequence, if the facial features of the expression include eye features, based on a first preset rule, a second deformation control unit for the eyelid in the digital human's facial model is configured, such that the second deformation control unit for the eyelid can control the deformation of points on the two outer ring feature curves of the eyelid to be consistent with the deformation of the nearest neighbor point on the feature curve where the eyelid is located, and the deformation of points on the remaining feature curves decreases towards the center, and the points on the remaining feature curves expand outwards from the eyeball according to a preset expansion rule.

[0149] Further optionally, in one embodiment of this disclosure, the deformation control unit configuration module 601 is further configured to:

[0150] For each frame of the expression sequence, if the facial features of the expression include eye features, a third deformation control unit of the lacrimal gland in the digital human's facial model is configured based on a second preset rule to control the lacrimal gland to follow the expression.

[0151] Further optionally, in one embodiment of this disclosure, the deformation control unit configuration module 601 is configured to:

[0152] For each frame of the expression sequence, if the facial features of the expression include eye features, based on the second preset rule, the third deformation control unit of the lacrimal gland in the digital human's facial model is configured so that the third deformation control unit of the lacrimal gland can control the deformation of the points on the two outer ring feature curves of the lacrimal gland to be consistent with the deformation of the nearest neighbor points on the feature curve where the eyelid is located. The deformation of the points on the inner ring feature curve and the points on the corner of the eye feature curve decreases towards the center. The points on the corner of the eye feature curve and the points on the inner ring feature curve expand outward of the eyeball according to the preset expansion rule.

[0153] Further optionally, in one embodiment of this disclosure, the deformation control unit configuration module 601 is further configured to:

[0154] For a pre-configured combination of at least two expressions with the same facial feature, a deformation control unit for the combination expression and a deformation control unit for the difference expression are configured based on the at least two expressions, the original deformation control unit corresponding to each of the at least two expressions, and the combination expression. The difference expression is used to identify the difference between the superposition of the at least two expressions and the combination expression.

[0155] Further optionally, in one embodiment of this disclosure, the deformation control unit configuration module 601 is further configured to:

[0156] For a pre-configured combination of at least two expressions with the same facial feature, a difference expression is configured based on the at least two expressions, the original deformation control unit corresponding to each of the at least two expressions, and the combination expression;

[0157] A fifth deformation control unit is configured for the aforementioned differential facial expressions;

[0158] The fourth deformation control unit of the combined expression is configured based on the original deformation control unit corresponding to each of the at least two expressions and the fifth deformation control unit of the difference expression.

[0159] Further optionally, in one embodiment of this disclosure, the binding module 603 is used for:

[0160] Based on the pre-configured binding relationship between multiple controllers and multiple configured deformation control units, at least one deformation control unit is bound to each controller in each control direction, so that each controller can control the bound deformation control unit, thereby controlling the digital human's facial model to express each frame of the expression sequence.

[0161] The digital human expression binding device 600 of this embodiment achieves the same implementation principle and technical effect as the above-mentioned related method embodiments by using the above-mentioned modules. For details, please refer to the description of the above-mentioned related method embodiments, which will not be repeated here.

[0162] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0163] Figure 7A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0164] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0165] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0166] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the methods of this disclosure. For example, in some embodiments, the methods of this disclosure may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the methods of this disclosure described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the methods of this disclosure by any other suitable means (e.g., by means of firmware).

[0167] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0168] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0169] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0170] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0171] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0172] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0173] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0174] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for binding facial expressions to a digital human, comprising: Based on each frame of the captured facial expression sequence, multiple deformation control units are configured to control the digital human's facial model to execute the facial expressions in each frame; Configure multiple controllers for the digital human's facial model; Based on the pre-configured binding relationship between multiple controllers and multiple deformation control units, the multiple controllers and the multiple deformation control units are bound together, so that each controller can control the bound deformation control unit, thereby controlling the digital human's facial model to express each frame of the expression sequence; Based on the captured facial expression sequences, multiple deformation control units are configured to control the digital human's facial model to execute the facial expressions in each frame, including: For each frame of the expression sequence, if the facial features of the expression include eye features, based on a first preset rule, a second deformation control unit for the eyelid in the digital human's facial model is configured, such that the second deformation control unit for the eyelid can control the deformation of points on the two outer ring feature curves of the eyelid to be consistent with the deformation of the nearest neighbor point on the feature curve where the eyelid is located, and the deformation of points on the remaining feature curves decreases towards the center, and the points on the remaining feature curves expand outwards from the eyeball according to a preset expansion rule.

2. The method according to claim 1, wherein, Based on the captured facial expression sequences, multiple deformation control units are configured to control the digital human's facial model to execute the facial expressions in each frame, and the system further includes: For each frame of the captured expression sequence, each facial feature in the expression is divided into at least two regions; Configure the first deformation control unit for each region of each facial feature corresponding to each frame of the expression sequence to obtain a plurality of first deformation control units.

3. The method according to claim 2, wherein, For each frame of the captured expression sequence, each facial feature in the expression is divided into at least two regions, including: For each frame of the expression sequence, if the facial features in the expression include mouth features, and the movement of the mouth in the expression is an active movement, the mouth features in the expression are divided into four regions: upper, lower, left, and right. For each frame of the expression sequence, if the facial features in the expression include mouth features, and the movement of the mouth in the expression is a passive movement, the mouth features in the expression are split into two regions, left and right.

4. The method according to claim 2, wherein, For each frame of the captured expression sequence, each facial feature in the expression is divided into at least two regions, including: For each frame of the expression sequence, if the facial features in the expression include eye features, ear features, nose features, or eyebrow features, the corresponding eye features, ear features, nose features, or eyebrow features in the expression are split into left and right regions.

5. The method according to claim 2, wherein, Before dividing each facial feature in the captured expression sequence into at least two regions, the method further includes: The facial expression model containing the expression is registered with the digital human's facial model.

6. The method according to claim 2, wherein, A first deformation control unit is configured for each region of each facial feature corresponding to each frame of the expression sequence, resulting in a plurality of first deformation control units, including: Obtain point information of each region of each facial feature corresponding to each expression in each frame of the expression sequence; Based on the point information of each region of each facial feature corresponding to each frame of the expression sequence, a first deformation control unit is configured for each region of each facial feature corresponding to each frame of the expression sequence, thereby obtaining a plurality of first deformation control units.

7. The method according to claim 6, wherein, Obtaining point information of each region of each facial feature corresponding to each frame of the expression sequence, including: If the facial features corresponding to the expression in each frame of the expression sequence include nose features, eyebrow features, or mouth features, the identifier, coordinates, and weight information of each point in the facial features corresponding to the expression are obtained; the weight information of the points is used to smooth the deformation. If the facial features corresponding to each frame of the expression sequence include eye features or ear features, obtain the identifiers and coordinates of points in each region of the facial features corresponding to the expression.

8. The method according to claim 6, wherein, The expression sequence includes at least one intermediate frame expression and one final frame expression with a sequential temporal relationship; the method further includes: If the last frame expression is adjusted, obtain the difference adjustment information between the nearest neighbor intermediate frame expression and the last frame expression relative to the last frame expression; Based on the difference adjustment information of the last frame expression, and in accordance with the preset adjustment constraints, the difference adjustment information of each intermediate frame expression relative to the previous nearest neighbor expression is obtained sequentially from the end to the beginning of the expression sequence; the preset adjustment constraints require that the difference adjustment magnitude of each expression in the expression sequence decreases from the end to the beginning. Based on the difference adjustment information corresponding to the last frame expression and the difference adjustment information corresponding to each of the intermediate frame expressions, the first deformation control unit corresponding to the last frame expression and the first deformation control unit corresponding to each of the intermediate frame expressions are updated respectively.

9. The method according to claim 1, wherein, Based on the captured facial expression sequences, multiple deformation control units are configured to control the digital human's facial model to execute the facial expressions in each frame, and the system further includes: For each frame of the expression sequence, if the facial features of the expression include eye features, a third deformation control unit of the lacrimal gland in the digital human's facial model is configured based on a second preset rule to control the lacrimal gland to follow the expression.

10. The method according to claim 9, wherein, For each frame of the expression sequence, if the facial features of the expression include eye features, based on a second preset rule, a third deformation control unit for the lacrimal gland in the digital human's facial model is configured to control the lacrimal gland to follow the expression, including: For each frame of the expression sequence, if the facial features of the expression include eye features, based on the second preset rule, the third deformation control unit of the lacrimal gland in the digital human's facial model is configured so that the third deformation control unit of the lacrimal gland can control the deformation of the points on the two outer ring feature curves of the lacrimal gland to be consistent with the deformation of the nearest neighbor points on the feature curve where the eyelid is located. The deformation of the points on the inner ring feature curve and the points on the corner of the eye feature curve decreases towards the center. The points on the corner of the eye feature curve and the points on the inner ring feature curve expand outward of the eyeball according to the preset expansion rule.

11. The method according to claim 1, wherein, Before configuring multiple controllers for the digital human's facial model, the method further includes: For a pre-configured combination of at least two expressions with the same facial feature, a deformation control unit for the combination expression and a deformation control unit for a difference expression are configured based on the at least two expressions, the original deformation control unit corresponding to each of the at least two expressions, and the combination expression. The difference expression is used to identify the difference between the superposition of the at least two expressions and the combination expression.

12. The method according to claim 11, wherein, For a pre-configured combination of at least two expressions with the same facial feature, based on the at least two expressions, the original deformation control units corresponding to each of the at least two expressions, and the combination expression, a fourth deformation control unit for the combination expression and a fifth deformation control unit for the difference expressions are configured, including: For a pre-configured combination of at least two expressions with the same facial feature, a difference expression is configured based on the at least two expressions, the original deformation control unit corresponding to each of the at least two expressions, and the combination expression; A fifth deformation control unit is configured for the aforementioned differential facial expressions; The fourth deformation control unit of the combined expression is configured based on the original deformation control unit corresponding to each of the at least two expressions and the fifth deformation control unit of the difference expression.

13. The method according to any one of claims 1-12, wherein, Based on the pre-configured binding relationships between multiple controllers and multiple deformation control units, the multiple controllers and the multiple deformation control units are bound together, enabling each controller to control the bound deformation control unit, thereby controlling the digital human's facial model to express each frame of the expression sequence, including: Based on the pre-configured binding relationship between multiple controllers and multiple configured deformation control units, at least one deformation control unit is bound to each controller in each control direction, so that each controller can control the bound deformation control unit, thereby controlling the digital human's facial model to express each frame of the expression sequence.

14. A digital human facial expression binding device, comprising: The deformation control unit configuration module is used to configure multiple deformation control units for controlling the digital human's facial model to execute the expressions in each frame based on the acquired expression sequence. The controller configuration module is used to configure multiple controllers for the digital human's facial model; A binding module is used to bind the multiple controllers and the multiple deformation control units based on a pre-configured binding relationship between multiple controllers and multiple deformation control units, so that each controller can control the bound deformation control unit, thereby controlling the digital human's facial model to express each frame of the expression sequence; The deformation control unit configuration module is used for: For each frame of the expression sequence, if the facial features of the expression include eye features, based on a first preset rule, a second deformation control unit for the eyelid in the digital human's facial model is configured, such that the second deformation control unit for the eyelid can control the deformation of points on the two outer ring feature curves of the eyelid to be consistent with the deformation of the nearest neighbor point on the feature curve where the eyelid is located, and the deformation of points on the remaining feature curves decreases towards the center, and the points on the remaining feature curves expand outwards from the eyeball according to a preset expansion rule.

15. The apparatus according to claim 14, wherein, The deformation control unit configuration module is further used for: For each frame of the captured expression sequence, each facial feature in the expression is divided into at least two regions; Configure the first deformation control unit for each region of each facial feature corresponding to each frame of the expression sequence to obtain a plurality of first deformation control units.

16. The apparatus according to claim 15, wherein, The deformation control unit configuration module is used for: For each frame of the expression sequence, if the facial features in the expression include mouth features, and the movement of the mouth in the expression is an active movement, the mouth features in the expression are divided into four regions: upper, lower, left, and right. For each frame of the expression sequence, if the facial features in the expression include mouth features, and the movement of the mouth in the expression is a passive movement, the mouth features in the expression are split into two regions, left and right.

17. The apparatus according to claim 15, wherein, The deformation control unit configuration module is used for: For each frame of the expression sequence, if the facial features in the expression include eye features, ear features, nose features, or eyebrow features, the corresponding eye features, ear features, nose features, or eyebrow features in the expression are split into left and right regions.

18. The apparatus according to claim 15, wherein, The device further includes: The registration module is used to register the expression model containing the expression with the digital human's facial model.

19. The apparatus according to claim 15, wherein, The deformation control unit configuration module is used for: Obtain point information of each region of each facial feature corresponding to each expression in each frame of the expression sequence; Based on the point information of each region of each facial feature corresponding to each frame of the expression sequence, a first deformation control unit is configured for each region of each facial feature corresponding to each frame of the expression sequence, thereby obtaining a plurality of first deformation control units.

20. The apparatus according to claim 19, wherein, The deformation control unit configuration module is used for: If the facial features corresponding to the expression in each frame of the expression sequence include nose features, eyebrow features, or mouth features, the identifier, coordinates, and weight information of each point in the facial features corresponding to the expression are obtained; the weight information of the points is used to smooth the deformation. If the facial features corresponding to each frame of the expression sequence include eye features or ear features, obtain the identifiers and coordinates of points in each region of the facial features corresponding to the expression.

21. The apparatus according to claim 19, wherein, The expression sequence includes at least one intermediate frame expression and one final frame expression with a sequential relationship; the device further includes: The update module is used for: If the last frame expression is adjusted, obtain the difference adjustment information between the nearest neighbor intermediate frame expression and the last frame expression relative to the last frame expression; Based on the difference adjustment information of the last frame expression, and in accordance with the preset adjustment constraints, the difference adjustment information of each intermediate frame expression relative to the previous nearest neighbor expression is obtained sequentially from the end to the beginning of the expression sequence; the preset adjustment constraints require that the difference adjustment magnitude of each expression in the expression sequence decreases from the end to the beginning. Based on the difference adjustment information corresponding to the last frame expression and the difference adjustment information corresponding to each of the intermediate frame expressions, the first deformation control unit corresponding to the last frame expression and the first deformation control unit corresponding to each of the intermediate frame expressions are updated respectively.

22. The apparatus according to claim 14, wherein, The deformation control unit configuration module is further used for: For each frame of the expression sequence, if the facial features of the expression include eye features, a third deformation control unit of the lacrimal gland in the digital human's facial model is configured based on a second preset rule to control the lacrimal gland to follow the expression.

23. The apparatus according to claim 22, wherein, The deformation control unit configuration module is used for: For each frame of the expression sequence, if the facial features of the expression include eye features, based on the second preset rule, the third deformation control unit of the lacrimal gland in the digital human's facial model is configured so that the third deformation control unit of the lacrimal gland can control the deformation of the points on the two outer ring feature curves of the lacrimal gland to be consistent with the deformation of the nearest neighbor points on the feature curve where the eyelid is located. The deformation of the points on the inner ring feature curve and the points on the corner of the eye feature curve decreases towards the center. The points on the corner of the eye feature curve and the points on the inner ring feature curve expand outward of the eyeball according to the preset expansion rule.

24. The apparatus according to claim 14, wherein, The deformation control unit configuration module is further used for: For a pre-configured combination of at least two expressions with the same facial feature, a deformation control unit for the combination expression and a deformation control unit for a difference expression are configured based on the at least two expressions, the original deformation control unit corresponding to each of the at least two expressions, and the combination expression. The difference expression is used to identify the difference between the superposition of the at least two expressions and the combination expression.

25. The apparatus according to claim 24, wherein, The deformation control unit configuration module is further used for: For a pre-configured combination of at least two expressions with the same facial feature, a difference expression is configured based on the at least two expressions, the original deformation control unit corresponding to each of the at least two expressions, and the combination expression; A fifth deformation control unit is configured for the aforementioned differential facial expressions; The fourth deformation control unit of the combined expression is configured based on the original deformation control unit corresponding to each of the at least two expressions and the fifth deformation control unit of the difference expression.

26. The apparatus according to any one of claims 14-25, wherein, The binding module is used for: Based on the pre-configured binding relationship between multiple controllers and multiple configured deformation control units, at least one deformation control unit is bound to each controller in each control direction, so that each controller can control the bound deformation control unit, thereby controlling the digital human's facial model to express each frame of the expression sequence.

27. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-13.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-13.

29. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-13.

Images