Facial expression restoration method, device and storage medium

By positioning feature points and creating bones on the facial mesh, generating maps and masking images, and matching maps with the bones with map action parameters, realizing realistic restoration and flexibility of facial expressions, solving the low accuracy and limitation of expression restoration in the existing technology Sexual problems save resources and time.

CN114332319BActive Publication Date: 2024-12-03ZHEJIANG VERSATILE MEDIA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210007887.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-06
Publication Date
2024-12-03
Estimated Expiration
2042-01-06

AI Technical Summary

Technical Problem

In the existing technology, in facial expression restoration, the map accuracy is low, and facial details cannot be effectively restored. There are limitations in facial restoration. Repeated shooting is required to generate different expressions, which is time-consuming and takes up a lot of disk space.

Method used

By positioning feature points at key parts of the facial mesh, creating bones and mapping distance data between feature points, generating color, normal maps and area masking maps, using map action parameters to match bones, driving expression texture mixing in real time, real-time to achieve facial Realistic restoration of expressions.

Benefits of technology

It realizes more realistic facial expression restoration, which can restore other expressions outside the shooting, without loading each frame of map, saves disk space and loading time, and reduces human resource consumption.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114332319B_ABST
    Figure CN114332319B_ABST
Patent Text Reader

Abstract

The present invention provides a facial expression restoration method, device and storage medium, which relates to the technical field of computers. By transmitting the data information of facial feature points to the corresponding mapping action parameters through bones, the mapping action parameters control the opening and closing of each channel in the regional mask map according to the change of the bone value when running the facial animation, triggering the corresponding color and normal display, thereby restoring the real person's facial expression. Compared with the prior art, the present invention has the advantages of more realistic expression restoration and the ability to restore other expressions without additional shooting. When running the facial expression animation, it is not necessary to load the facial color and normal map of each frame. Only by providing the color and normal map of four specified expressions, the mixing of the expression textures of the corresponding area can be driven in real time. It not only has flexible inclusiveness for animation data, but also saves disk space to a great extent, shortens the time for the engine to load cache assets, and saves manpower and resources.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of computers, and in particular to a facial expression restoration method, device and storage medium. Background Art

[0002] Facial restoration is a technology that restores facial details including real-life facial texture and expressions in a three-dimensional model. In the prior art, a real-life facial image is generally captured first, and then a 3D mesh and a color map corresponding to the 3D mesh are generated based on the acquired real-life facial image. Then, a normal generator is used to directly generate a normal map based on the color map. The color map is used to give the 3D mesh color and material, while the normal map is a map that records concave and convex information. After these maps are post-processed such as color calibration, they are made into material balls and assigned to the 3D mesh to achieve facial texture restoration.

[0003] By photographing the face of a real performer making a single static expression or a series of dynamic expressions, a mesh model with expression deformation animation that restores the face shape of the real performer is processed and generated. When the deformation animation of the model is played, the material map assigned to the mesh body is played synchronously frame by frame in sequence to achieve the restoration of the real facial expression in the three-dimensional model.

[0004] In the prior art, the accuracy of the map generated by the image is low, and it can only restore some small details of the pores and skin texture. Details on structures such as eye sockets and noses will be lost, and the texture details will deviate slightly from the details of real people's faces. On the other hand, the above method has great limitations in restoring expressions. It can only restore the captured expressions. If you want to restore other expressions, you need to shoot and generate them again, which will take up a lot of time and manpower. Furthermore, the length of a piece of shooting data is usually not too short, generally requiring hundreds of frames or more, and each frame will generate a corresponding asset file. The generated asset file will occupy a large amount of disk memory space to a certain extent. Summary of the invention

[0005] The purpose of the present invention is to solve the problems mentioned in the background technology and to provide a facial expression restoration method, device and storage medium.

[0006] To achieve the above object, the present invention first proposes a facial expression restoration method, comprising the following steps:

[0007] Two feature points are located at several key positions of the facial mesh respectively; the distance between the two feature points is set to vary within a numerical range of 0 to 1; a skeleton is created and the distance data between the two feature points is mapped to the corresponding skeleton; color and normal maps are generated in four states: natural expressionless, closed eyes and raised eyebrows with pouting, grinning with teeth showing, and lowering eyebrows with eyes closed and grinning; a regional mask map is generated in three states: closed eyes and raised eyebrows with pouting, grinning with teeth showing, and lowering eyebrows with eyes closed and grinning, wherein the regional mask map is composed of multiple channels, and each channel distinguishes different facial regions; a color map, a normal map, and a regional mask map are assigned to a facial material ball; by matching the map action parameters with the skeleton, the map action parameters control the opening and closing of each channel in the regional mask map according to the skeleton value, thereby triggering the corresponding color and normal display; according to the change of the distance data when running the facial animation, the mixing of the expression textures of the corresponding region is driven in real time, thereby restoring the real facial expression.

[0008] Optionally, setting the distance between two feature points to vary within a numerical range of 0 to 1 includes the following steps: when the amplitude of the key part under the key expression reaches a maximum, obtaining the distance between the two feature points and setting it to a maximum value, and outputting it as 1; when the amplitude of the key part under the key expression reaches a minimum, obtaining the distance between the two feature points and setting it to a minimum value, and outputting it as 0.

[0009] Optionally, matching the mapping action parameters with the skeleton comprises the following steps: grouping the mapping action parameters according to key expressions; and transferring the skeleton values ​​of the key parts to all the mapping action parameters in the key expression group corresponding to the key parts.

[0010] Optionally, the key expressions include blinking, lowering eyebrows, raising eyebrows, wrinkling nose, pouting, and smiling.

[0011] Optionally, the region mask image consists of red, green, blue and alpha channels.

[0012] The present invention also proposes a facial expression restoration device, comprising: a positioning module, configured to respectively locate two feature points at several key parts of a facial grid; a distance module, configured to set the distance between the two feature points to vary within a numerical range of 0 to 1; a skeleton module, configured to create skeletons and map the distance data between the two feature points to the corresponding skeletons; a mapping module, configured to generate color and normal maps in four states: natural expressionless, closed eyes and raised eyebrows with pouting, grinning with teeth showing, and lowering eyebrows with forced closed eyes and grinning; an area mask module, configured to generate color and normal maps in four states: closed eyes and raised eyebrows with pouting, grinning with teeth showing, and lowering eyebrows with forced closed eyes and grinning; and an area mask module, configured to generate color and normal maps in four states: closed eyes and raised eyebrows with pouting, grinning with teeth showing, and lowering eyebrows with forced closed eyes and grinning. A regional mask map in three states of eyes closed and mouth opened, wherein the regional mask map is composed of multiple channels, each channel distinguishing different facial areas; a material ball module, configured to assign a color map, a normal map, and a regional mask map to a facial material ball; a matching module, configured to control the opening and closing of each channel in the regional mask map according to the bone value by matching the map action parameters with the bones, thereby triggering the corresponding color and normal display; a driving module, configured to drive the mixing of the expression textures of the corresponding area in real time according to the change of the distance data when running the facial animation, thereby restoring the facial expression of a real person.

[0013] Optionally, the distance module also includes: a maximum distance module, which is configured to obtain the distance between two feature points and set it to the maximum value when the amplitude of the key part under the key expression reaches the maximum, and the output is 1; a minimum distance module, which is configured to obtain the distance between two feature points and set it to the minimum value when the amplitude of the key part under the key expression reaches the minimum, and the output is 0.

[0014] Optionally, the matching module further includes: a grouping module configured to group the mapping action parameters according to key expressions; and a transfer module configured to transfer the bone values ​​of key parts to all mapping action parameters in the key expression group corresponding to the key parts.

[0015] The present invention also proposes a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned facial expression restoration is realized.

[0016] Beneficial effects of the present invention:

[0017] A facial expression restoration method, device and storage medium of an embodiment of the present invention transmits the data information of facial feature points to the corresponding mapping action parameters through the skeleton, so that the mapping action parameters control the opening and closing of each channel in the regional mask map according to the change of the skeleton value when running the facial animation, triggering the corresponding color and normal display, thereby restoring the real person's facial expression. Compared with the prior art, the embodiment of the present invention has the advantages of more realistic expression restoration and the ability to restore other expressions outside of shooting. When running facial expression animation, there is no need to load the facial color and normal map of each frame. Only by providing the color and normal map of four specified expressions, the mixing of the expression texture of the corresponding area can be driven in real time. It not only has flexible inclusiveness for animation data, but also saves disk space to a great extent, shortens the time for the engine to load cache assets, and saves manpower and resources.

[0018] In addition, the normal map in a facial expression restoration method, device and storage medium of an embodiment of the present invention is baked from a high-poly model after detail processing. Compared with the original method of directly generating it with a color map, the normal map baked and generated by a facial expression restoration method of an embodiment of the present invention retains a large number of details, making the restoration effect more realistic.

[0019] The features and advantages of the present invention will be described in detail through embodiments in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is one of the flow charts of a facial expression restoration method according to an embodiment of the present invention;

[0021] Figure 2 A schematic diagram of connecting characteristic points of a facial expression restoration method according to an embodiment of the present invention;

[0022] Figure 3 A schematic diagram of a specified expression of a facial expression restoration method according to an embodiment of the present invention;

[0023] Figure 4 The second flowchart of a facial expression restoration method according to an embodiment of the present invention;

[0024] Figure 5 This is one of the structural block diagrams of a facial expression restoration device according to an embodiment of the present invention;

[0025] Figure 6 This is a second structural block diagram of a facial expression restoration device according to an embodiment of the present invention;

[0026] Figure 7 This is a third structural block diagram of a facial expression restoration device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to facilitate understanding by those skilled in the art, the present invention will be further described in detail below in conjunction with specific embodiments.

[0028] According to the prior art, only the color and normal map generated by the performer can be obtained, and the expression obtained is the same as the expression captured, which is limited. The innovation of a facial expression restoration method in an embodiment of the present invention is that it can display the corresponding facial texture when making any expression in real time, thereby realizing the restoration of real facial expressions.

[0029] Figure 1 The following is a schematic flow chart of a facial expression restoration method according to an embodiment of the present invention.

[0030] like Figure 1 As shown, the facial expression restoration method includes steps S10 to S80:

[0031] Step S10, locating two feature points at several key parts of the facial grid respectively;

[0032] Step S20, setting the distance between the two feature points to vary within a value range of 0 to 1;

[0033] Step S30, creating a skeleton and mapping the distance data between two feature points to the corresponding skeleton;

[0034] Step S40, generating color and normal maps in four states: natural expressionless state, closed eyes and raised eyebrows with pouting lips, grinning with teeth showing, and lowered eyebrows with closed eyes and grinning lips;

[0035] Step S50, generating a regional mask map in three states: eyes closed, eyebrows raised, pouting, teeth showing, and lowered eyebrows, eyes closed, and grinning, wherein the regional mask map is composed of multiple channels, and each channel distinguishes a different facial area;

[0036] Step S60, assigning the color map, normal map, and region mask map to the facial material ball;

[0037] Step S70, by matching the mapping action parameters with the bones, the mapping action parameters are used to control the opening and closing of each channel in the region mask map according to the bone values, thereby triggering the corresponding color and normal display;

[0038] Step S80, according to the change of distance data when running facial animation, the mixing of expression textures in the corresponding area is driven in real time to restore the real person's facial expression.

[0039] A facial expression restoration method according to an embodiment of the present invention transmits data information of facial feature points to corresponding mapping action parameters through bones, so that the mapping action parameters control the opening and closing of each channel in the regional mask map according to the change of the bone value when running the facial animation, triggering the corresponding color and normal display, thereby restoring the real person's facial expression. Compared with the prior art, the embodiment of the present invention has the advantages of more realistic expression restoration and the ability to restore other expressions without additional shooting. When running facial expression animation, there is no need to load the facial color and normal map of each frame. Only by providing the color and normal map of four specified expressions, the mixing of the expression textures of the corresponding area can be driven in real time. It not only has flexible inclusiveness for animation data, but also saves disk space to a great extent, shortens the time for the engine to load cache assets, and saves manpower and resources.

[0040] Below, each step of the facial expression restoration method in the embodiment of the present invention will be described in more detail with reference to the accompanying drawings and embodiments.

[0041] Step S10, locating two feature points at several key parts of the facial grid respectively.

[0042] In one embodiment, a real-life performer making a series of different extreme expressions (such as blinking, frowning, wrinkling the nose, pulling the corners of the mouth, pouting, smiling, etc.) is first filmed using a 3D / 4D scanning device, and a facial mesh model with expression-changing animation that highly restores the facial shape of the real-life performer is processed and generated.

[0043] Then, two feature points are located at key parts of the facial mesh, where key parts refer to the parts used when a real person makes key expressions, such as blinking, smiling, opening the chin, lowering the eyebrows, raising the eyebrows, wrinkling the nose, and pouting.

[0044] In the subsequent steps, the distance between two points needs to be measured, so the positions of the two points need to be determined separately according to each facial expression. For example, when blinking or opening the mouth, the relative distance between the upper and lower eyelids or the upper and lower lips needs to be measured. For expressions such as raising eyebrows and smiling, the eyebrows and the corners of the mouth are displaced relative to the entire face. Therefore, it is necessary to ensure that one of the two feature points is at the position with the most obvious movement, and the other point is fixed at a place where there is basically no displacement change relative to the entire face.

[0045] Please refer to Figure 2 When locating the key points of the eyebrow lowering expression, one of the feature points is set at the eyebrow and the other is set at the middle of the nose bridge, because the middle point of the nose bridge is a position that basically does not move relative to the face.

[0046] Step S20: setting the distance between two feature points to vary within a value range of 0 to 1.

[0047] Specifically, two feature points in the same group may be connected by a straight line to obtain the length of the straight line, that is, the distance between the two points.

[0048] In one embodiment, the specific method of setting the distance between two feature points to vary within the range of 0 to 1 is: adsorb the two end points of the straight line onto a pair of feature points respectively, and play the generated facial animation clip with key expressions. Because the feature points are located at a certain position on the surface of the model, the feature points will move with the surface when making expressions, which will drive the two ends of the straight line to change accordingly, so the distance between the feature points will change with the facial movements, and the length of the straight line will also change accordingly. When the amplitude of the key expression of the key part reaches the maximum, the obtained straight line length is set to the input maximum value, and the output is 1; conversely, when the amplitude of the key expression of the key part reaches the minimum, the obtained straight line length is set to the input minimum value, and the output is 0, thereby realizing the control of the data of the straight line to vary between 0 and 1, so as to unify the units when the data is transmitted in subsequent operations.

[0049] For example, when the chin is opened, the upper and lower lips are completely closed or in a naturally closed state, the obtained straight line length value is set to the minimum input value, and the output is 0; when the upper and lower distances of the mouth are opened to the maximum, the obtained straight line length is set as the maximum input value, and the output is 1.

[0050] Step S30, creating a skeleton and mapping the distance data between two feature points to the corresponding skeleton.

[0051] In one embodiment, a skeleton is created that matches the number of lines, and one skeleton transmits data information of one line. In addition, the skeleton can be named according to the key expression name, and the output value of the line mapped in the range of 0 to 1 is transmitted to the value of the corresponding named skeleton. The skeleton is used as a medium for transmitting data, and the change of the corresponding data can be observed more conveniently through the skeleton.

[0052] After the above three steps, the distance data between two feature points is mapped to the skeleton and kept in the range of 0 to 1. The bone value represents the information of the feature point.

[0053] Step S40, generating color and normal maps in four states: natural expressionless state, closed eyes and raised eyebrows with pouting lips, grinning with teeth showing, and lowered eyebrows with closed eyes and grinning lips.

[0054] In one embodiment, a live performer making four designated expressions is photographed by a 3D / 4D scanning device.

[0055] Please refer to Figure 3 The four designated expressions are closed eyes, raised eyebrows and pouting lips, natural expressionless face, toothy smile, and lowered eyebrows with eyes closed and mouth grinning.

[0056] The above four specified expressions contain all the textures that will appear on the face of a real person when making an expression, so the facial details of the real person can be basically accurately restored when driving expressions. The natural expressionless map will be used as the default map, that is, the map displayed when the facial mesh does not make any expression. The usage of the other three maps will be mentioned in the subsequent steps.

[0057] In one embodiment, the captured image is used with Reality Capture, a software for generating 3D meshes, to generate a mesh and a color map. The generated mesh is processed by sculpting and shaping to restore the skin texture and wrinkles of a real face as much as possible, and finally a normal map is baked. The normal map is baked after the high-poly model is processed with details. Compared with the original method of directly generating the color map, in this embodiment, the baked normal map retains a large number of details, making the restoration effect more realistic. The color map needs to be calibrated with skin color and then the more obvious blemishes and hair on the skin are removed. The parts outside the face are all repaired to smooth skin and naturally connected with the face. At the same time, the realism after the map is assigned to the model must be guaranteed.

[0058] Step S50, generating a regional mask map in three states: eyes closed, eyebrows raised and pouting, teeth showing, and lowered eyebrows with eyes closed and mouth grinning, wherein the regional mask map is composed of multiple channels, and each channel distinguishes different facial regions.

[0059] Specifically, the region mask refers to a picture composed of red, green, blue and alpha channels, containing red, green, blue and black. Different colors distinguish different regions. The region mask will be superimposed on the color map, normal map and other facial material maps, covering part of the map content and showing the expression texture of a specific area of ​​the specified map.

[0060] For ease of understanding, the functions of the mask images of a portion of the area in three states of closed eyes, raised eyebrows and pouting, toothy smile, and lowered eyebrows and forced closed eyes and grinning are specifically described. For ease of description, the closed eyes, raised eyebrows and pouting generated in step S40 is called map No. 1, the lowered eyebrows and forced closed eyes and grinning is called map No. 2, and the toothy smile is called map No. 3.

[0061] Every three or four area masks are applied to one expression map. The first area mask is a common mask for the lips of maps 1 and 3, which can show the texture of pouting and stretching lips.

[0062] The second to fifth area mask maps are all used for map No. 1. The second area mask map can display the texture of the upper eyelid and the inner side of the raised eyebrow (near the middle of the eyebrows) when the eyes are closed. The third area mask map can display the texture of the outer side of the raised eyebrow and the chin. The fourth area mask map can display the texture of the lower jaw when the mouth is open, and the texture of the lower part of the lips when pouting. The fifth area mask map can display the texture of the upper part of the lips when pouting, and the texture of the lower eyelid when the eyes are closed.

[0063] The sixth to eighth area mask maps are used for map No. 2. The sixth area mask map can show the texture of the inner corners of the eyes and eyebrows when the eyes are closed tightly, the seventh area mask map can show the texture of the corners of the mouth and neck when the mouth is grinning, and the eighth area mask map is the texture of the nose root to the nose wing when the nose is wrinkled.

[0064] The ninth and tenth area mask images are masks of map No. 3. The ninth area mask image can display the inner and outer textures under the eyes when smiling, and the tenth area mask image can display the textures of the outer corners of the eyes and the nasolabial folds of the cheeks when smiling.

[0065] In addition, the method and principle of generating regional masks are as follows: on the layer with a black background color, corresponding to the plane space position of the color map, open the red channel, green channel, blue channel and alpha channel separately, and draw white areas respectively. This area is the area where the texture changes when the face makes expressions. The layer under the mask can be displayed through the white area. On the contrary, the black area will cover the layer below and not display it. When the red, green and blue channels are all turned on, an image with four colors of red, green, blue and black is presented. This is the final regional mask map. The white areas previously drawn in each channel will be replaced by the colors in the corresponding channels, but the effect remains unchanged.

[0066] By generating a regional mask map that divides the facial regions, the subsequent control of the display of the specified region texture is realized. In order to maximize resource utilization, the four mask maps of different regions are stored in four channels of one image. The generated mask map has different colors to display different masks visually, which also has the advantage of being easy to observe and distinguish.

[0067] Step S60, assigning the color map, normal map, and region mask map to the facial material ball.

[0068] By assigning color maps, normal maps, and area mask maps to the facial material ball, when the facial subject is performing or running animations to make different facial expressions, the facial map material drives the specified channels of the area mask map to open and close, and triggers the corresponding color and normal display in real time, and blends between these expressions, thereby restoring the expression texture of real human facial skin.

[0069] Step S70, by matching the mapping action parameters with the bones, the mapping action parameters are used to control the opening and closing of each channel in the region mask map according to the bone values, thereby triggering the corresponding color and normal display.

[0070] Specifically, you can follow Figure 4 The process shown here is to match the map action parameters with the bones, including the following steps:

[0071] Step S710, grouping the sticker action parameters according to key expressions.

[0072] In one embodiment, the mapping action parameters are named according to the expression actions of the controlled texture, and the mapping action parameters are grouped according to key expressions.

[0073] For example, the parameters affected by smile include mouth corner, lower lip pursing, upper lip pursing, upper cheek raised, inner cheek raised, and outer cheek raised. The parameters affected by chin opening include chin opening, wrinkling around upper right lip, and wrinkling around upper lip. The parameters affected by blinking include blinking. The parameters affected by eyebrow lowering include eyebrow lowering and eyebrow wrinkling. The parameters affected by eyebrow raising include inner eyebrow raised and outer eyebrow raised. The parameters affected by nose wrinkling include nose wrinkling, mouth stretching, and neck stretching. The parameters affected by pouting include chin raised, lower lip pouting, and upper lip pouting.

[0074] Step S720, passing the bone values ​​of the key parts to all the mapping action parameters in the key expression group corresponding to the key parts.

[0075] The obtained bone value is given to each parameter in the corresponding group, that is, all parameter values ​​in the same group are equal. These parameters also vary in the range of 0 to 1. The size of the value represents the opening and closing of the mask of the controlled area. When the value is between 0 and 1, it can also be understood as the parameter controlling the transparency of the mask. When the parameter is 0.5, the transparency of the mask is 50%, and the transparency of the map texture displayed from the mask area will also be 50%. When the face makes a certain expression, it will drive the corresponding bone value to change, and the map action parameters will also change accordingly, thereby driving the corresponding area to display the map of the corresponding expression.

[0076] Step S80, according to the change of distance data when running facial animation, the mixing of expression textures in the corresponding area is driven in real time to restore the real person's facial expression.

[0077] Specifically, the facial expression animation is run, and each link is driven successively. The distance between the feature points in each area of ​​the face changes with the expression, so that the arc length value of the curve connecting the two points changes, driving the corresponding bone value to change, and then the bone drives the corresponding mapping action parameter value, and the mapping action parameter value controls the mask of the corresponding area. Multiple masks are mixed, and finally the mixed mapping texture is displayed, restoring the real face texture details.

[0078] In one embodiment, the corresponding texture displayed when making an expression is driven by a dynamic map function. The function controls the mixing of maps under different facial actions through four colors: red, green, blue, and white, wherein the face mask function is used to mix different facial map weights. The face mask function controls the display of four channels of R, G, B (red, green, blue) and Alpha (transparency) channels of the regional mask map through the map action parameter value, and then superimposes and mixes multiple mask maps to obtain the final displayed map texture. The map animation parameter specifies the controlled channel (one of R, G, B and Alpha). For example, if a certain parameter controls the R channel (red channel), then it can control the red area in the mask map. Each of the four parameters controls the R, G, B and Alpha channels of a mask map respectively. When the parameter value is 0, the channel is closed; when the parameter value is 1, the color channel it controls is opened, and the rest of the area is closed, and the map of the area marked by that color will be displayed.

[0079] Based on the above-mentioned facial expression restoration method, the embodiment of the present invention also provides a facial expression restoration device, such as Figure 5 As shown, the device includes the following modules:

[0080] The positioning module 100 is configured to respectively locate two feature points at several key parts of the facial grid.

[0081] The distance module 200 is configured to set the distance between two feature points to vary within a value range of 0 to 1.

[0082] The skeleton module 300 is configured to create a skeleton and map the distance data between two feature points to the corresponding skeleton.

[0083] The mapping module 400 is configured to generate color and normal maps in four states: natural expressionless state, closed eyes and raised eyebrows with pouting lips, grinning with teeth showing, and lowered eyebrows with closed eyes and grinning lips.

[0084] The area mask module 500 is configured to generate area mask images in three states: closed eyes, raised eyebrows and pouting, grinning with teeth showing, and lowered eyebrows and closed eyes and grinning. The area mask image is composed of multiple channels, and each channel distinguishes different facial areas.

[0085] The material ball module 600 is configured to assign a color map, a normal map, and a region mask map to the face material ball.

[0086] The matching module 700 is configured to match the mapping action parameters with the bones so that the mapping action parameters control the opening and closing of each channel in the region mask map according to the bone values, thereby triggering the corresponding color and normal display.

[0087] The driving module 800 is configured to drive the mixing of expression textures in the corresponding area in real time according to the change of distance data when running facial animation, so as to restore the facial expression of a real person.

[0088] like Figure 6 As shown, in one embodiment, the distance module further includes:

[0089] The maximum distance module 2100 is configured to obtain the distance between two feature points and set it as the maximum value when the amplitude of the key part under the key expression reaches the maximum, and the output is 1.

[0090] The minimum distance module 2200 is configured to obtain the distance between two feature points and set it as the minimum value when the amplitude of the key part under the key expression reaches the minimum, and the output is 0.

[0091] like Figure 7 As shown, in one embodiment, the matching module further includes:

[0092] The grouping module 7100 is configured to group the sticker action parameters according to key expressions.

[0093] The transfer module 7200 is configured to transfer the bone values ​​of the key parts to all the mapping action parameters in the key expression group corresponding to the key parts.

[0094] In summary, a facial expression restoration device according to an embodiment of the present invention can be implemented in the form of a program and run on a computer device. The memory of the computer device can store various program modules constituting the facial expression restoration device, such as: Figure 5 The positioning module 100, distance module 200, skeleton module 300, mapping module 400, area mask module 500, material ball module 600, matching module 700, and driving module 800 are shown. The program composed of each program module enables the processor to execute the steps of a facial expression restoration method of each embodiment of the present application described in this specification.

[0095] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in a facial expression restoration method in each embodiment of the present application are implemented.

[0096] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0097] The above embodiments are explanations of the present invention, not limitations of the present invention. Any solution that is simply modified from the present invention belongs to the protection scope of the present invention. The above is only a preferred embodiment of the present invention. The protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A facial expression restoration method, It is characterized in that The following steps are involved: Locate two feature points at several key locations of the facial mesh; Set the distance between two feature points to vary within the value range of 0 to 1; Create bones and map the distance data between two feature points to the corresponding bones; Generate color and normal maps in four states: natural expressionless, closed eyes and raised eyebrows with pouting, grinning with teeth showing, and lowered eyebrows with closed eyes and grinning; Generate regional mask images in three states: closed eyes, raised eyebrows, pouting, grinning, and lowered eyebrows, closed eyes, and grinning, wherein the regional mask images are composed of multiple channels, and each channel distinguishes different facial regions; Assign color map, normal map, and area mask map to the face material ball; By matching the texture action parameters with the bones, the texture action parameters control the opening and closing of each channel in the area mask map according to the bone values, thereby triggering the corresponding color and normal display; According to the changes in distance data when running facial animation, the mixing of expression textures in the corresponding area is driven in real time to restore real facial expressions.

2. The facial expression restoration method according to claim 1, It is characterized in that Setting the distance between two feature points to vary within a value range of 0 to 1 includes the following steps: When the amplitude of the key part under the key expression reaches the maximum, the distance between the two feature points is obtained and set to the maximum value, and the output is 1; When the amplitude of the key part under the key expression reaches the minimum, the distance between the two feature points is obtained and set to the minimum value, and the output is 0.

3. The facial expression restoration method according to claim 1, It is characterized in that Matching the mapping action parameters to the bones involves the following steps: Group sticker action parameters according to key expressions; The bone values ​​of the key parts are passed to all the map action parameters in the key expression group corresponding to the key parts.

4. The facial expression restoration method according to claim 3, It is characterized in that The key expressions include winking, lowering eyebrows, raising eyebrows, wrinkling nose, pouting lips, and smiling.

5. The facial expression restoration method according to claim 1, It is characterized in that The region mask is composed of red, green, blue and alpha channels.

6. A facial expression restoration device, It is characterized in that include: A positioning module, configured to respectively locate two feature points at a plurality of key parts of the facial mesh; A distance module is configured to set the distance between two feature points to vary within a value range of 0 to 1; A skeleton module, configured to create a skeleton and map the distance data between two feature points to the corresponding skeleton; The mapping module is configured to generate color and normal maps in four states: natural expressionless state, closed eyes and raised eyebrows with pouting, grinning with teeth showing, and lowered eyebrows with closed eyes and grinning; A region mask module is configured to generate region mask images in three states: closed eyes, raised eyebrows, pouting, grinning, and lowered eyebrows, closed eyes, and grinning, wherein the region mask images are composed of multiple channels, and each channel distinguishes different facial regions; A shader module, configured to assign a color map, a normal map, and an area mask map to a face shader; A matching module is configured to match the texture action parameters with the bones so that the texture action parameters control the opening and closing of each channel in the region mask map according to the bone values, thereby triggering the corresponding color and normal display; The driving module is configured to drive the mixing of expression textures in corresponding areas in real time according to the changes in distance data when running facial animation, so as to restore real facial expressions.

7. The facial expression restoration device according to claim 6, It is characterized in that The distance module also includes: The maximum distance module is configured to obtain the distance between two feature points and set it to the maximum value when the amplitude of the key part under the key expression reaches the maximum, and the output is 1; The minimum distance module is configured to obtain the distance between two feature points and set it to the minimum value when the amplitude of the key part under the key expression reaches the minimum, and the output is 0.

8. The facial expression restoration device according to claim 7, It is characterized in that The matching module also includes: a grouping module, configured to group sticker action parameters according to key expressions; The transfer module is configured to transfer the bone values ​​of the key parts to all the mapping action parameters in the key expression group corresponding to the key parts.

9. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the facial expression restoration method according to any one of claims 1 to 5 is implemented.