Wrinkle special effects rendering method, device, electronic device and storage medium
By detecting the brightness of the wrinkle texture image of the face feature points and deformation processing, the problem of high complexity of wrinkle rendering in the existing technology is solved, real-time rendering and more natural wrinkle effects are achieved, and user experience is improved.
Patent Information
- Application Number
- CN202011141155.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-10-22
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-10-22
AI Technical Summary
The existing wrinkle special effect rendering methods are based on physical simulation, with complex operations and difficult to achieve real-time rendering, resulting in poor user experience.
By detecting the feature points of the face image, establishing a model, and adjusting the brightness value according to the deformation processing of the expression ratio map and wrinkle texture image, generating a natural wrinkle texture image, superimposing it on the face model to simulate the wrinkle effect.
Simplifies the computing process, implements real-time rendering, improves user experience, and eliminates the weird color block effects that may occur based on physical models.
Smart Images

Figure CN112562026B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a wrinkle special effect rendering method, device, electronic device and storage medium. Background Art
[0002] With the development of short video social networking, drawing facial animations has become a common entertainment activity. In the process of drawing facial animations, the skin needs to be rendered. Skin rendering includes generating pores, wrinkles, freckles, etc. Among them, the generation of facial wrinkles is particularly important.
[0003] In the prior art, wrinkle special effects rendering methods use physical simulation methods. Based on pre-acquired biomechanical models such as skin models, muscle models, and skull models, skin deformation is simulated by simulating mass springs or finite elements, and the tension formed on the skin surface is calculated to generate wrinkles.
[0004] However, the calculations involved in this wrinkle special effects rendering method are relatively complex, require a long calculation time, are difficult to use for real-time rendering, and provide a poor user experience. Summary of the invention
[0005] To solve the above technical problems, the present application shows a wrinkle special effects rendering method, device, electronic device and storage medium.
[0006] In a first aspect, the present application provides a wrinkle special effect rendering method, the method comprising:
[0007] Detecting feature points of a target face image as target feature points, and modeling the target face image to obtain a face model to be processed;
[0008] According to the position of the target feature point in the target face image and the correspondence between the target feature point and the first feature point in the pre-acquired expression ratio map, the expression ratio map is deformed to obtain a target expression ratio map;
[0009] According to the position of the target feature point in the target face image and the correspondence between the target feature point and the second feature point in the preset wrinkle texture image, the preset wrinkle texture image is deformed to generate a wrinkle texture image to be processed;
[0010] According to the target expression ratio map, adjusting the brightness value of the wrinkle texture image to be processed to obtain a target wrinkle texture image;
[0011] The target wrinkle texture image is superimposed on the face model to be processed for rendering to obtain a face model with wrinkle effect.
[0012] In a second aspect, the present application shows a wrinkle special effect rendering device, the device comprising:
[0013] A detection module is used to detect feature points of a target face image as target feature points, and to model the target face image to obtain a face model to be processed;
[0014] A first deformation module is used to perform deformation processing on the expression ratio map according to the position of the target feature point in the target face image and the correspondence between the target feature point and the first feature point in the pre-acquired expression ratio map to obtain a target expression ratio map;
[0015] A second deformation module, configured to perform deformation processing on the preset wrinkle texture image according to the position of the target feature point in the target face image and the correspondence between the target feature point and the second feature point in the preset wrinkle texture image, so as to generate a wrinkle texture to be processed;
[0016] An adjustment module, used for adjusting the brightness value of the wrinkle texture image to be processed according to the target expression ratio map to obtain a target wrinkle texture image;
[0017] A rendering module is used to superimpose the target wrinkle texture image on the face model to be processed for rendering, so as to obtain a face model with wrinkle effect.
[0018] In a third aspect, the present application shows an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the wrinkle special effects rendering method as described in any one of the above items are implemented.
[0019] In a fourth aspect, the present application illustrates a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above-mentioned wrinkle special effects rendering methods are implemented.
[0020] Compared with the prior art, this application has the following advantages:
[0021] Detect feature points of a target facial image as target feature points, and model the target facial image to obtain a face model to be processed; deform the expression ratio map according to the position of the target feature points in the target facial image and the correspondence between the target feature points and a first feature point in a pre-acquired expression ratio map to obtain a target expression ratio map; deform the preset wrinkle texture image according to the position of the target feature points in the target facial image and the correspondence between the target feature points and a second feature point in a preset wrinkle texture image to generate a wrinkle texture image to be processed; adjust the brightness value of the wrinkle texture image to be processed according to the target expression ratio map to obtain a target wrinkle texture image; superimpose the target wrinkle texture image on the face model to be processed for rendering to obtain a face model with a wrinkle effect.
[0022] That is to say, in the present application, the pre-acquired expression ratio map and the preset wrinkle texture image are first deformed according to the correspondence between the facial feature points to match the target facial image. Then, the brightness of the deformed wrinkle texture image is adjusted according to the deformed expression ratio map to make the deformed wrinkle texture image more natural. The texture image with adjusted brightness is then rendered onto the target facial model to simulate the wrinkle effect. Compared with the method based on the physical model, the calculation is relatively simple and can be used for real-time rendering, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0024] Figure 1 It is a flowchart of the steps of a wrinkle special effect rendering method of the present application;
[0025] Figure 2 is a flowchart of another wrinkle special effect rendering method of the present application;
[0026] Figure 3 This is an application example diagram of the wrinkle special effects rendering method of the present application;
[0027] Figure 4 It is a structural block diagram of a wrinkle special effect rendering device of the present application;
[0028] Figure 5 It is a structural block diagram of an electronic device of the present application. DETAILED DESCRIPTION
[0029] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.
[0030] In the related art, wrinkle special effects rendering methods can adopt physical simulation methods, based on pre-acquired biomechanical models such as skin models, muscle models, and skull models, simulate skin deformation by simulating mass springs or finite elements, calculate the tension formed on the skin surface, and then generate wrinkles. However, the calculations involved in this wrinkle special effects rendering method are relatively complex, requiring a long calculation time, making it difficult to use for real-time rendering, and providing a poor user experience.
[0031] In order to solve the above problems, the present application proposes a wrinkle special effect rendering method. The wrinkle special effect rendering method provided by an embodiment of the present invention is generally described below. The method comprises the following steps:
[0032] Detecting feature points of a target face image as target feature points, and modeling the target face image to obtain a face model to be processed;
[0033] According to the position of the target feature point in the target face image and the correspondence between the target feature point and the first feature point in the pre-acquired expression ratio map, the expression ratio map is deformed to obtain a target expression ratio map;
[0034] According to the position of the target feature point in the target face image and the correspondence between the target feature point and the second feature point in the preset wrinkle texture image, the preset wrinkle texture image is deformed to generate a wrinkle texture image to be processed;
[0035] According to the target expression ratio map, adjusting the brightness value of the wrinkle texture image to be processed to obtain a target wrinkle texture image;
[0036] The target wrinkle texture image is superimposed on the face model to be processed for rendering to obtain a face model with wrinkle effect.
[0037] As can be seen from the above, in the present application, the pre-acquired expression ratio map and the preset wrinkle texture image are first deformed according to the correspondence between the facial feature points to match the target facial image. Then, the brightness of the deformed wrinkle texture image is adjusted according to the deformed expression ratio map to make the deformed wrinkle texture image more natural. The texture image with adjusted brightness is then rendered onto the target facial model to simulate the wrinkle effect. Compared with the method based on the physical model, the calculation is relatively simple and can be used for real-time rendering, thereby improving the user experience.
[0038] The wrinkle special effect rendering method provided by the embodiment of the present invention will be described in detail below through specific embodiments.
[0039] Reference Figure 1 , shows a flowchart of a wrinkle special effect rendering method of the present application, which may specifically include the following steps:
[0040] S101: Detect feature points of a target face image as target feature points, and model the target face image to obtain a face model to be processed.
[0041] In the present application, the target facial image can be acquired in real time through the camera of the device, or can also be acquired through a remote camera or from a video file, without specific limitation.
[0042] Among them, a pre-acquired facial feature point detection algorithm can be used to perform feature point detection on the target facial image, and the facial feature point detection algorithm can be any current algorithm. Alternatively, a feature point configuration file can be obtained from a server in real time to perform feature point detection on the target facial image. The feature point configuration file includes multiple preset facial feature points, and the feature information of any facial image can be determined based on these preset facial feature points.
[0043] The feature point configuration file can be generated by the server. According to the different services corresponding to the current device, the server can adjust the distribution and quantity of facial feature points. For example, more facial feature points can be set in areas with dense wrinkles such as the forehead, corners of the eyes, and corners of the mouth, so that the wrinkle special effects rendering effect is more realistic.
[0044] For example, in this step, 3DMM (3D morphable model, three-dimensional face deformation statistical model) technology can be used to perform three-dimensional modeling on the target face image to obtain a face model to be processed. Among them, 3DMM technology can represent different face images as three-dimensional grids with the same sequence number, so that it is easy to locate and process the obtained face model to be processed.
[0045] S102: According to the position of the target feature point in the target face image and the correspondence between the target feature point and the first feature point in the pre-acquired expression ratio map, the expression ratio map is deformed to obtain a target expression ratio map.
[0046] In the present application, the first feature point and the target feature point of the expression ratio map are obtained using the same facial feature point algorithm or feature point configuration file, that is, the facial feature points of the expression ratio map and the target facial image correspond.
[0047] According to the position of the target feature point in the target face image, and the correspondence between the target feature point and the first feature point in the pre-acquired expression ratio map, the expression ratio map can be deformed, so that the position of the first feature point in the expression ratio map is the same as the position of the target feature point in the target face image. In this step, a triangulation algorithm can be used to deform the expression ratio map.
[0048] In one implementation, the expression ratio graph can be obtained in the following manner:
[0049] First, you need to prepare a wrinkle-free photo and a wrinkle-free photo. If the facial feature points of the wrinkle-free photo and the wrinkle-free photo are different, you need to calibrate them. The wrinkle-free photo can be taken naturally or processed from the wrinkle-free photo. Then, convert the wrinkle-free photo and the wrinkle-free photo into YUV color space respectively, calculate the ratio of the brightness value of the brightness channel of the wrinkle-free photo and the wrinkle-free photo, and save the calculation result as a single-channel image to obtain ERI (expression ratio image).
[0050] The calculation formula for the above process is:
[0051] ERI=Y old / Y young
[0052] Among them, ERI represents the pixel value of each pixel in the expression ratio map, Y old Represents the brightness value of the brightness channel of the wrinkled photo, Y young Represents the brightness value of the lightness channel of the wrinkle-free photo.
[0053] In the present application, wrinkle photos can be detected, and the identified facial feature points can be used as the first feature points in the expression ratio map.
[0054] S103: According to the position of the target feature point in the target face image and the correspondence between the target feature point and the second feature point in the preset wrinkle texture image, a deformation process is performed on the preset wrinkle texture image to generate a wrinkle texture image to be processed.
[0055] In the present application, the second feature point of the preset wrinkle texture image and the target feature point are obtained using the same facial feature point algorithm or feature point configuration file, that is, the facial feature points of the preset wrinkle texture image and the target facial image correspond.
[0056] According to the position of the target feature point in the target face image and the correspondence between the target feature point and the second feature point in the preset wrinkle texture image, the preset wrinkle texture image can be deformed, so that the position of the second feature point in the preset wrinkle texture image is the same as the position of the target feature point in the target face image. For example, in this step, a triangulation algorithm can be used to deform the preset wrinkle texture image.
[0057] S104: According to the target expression ratio map, the brightness value of the wrinkle texture image to be processed is adjusted to obtain a target wrinkle texture image.
[0058] In one implementation, the wrinkle texture image to be processed is an RGB image, and the step of adjusting the brightness value of the wrinkle texture image to be processed according to the target expression ratio map may include:
[0059] First, the wrinkle texture image to be processed is converted into a YUV image; then, for each pixel in the wrinkle texture image to be processed, the value of the brightness channel of the pixel is multiplied by the value of the corresponding pixel in the target expression ratio map to obtain the value of the brightness channel of the updated YUV image; then, the updated YUV image is converted into an RGB image.
[0060] Specifically, the above process can be expressed by the following formula:
[0061] Y old′ =ERI*Y young′
[0062] Among them, Y old’ Represents the brightness value of the brightness channel of the updated YUV image, Y young’ Indicates the brightness value of the brightness channel after the wrinkle texture image is converted to a YUV image.
[0063] In a face image, different types of wrinkles can be generated in multiple regions, for example, forehead wrinkles can be generated in the forehead region, crow's feet can be generated in the eye corner region, nasolabial folds can be generated in the mouth corner region, etc. Generating wrinkles in different regions can make the obtained face model with wrinkles more realistic. In one implementation, a mask image can be used to control the generation of wrinkles in different regions.
[0064] For example, before adjusting the brightness value of the wrinkle texture image to be processed according to the target expression ratio map to obtain the target wrinkle texture image, the mask image may be deformed according to the position of the target feature point in the target face image and the correspondence between the target feature point and the third feature point in the pre-acquired mask image. The mask image includes a mask area and a blank area, the mask area is the area where the wrinkle effect needs to be generated, and the blank area does not need to generate the wrinkle effect.
[0065] Furthermore, before adjusting the brightness value of the wrinkle texture image to be processed according to the deformed expression ratio map to obtain the target wrinkle texture image, it is possible to determine for each pixel in the wrinkle texture image to be processed whether the pixel is within the mask area of the deformed mask map. If so, the brightness value of the pixel is adjusted according to the target expression ratio map. If not, the pixel value is reset to zero to obtain the target wrinkle texture image.
[0066] Among them, the mask image can be generated according to the feature points, or it can be made manually. This application may include one or more mask images, and each mask image may have one or more mask areas, which are not specifically limited. For example, if a forehead wrinkle effect is required, a mask for the forehead wrinkle area needs to be generated; similarly, masks for the corners of the eyes and the corners of the mouth can also be generated. For eye wrinkles and nasolabial folds, because the intensity on the left and right may be different, masks need to be generated separately. In this way, wrinkle special effects rendering can be extended to any area of the face, and can also be adjusted according to user needs.
[0067] In one implementation, the mask image includes a plurality of regions, each of the regions includes two reference points, the target face image is a single frame image in a face image sequence, and the brightness value of the pixel is adjusted according to the target expression ratio map, including:
[0068] Performing expression recognition on the facial image sequence to determine a neutral facial image; modeling the neutral facial image to obtain a neutral facial model; for each mask area in the mask image, calculating the compression ratio of the mask area according to the ratio of the distances between two reference points in the mask area in the facial model to be processed and the neutral facial model; adjusting the brightness value of the pixel according to the target expression ratio map and the compression ratio.
[0069] Wherein, the compression ratio of the mask area is calculated according to the ratio of the distances between the two reference points in the mask area in the face model to be processed and the neutral face model, and the calculated ratio can be directly used as the compression ratio, or the calculated ratio can be processed using a preset positive correlation function, and there is no specific limitation. According to the target expression ratio map and the compression ratio, the brightness value of the pixel is adjusted, and the value of the pixel in the target expression ratio map, the compression ratio and the brightness value of the pixel can be directly multiplied, or a coefficient can be multiplied for fine-tuning according to actual needs.
[0070] For example, the wrinkle texture image to be processed can be converted into a YUV image; then, for each pixel in the wrinkle texture image to be processed, the value of the brightness channel of the pixel is multiplied by the value of the corresponding pixel in the target expression ratio map and the compression ratio of the corresponding mask area to obtain the value of the brightness channel of the updated YUV image; then, the updated YUV image is converted into an RGB image.
[0071] In an embodiment of the present invention, the position of the reference point can be determined based on a pre-acquired configuration file, in which the serial numbers of the reference points a and b selected for each region in the three-dimensional grid are recorded. For forehead wrinkles, the center point of the forehead at the hairline is selected as point a, and the center point of the brow bone is selected as point b; for nasolabial folds, the center points of both cheeks are selected as point a, and the corner of the mouth is selected as point b. The positions of point a and point b can be located within the mask area or outside the mask area, and are not specifically limited.
[0072] In this way, a is taken as a fixed reference point, that is, it is considered that this point will only have rigid changes, and b is taken as a variable reference point. The generation of wrinkles is simply considered to be a process of compression of b toward a, and the intensity change depends on the degree of compression, so that the generated wrinkles can be more natural without uneven thickness.
[0073] S105: superimposing the target wrinkle texture image on the face model to be processed for rendering, to obtain a face model with wrinkle effect.
[0074] The target wrinkle texture image includes a fourth feature point, and the fourth feature point and the target feature point are obtained by using the same facial feature point algorithm or feature point configuration file, that is, the facial feature points of the target wrinkle texture image and the target facial image correspond.
[0075] In this step, the position of the target feature point can be transformed into the coordinates corresponding to the face model to be processed, and then the pixel value of the target wrinkle texture image corresponding to the target feature point is multiplied with the pixel value of the corresponding pixel point in the target face model to obtain a face model with wrinkle effect.
[0076] As can be seen from the above, in this application, the pre-acquired expression ratio map and the preset wrinkle texture image are deformed according to the correspondence between the facial feature points to match the target facial image. Then, the brightness of the deformed wrinkle texture image is adjusted according to the deformed expression ratio map to make the deformed wrinkle texture image more natural. Then, the texture image with adjusted brightness is rendered onto the target facial model to simulate the wrinkle effect. Compared with the method based on the physical model, the calculation is simpler and can be used for real-time rendering, thereby improving the user experience. Moreover, alpha blending is no longer required in the wrinkle special effect rendering, so some strange color block effects can be eliminated.
[0077] Reference Figure 2 , shows a flowchart of another wrinkle special effect rendering method of the present application, which may specifically include the following steps:
[0078] S201: Detect feature points of a target face image as target feature points, and model the target face image to obtain a face model to be processed.
[0079] In the present application, the target facial image can be acquired in real time through the camera of the device, or can also be acquired through a remote camera or from a video file, without specific limitation.
[0080] Among them, a pre-acquired facial feature point detection algorithm can be used to perform feature point detection on the target facial image, and the facial feature point detection algorithm can be any current algorithm. Alternatively, a feature point configuration file can be obtained from a server in real time to perform feature point detection on the target facial image. The feature point configuration file includes multiple preset facial feature points, and the feature information of any facial image can be determined based on these preset facial feature points.
[0081] The feature point configuration file can be generated by the server. According to the different services corresponding to the current device, the server can adjust the distribution and quantity of facial feature points. For example, more facial feature points can be set in areas with dense wrinkles such as the forehead, corners of the eyes, and corners of the mouth, so that the wrinkle special effects rendering effect is more realistic.
[0082] In this step, the target face image can be 3D modeled by 3DMM (3D morphable model, 3D face deformation statistical model) technology to obtain the face model to be processed. Among them, 3DMM technology can represent different face images as 3D grids with the same sequence number, so that it is easy to locate and process the obtained face model to be processed.
[0083] S202: According to the position of the target feature point in the target face image and the correspondence between the target feature point and the first feature point in the pre-acquired expression ratio map, the expression ratio map is deformed to obtain a target expression ratio map.
[0084] In the present application, the first feature point and the target feature point of the expression ratio map are obtained using the same facial feature point algorithm or feature point configuration file, that is, the facial feature points of the expression ratio map and the target facial image correspond.
[0085] According to the position of the target feature point in the target face image, and the correspondence between the target feature point and the first feature point in the pre-acquired expression ratio map, the expression ratio map can be deformed, so that the position of the first feature point in the expression ratio map is the same as the position of the target feature point in the target face image. In this step, a triangulation algorithm can be used to deform the expression ratio map.
[0086] Among them, the expression ratio map can be obtained in the following way:
[0087] First, you need to prepare a wrinkle-free photo and a wrinkle-free photo. If the facial feature points of the wrinkle-free photo and the wrinkle-free photo are different, you need to calibrate them. The wrinkle-free photo can be taken naturally or processed from the wrinkle-free photo. Then, convert the wrinkle-free photo and the wrinkle-free photo into YUV color space respectively, calculate the ratio of the brightness value of the brightness channel of the wrinkle-free photo and the wrinkle-free photo, and save the calculation result as a single-channel image to obtain ERI (expression ratio image).
[0088] S203: Performing deformation processing on the preset wrinkle texture image according to the position of the target feature point in the target face image and the correspondence between the target feature point and the second feature point in the preset wrinkle texture image to generate a wrinkle texture image to be processed.
[0089] In the present application, the second feature point of the preset wrinkle texture image and the target feature point are obtained using the same facial feature point algorithm or feature point configuration file, that is, the facial feature points of the preset wrinkle texture image and the target facial image correspond.
[0090] According to the position of the target feature point in the target face image and the correspondence between the target feature point and the second feature point in the preset wrinkle texture image, the preset wrinkle texture image can be deformed, so that the position of the second feature point in the preset wrinkle texture image is the same as the position of the target feature point in the target face image. In this step, a triangulation algorithm can be used to deform the preset wrinkle texture image.
[0091] S204: deforming the mask image according to the position of the target feature point in the target face image and the correspondence between the target feature point and the third feature point in the pre-acquired mask image, wherein the mask image includes a plurality of regions, each of which includes two reference points.
[0092] In a face image, different types of wrinkles can be generated in multiple regions, for example, forehead wrinkles can be generated in the forehead region, crow's feet can be generated in the eye corner region, nasolabial folds can be generated in the mouth corner region, etc. Generating wrinkles in different regions can make the obtained face model with wrinkles more realistic. In one implementation, a mask image can be used to control the generation of wrinkles in different regions.
[0093] Among them, the mask image can be generated according to the feature points, or it can be made manually. This application may include one or more mask images, and each mask image may have one or more mask areas, which are not specifically limited. For example, if a forehead wrinkle effect is required, a mask for the forehead wrinkle area needs to be generated; similarly, masks for the corners of the eyes and the corners of the mouth can also be generated. For eye wrinkles and nasolabial folds, because the intensity on the left and right may be different, masks need to be generated separately. In this way, wrinkle special effects rendering can be extended to any area of the face, and can also be adjusted according to user needs.
[0094] Among them, the mask can be generated based on feature points or manually made. If a forehead wrinkle effect is required, a mask for the forehead wrinkle area needs to be generated; similarly, masks for the corners of the eyes and mouth can also be generated. For eye wrinkles and nasolabial folds, because the strength of the left and right sides may be different, masks need to be generated separately. In this way, wrinkle special effect rendering can be extended to any area of the face.
[0095] The position of the reference point can be determined according to a pre-acquired configuration file, in which the serial numbers of the reference points a and b selected for each area in the three-dimensional grid are recorded. For forehead wrinkles, the center point of the hairline on the forehead is selected as point a, and the center point of the brow bone is selected as point b; for nasolabial folds, the center points of both cheeks are selected as point a, and the corner of the mouth is selected as point b. The selection of ab points does not need to be within the mask range. This can make the generated wrinkles more natural without uneven thickness.
[0096] S205: Acquire the current face image; model the current face image to obtain the current face model; for each mask area in the mask image, calculate the ratio of the distances between two reference points in the mask area in the face model to be processed and the current face model as the compression ratio of the area.
[0097] S206: For each pixel in the preset wrinkle texture image, determine whether the pixel is within the mask area of the deformed mask image. If so, adjust the brightness value of the pixel according to the target expression ratio map and the compression ratio. If not, reset the pixel value to zero to obtain the target wrinkle texture image.
[0098] For example, the wrinkle texture image to be processed can be converted into a YUV image; then, for each pixel in the wrinkle texture image to be processed, the value of the brightness channel of the pixel is multiplied by the value of the corresponding pixel in the target expression ratio map and the compression ratio of the corresponding mask area to obtain the value of the brightness channel of the updated YUV image; then, the updated YUV image is converted into an RGB image, which is the target wrinkle texture image.
[0099] S207: superimposing the target wrinkle texture image on the face model to be processed for rendering to obtain a face model with wrinkle effect.
[0100] The target wrinkle texture image includes a fourth feature point, and the fourth feature point and the target feature point are obtained by using the same facial feature point algorithm or feature point configuration file, that is, the facial feature points of the target wrinkle texture image and the target facial image correspond.
[0101] In this step, the position of the target feature point can be transformed into the coordinates corresponding to the face model to be processed, and then the pixel value of the target wrinkle texture image corresponding to the target feature point is multiplied with the pixel value of the corresponding pixel point in the target face model to obtain a face model with wrinkle effect.
[0102] As can be seen from the above, in the present application, the pre-acquired expression ratio map and the preset wrinkle texture image are first deformed according to the correspondence between the facial feature points to match the target facial image. Then, the brightness of the deformed wrinkle texture image is adjusted according to the deformed expression ratio map to make the deformed wrinkle texture image more natural. The texture image with adjusted brightness is then rendered onto the target facial model to simulate the wrinkle effect. Compared with the method based on the physical model, the calculation is relatively simple and can be used for real-time rendering, thereby improving the user experience.
[0103] Reference Figure 3 , shows an application example diagram of a wrinkle special effects rendering method of the present application. It can be seen that wrinkles can be drawn on the forehead, corners of the mouth, corners of the eyes and other parts of the face, and it is also more natural.
[0104] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions involved are not necessarily required by the present application.
[0105] Reference Figure 3 , shows a structural block diagram of a wrinkle special effect rendering device of the present application, and the device may specifically include the following modules:
[0106] The detection module 301 is used to detect feature points of a target face image as target feature points, and to model the target face image to obtain a face model to be processed;
[0107] A first deformation module 302 is used to perform deformation processing on the expression ratio map according to the position of the target feature point in the target face image and the correspondence between the target feature point and the first feature point in the pre-acquired expression ratio map to obtain a target expression ratio map;
[0108] The second deformation module 303 is used to perform deformation processing on the preset wrinkle texture image according to the position of the target feature point in the target face image and the correspondence between the target feature point and the second feature point in the preset wrinkle texture image to generate a wrinkle texture to be processed;
[0109] An adjustment module 304 is used to adjust the brightness value of the wrinkle texture image to be processed according to the target expression ratio map to obtain a target wrinkle texture image;
[0110] The rendering module 305 is used to superimpose the target wrinkle texture image on the face model to be processed for rendering, so as to obtain a face model with wrinkle effect.
[0111] In one implementation, the wrinkle texture image is an RGB image, and the adjustment module 304 is specifically used to:
[0112] Convert the pre-acquired wrinkle texture image into a YUV image;
[0113] For each pixel in the wrinkle texture image, multiply the value of the brightness channel of the pixel by the value of the corresponding expression ratio map to obtain an updated value of the brightness channel;
[0114] Convert the updated YUV image to RGB image.
[0115] In one implementation, the device further includes:
[0116] A third deformation module, configured to perform deformation processing on the mask image according to the position of the target feature point in the target face image and the correspondence between the target feature point and the second feature point in the pre-acquired mask image;
[0117] The adjustment module 304 is specifically used to determine, for each pixel in the preset wrinkle texture image, whether the pixel is within the deformed mask image; if so, adjust the brightness value of the pixel according to the deformed expression ratio image; if not, reset the pixel value to zero to obtain the target wrinkle texture image.
[0118] In one implementation, the mask image includes a plurality of regions, each of the regions includes two reference points, and the adjustment module 304 is specifically configured to:
[0119] Get the current face image;
[0120] Modeling the current face image to obtain a current face model;
[0121] For each region in the mask image, calculating the ratio of the distances between two reference points in the region in the to-be-processed face model and the current face model as the compression ratio of the region;
[0122] The brightness value of the pixel is adjusted according to the deformed expression ratio map and the compression ratio.
[0123] In one implementation, the device further includes:
[0124] The expression ratio map generation module is used to obtain a wrinkled face image and a wrinkle-free face image with the same facial feature point positions; convert the wrinkled face image and the wrinkle-free face image into YUV images respectively; for each pixel in the wrinkled face image, calculate the ratio of the value of the pixel brightness channel to the value of the brightness channel of the corresponding pixel in the wrinkle-free face image, and obtain the value of the pixel at the same position in the expression ratio map.
[0125] As can be seen from the above, in the present application, the pre-acquired expression ratio map and the preset wrinkle texture image are first deformed according to the correspondence between the facial feature points to match the target facial image. Then, the brightness of the deformed wrinkle texture image is adjusted according to the deformed expression ratio map to make the deformed wrinkle texture image more natural. The texture image with adjusted brightness is then rendered onto the target facial model to simulate the wrinkle effect. Compared with the method based on the physical model, the calculation is relatively simple and can be used for real-time rendering, thereby improving the user experience.
[0126] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0127] The wrinkle special effect rendering device provided in the embodiment of the present invention has a functional module corresponding to the wrinkle special effect rendering method, and can execute the wrinkle special effect rendering method provided in the embodiment of the present invention and achieve the same beneficial effects.
[0128] In another embodiment of the present invention, an electronic device is provided. The electronic device may include: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the program, each process of the wrinkle special effect rendering method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not described here. For example, Figure 5 As shown, the electronic device may specifically include: a processor 401, a storage device 402, a display screen 403 with a touch function, an input device 404, an output device 405, and a communication device 406. The number of processors 401 in the electronic device may be one or more. Figure 4 A processor 401 is taken as an example. The processor 401, storage device 402, display screen 403, input device 404, output device 405 and communication device 406 of the electronic device can be connected via a bus or other means.
[0129] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions. When the instructions are executed on a computer, the computer executes the wrinkle special effect rendering method described in any of the above embodiments.
[0130] In another embodiment provided by the present invention, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the wrinkle special effect rendering method described in any one of the above embodiments.
[0131] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0132] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0133] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.
Claims
1. A wrinkle special effects rendering method, It is characterized in that The method comprises: Detecting feature points of a target face image as target feature points, and modeling the target face image to obtain a face model to be processed; According to the position of the target feature point in the target facial image and the correspondence between the target feature point and the first feature point in the pre-acquired expression ratio map, the expression ratio map is deformed to obtain a target expression ratio map; the expression ratio map is determined according to the wrinkled facial image and the wrinkle-free facial image, and the pixel value of each pixel in the expression ratio map is the ratio of the brightness value of the corresponding pixel in the wrinkled facial image and the wrinkle-free facial image on the brightness channel; According to the position of the target feature point in the target face image and the correspondence between the target feature point and the second feature point in the preset wrinkle texture image, the preset wrinkle texture image is deformed to generate a wrinkle texture image to be processed; According to the target expression ratio map, adjusting the brightness value of the wrinkle texture image to be processed to obtain a target wrinkle texture image; The target wrinkle texture image is superimposed on the face model to be processed for rendering to obtain a face model with wrinkle effect.
2. The method according to claim 1, It is characterized in that The wrinkle texture image to be processed is an RGB image, and the brightness value of the wrinkle texture image to be processed is adjusted according to the target expression ratio map to obtain the target wrinkle texture image, including: Converting the wrinkle texture image to be processed into a YUV image; For each pixel in the wrinkle texture image to be processed, multiply the value of the brightness channel of the pixel by the value of the corresponding pixel in the target expression ratio image to obtain the value of the brightness channel of the updated YUV image; Convert the updated YUV image to an RGB image as the target wrinkle texture image.
3. The method according to claim 1, It is characterized in that Before adjusting the brightness value of the wrinkle texture image to be processed according to the target expression ratio map to obtain the target wrinkle texture image, the method further includes: According to the position of the target feature point in the target face image and the correspondence between the target feature point and a third feature point in the pre-acquired mask image, the mask image is deformed, wherein the mask image includes a mask area and a blank area; The step of adjusting the brightness value of the wrinkle texture image to be processed according to the target expression ratio map to obtain a target wrinkle texture image comprises: For each pixel in the wrinkle texture image to be processed, determine whether the pixel is in the mask area of the deformed mask image. If so, adjust the brightness value of the pixel according to the target expression ratio map. If not, reset the pixel value to zero to obtain the target wrinkle texture image.
4. The method according to claim 3, It is characterized in that The mask image includes a plurality of regions, each of which includes two reference points. The target face image is a single frame image in a face image sequence. The target wrinkle texture image is obtained by adjusting the brightness value of the pixel according to the target expression ratio map, including: Performing expression recognition on the facial image sequence to determine a neutral facial image; Modeling the neutral face image to obtain a neutral face model; For each mask area in the mask image, calculating a compression ratio of the mask area according to a ratio of distances between two reference points in the mask area in the to-be-processed face model and the neutral face model; According to the target expression ratio map and the compression ratio, the brightness value of the pixel is adjusted to obtain a target wrinkle texture image.
5. The method according to claim 3, It is characterized in that The method for obtaining the expression ratio map includes: Obtaining a wrinkled face image and a wrinkle-free face image with the same facial feature point positions; Converting the wrinkled face image and the wrinkle-free face image into YUV images respectively; For each pixel in the wrinkled face image, the ratio of the value of the brightness channel of the pixel to the value of the brightness channel of the corresponding pixel in the wrinkle-free face image is calculated to obtain the value of the pixel at the same position in the expression ratio map.
6. A wrinkle special effects rendering device, It is characterized in that The device comprises: A detection module is used to detect feature points of a target face image as target feature points, and to model the target face image to obtain a face model to be processed; A first deformation module is used to perform deformation processing on the expression ratio map according to the position of the target feature point in the target face image and the correspondence between the target feature point and the first feature point in the pre-acquired expression ratio map to obtain a target expression ratio map; the expression ratio map is determined according to the wrinkled face image and the wrinkle-free face image, and the pixel value of each pixel in the expression ratio map is the ratio of the brightness value of the corresponding pixel in the wrinkled face image and the wrinkle-free face image on the brightness channel; A second deformation module, configured to perform deformation processing on the preset wrinkle texture image according to the position of the target feature point in the target face image and the correspondence between the target feature point and the second feature point in the preset wrinkle texture image, so as to generate a wrinkle texture image to be processed; An adjustment module, used for adjusting the brightness value of the wrinkle texture image to be processed according to the target expression ratio map to obtain a target wrinkle texture image; A rendering module is used to superimpose the target wrinkle texture image on the face model to be processed for rendering, so as to obtain a face model with wrinkle effect.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the steps of the wrinkle special effect rendering method according to any one of claims 1 to 5 are implemented.
8. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the wrinkle special effect rendering method according to any one of claims 1 to 5 are implemented.
Citation Information
Patent Citations
High-reality dynamic expression modeling method
CN103854306A
Wrinkle generation method and electronic equipment
CN107292939A
Image processing method and apparatus, mobile terminal and computer readable storage medium
CN107766831A
Face wrinkle generation method and device, computer storage medium and terminal
CN109753892A
Human face countenance synthesis method based on dense characteristic corresponding and morphology
CN1870049A
