Virtual makeup method, device, equipment and computer-readable storage medium

Abstract

The present application provides a virtual makeup method, device, equipment and computer-readable storage medium, which obtains a target image containing a face; determines the face rotation angle and multiple face key points on the target image; wherein the face rotation angle refers to the angle formed in the horizontal direction when the face is deflected based on the front standard posture; adjusts each face key point based on the face rotation angle; and applies makeup to the target image based on each adjusted face key point and makeup material. When the face rotates from the front standard posture (i.e., when it turns into a side face), the virtual makeup method adjusts (or corrects) the face key points based on the face rotation angle generated during the rotation, and then performs makeup processing based on the corrected face key points, which can effectively prevent the makeup material from being stretched. In addition, the makeup material can be drawn at a precise position on the face to further improve the makeup effect.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a virtual makeup method, device, equipment and computer-readable storage medium. Background Art

[0002] At present, there is often a need to automatically apply makeup to the user's face in video processing scenarios (such as live webcast scenarios, short video scenarios, etc.). Currently, the commonly used facial makeup methods are mostly based on facial key points, that is, the facial key points are first identified, and then the makeup is drawn based on the corresponding makeup material at the location of the facial key points to complete the makeup. This method has a better makeup effect on faces in standard postures. When the face is deflected, the facial makeup material will be stretched, which affects the makeup effect. Summary of the invention

[0003] In view of this, embodiments of the present application provide a virtual makeup method, apparatus, device, and computer-readable storage medium.

[0004] In a first aspect, an embodiment of the present application provides a virtual makeup method, the method comprising:

[0005] Get a target image containing a face;

[0006] Determining a face rotation angle and a plurality of face key points on the target image; wherein the face rotation angle refers to an angle formed in the horizontal direction when the face is deflected based on a frontal standard posture;

[0007] Adjusting each of the facial key points based on the facial rotation angle;

[0008] The target image is made up based on each adjusted facial key point and makeup material.

[0009] In a second aspect, an embodiment of the present application provides a virtual makeup device, the device comprising:

[0010] A template image module is used to obtain a target image containing a face;

[0011] A key point and rotation angle determination module, used to determine the face rotation angle and multiple face key points based on the target image; wherein the face rotation angle refers to the angle formed in the horizontal direction when the face is deflected based on the front standard posture;

[0012] A key point adjustment module, used for adjusting each of the key points of the face based on the face rotation angle;

[0013] The makeup module is used to apply makeup to the target image based on the adjusted key points of each face and the makeup material.

[0014] In a third aspect, an embodiment of the present application provides a terminal device, comprising: a memory; one or more processors coupled to the memory; one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the virtual makeup method provided in the first aspect above.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute the virtual makeup method provided in the first aspect above.

[0016] The virtual makeup method, device, equipment and storage medium provided in the embodiments of the present application first obtain a target image containing a face; then determine the face rotation angle and multiple face key points on the target image; wherein the face rotation angle refers to the angle formed in the horizontal direction when the face is deflected based on the front standard posture; adjust each face key point based on the face rotation angle; finally, apply makeup to the target image based on each adjusted face key point and makeup material.

[0017] Since the key points of the face are usually offset during the rotation of the face (i.e., from the front standard posture to the side face state), that is, there is a deviation between the actual key points of the face and the detected key points of the face, if makeup is applied based on the detected key points of the face, the makeup material will be stretched. Therefore, the virtual makeup method in this embodiment adjusts (or corrects) the key points of the face based on the rotation angle of the face generated during the rotation when the face rotates from the front standard posture (i.e., when it turns to the side face), and then performs makeup processing based on the corrected key points of the face, which can effectively prevent the makeup material from being stretched. In addition, the makeup material can be drawn at the precise position of the face to further improve the makeup effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0019] Figure 1 A schematic diagram of an application scenario of the virtual makeup application method provided in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of makeup in a standard frontal posture (i.e., frontal face) and a side face provided in an embodiment of the present application;

[0021] Figure 3 A schematic diagram of a triangle mesh of a frontal standard posture (i.e., frontal face), a near-side face before and after correction is provided for one embodiment of the present application;

[0022] Figure 4 A schematic diagram of a virtual makeup application method according to an embodiment of the present application;

[0023] Figure 5 Provide a schematic diagram of a triangle mesh of a frontal standard posture (i.e., front face), a far-side face before and after correction for one embodiment of the present application;

[0024] Figure 6 A schematic diagram of a face mask provided for one embodiment of the present application;

[0025] Figure 7 A flowchart of a virtual makeup application method provided in yet another embodiment of the present application;

[0026] Figure 8 A structural diagram of a virtual makeup application device provided in one embodiment of the present application;

[0027] Fig. 9 A schematic diagram of the structure of a terminal device provided in one embodiment of the present application;

[0028] Fig.10 A schematic diagram of the structure of a computer-readable storage medium provided in one embodiment of the present application. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0030] In order to explain the present application in more detail, a virtual makeup method, apparatus, terminal device and computer storage medium provided by the present application are specifically described below in conjunction with the accompanying drawings.

[0031] Please refer to Figure 1 , Figure 1A schematic diagram of an application scenario of the virtual makeup method provided in an embodiment of the present application is shown, and the application scenario includes a terminal device 100 provided in an embodiment of the present application, and the terminal device 100 can be various electronic devices with a display screen (such as the structural diagrams of 102, 104, 106 and 108), including but not limited to smart phones and computer devices, wherein the computer device can be at least one of a desktop computer, a portable computer, a laptop computer, a tablet computer, etc. First, a beauty application with a virtual makeup function is usually installed on the terminal device 100.

[0032] Optionally, the beauty application with a virtual makeup function may also be a live broadcast application with a beauty function, which can be used to perform virtual makeup (such as adding some makeup materials) on the face of the person (usually the host who is broadcasting) captured in the image, and then push the local video stream data to other live broadcast clients through the live broadcast server. Furthermore, the beauty application with a virtual makeup function may also be a short video or small video application with a beauty function, which is used to perform virtual makeup (such as adding some makeup materials) on the face during the short video shooting process, and publish the shot short video to the short video platform for other users to click and watch. The beauty application with a virtual makeup function in the embodiment of the present application may be one or more of the beauty applications described above, and is not specifically limited here.

[0033] Secondly, the terminal device 100 may generally refer to one of a plurality of terminal devices, and this embodiment is only illustrated by the terminal device 100. Those skilled in the art may know that the number of the above-mentioned terminal devices may be more or less. For example, the above-mentioned terminal devices may be only a few, or the above-mentioned terminal devices may be dozens or hundreds, or more. The embodiment of the present application does not limit the number and type of terminal devices. The terminal device 100 may be used to execute a virtual makeup method provided in the embodiment of the present application.

[0034] Optionally, a GPU may be integrated in the terminal device 100, and the GPU may be used to execute a virtual makeup method provided in an embodiment of the present application. The use of a GPU may improve the makeup processing process.

[0035] In an optional implementation, the application scenario includes, in addition to the terminal device 100 provided in the embodiment of the present application, a server, wherein a network is provided between the server and the terminal device. The network is used to provide a medium for a communication link between the terminal device and the server. The network may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0036] It should be understood that the number of terminal devices, networks and servers is only schematic. Depending on the implementation needs, there can be any number of terminal devices, networks and servers. For example, the server can be a server cluster composed of multiple servers, etc. Among them, the terminal device interacts with the server through the network to receive or send messages, etc. The server can be a server that provides various services. The server can be used to execute the steps of a virtual makeup method provided in an embodiment of the present application. In addition, when the terminal device executes a virtual makeup method provided in an embodiment of the present application, part of the steps can be executed on the terminal device and part of the steps can be executed on the server, which is not limited here.

[0037] The virtual makeup method provided in the embodiment of the present application can be used to apply makeup to a human face, and can be widely used in the fields of online live broadcasting and short videos. For ease of understanding, the image processing method is described below in the online live broadcasting application scenario.

[0038] In addition, virtual makeup is usually based on face recognition key points, and then the face mesh (such as a triangle mesh) is drawn, and then the makeup is drawn. This method works better when the face is in a standard frontal posture and the face deflection angle is small, but when the face deflection angle is large, the facial makeup material will be stretched. Please refer to Figure 2 As shown in the figure, when the face is facing forward, the heart on the cheek is still in normal state. When the face is turned to the side, the heart on the cheek is stretched and deformed. The main reason for this is that the facial triangle mesh for drawing makeup is based on facial key point recognition. When the face is deflected at a large angle (especially the edge key points), the facial key points are in different positions than when the face is in a standard frontal posture and the face is deflected at a small angle. Please refer to Figure 3 As shown in the figure, the key points of the face will be offset when the face is in the standard frontal posture and in the side face state, so the generated triangle mesh is also different. Then, if the original key points of the face (that is, the key points of the face in the standard frontal posture and when the face is deflected at a small angle) are used to draw the makeup, the makeup material will be stretched.

[0039] Based on this, a virtual makeup method is provided in the embodiment of the present application. Figure 4 , Figure 4 A schematic diagram of a virtual makeup application method provided in an embodiment of the present application is shown. Figure 1 The terminal device in the example is used to illustrate, including the following steps:

[0040] Step S110, obtaining a target image containing a face.

[0041] The target image refers to an image that needs to be virtually made up and includes facial data or information.

[0042] Virtual makeup means that the user's facial image can be processed through beauty applications, and the corresponding makeup is drawn on the corresponding parts of the facial features or cheeks using makeup materials, so that the processed user's facial image has the effect of makeup.

[0043] Makeup materials are usually some makeup pictures or elements that can be covered on the corresponding parts of the face to produce a makeup effect. Makeup materials can include but are not limited to lipstick, blush, eyelashes, eyebrows, eyeliner, eye shadow and other stickers that can be attached to the face (such as love, flags, etc.).

[0044] Step S120, determining the face rotation angle and multiple face key points on the target image.

[0045] The face rotation angle refers to the angle formed in the horizontal direction when the face is deflected based on the front standard posture.

[0046] When the face is completely frontal and the eyes are looking straight ahead, the posture formed is the front standard posture. When the face is rotated to the right or left based on the front standard posture, the left face or right face is formed. The angle formed by the left face or right face in the horizontal direction is recorded as the face rotation angle.

[0047] Step S130: adjusting each facial key point based on the face rotation angle.

[0048] Specifically, after knowing the position or coordinates of the facial key points and the face rotation angle, the position or coordinates of the facial key points after the face rotation can be calculated, and then the facial key points can be updated or adjusted.

[0049] Step S140, applying makeup to the target image based on the adjusted facial key points and makeup materials.

[0050] After adjusting the key points of the face, a triangular mesh is drawn based on the adjusted key points of the face, and then makeup is drawn based on the drawn triangular mesh and the makeup material, thereby completing the makeup processing on the target image.

[0051] Since the key points of the face are usually offset during the rotation of the face (i.e., from the front standard posture to the side face state), that is, there is a deviation between the actual key points of the face and the detected key points of the face, if makeup is applied based on the detected key points of the face, the makeup material will be stretched. Therefore, the virtual makeup method in this embodiment adjusts (or corrects) the key points of the face based on the rotation angle of the face generated during the rotation when the face rotates from the front standard posture (i.e., when it turns to the side face), and then performs makeup processing based on the corrected key points of the face, which can effectively prevent the makeup material from being stretched. In addition, the makeup material can be drawn at the precise position of the face to further improve the makeup effect.

[0052] In one embodiment, step S130 is executed to adjust each facial key point based on the face rotation angle, including: determining the starting vector and the end point vector of the vertical central axis of the face according to the target image; calculating the vector difference of the vertical central axis of the face according to the starting vector and the end point vector; and adjusting each facial key point according to the starting vector, the vector difference and the face rotation angle.

[0053] Specifically, the vertical axis of the face refers to the central axis of the left and right faces. The starting point vector refers to the vector formed from the origin of the coordinate system (usually (0,0)) to the starting point of the central axis; the end point vector refers to the vector formed from the origin of the coordinate system (usually (0,0)) to the end point of the central axis. The starting point of the central axis can be the center point of the forehead or the center point of the two eyes. The end point can be the center point of the chin.

[0054] In practical applications, on the one hand, the forehead points are estimated, so the forehead points may not be very accurate; and there are large individual differences, and the forehead points of different users are not in the same position; on the other hand, pattern makeup is generally rarely used on the forehead, and the forehead points exist mainly to provide a transition buffer zone for makeup such as eyebrows. Based on this, in a preferred implementation, the starting point of the central axis is selected as the center point of the two eyes. The end point can be the center point of the chin. After determining the starting vector and the end point vector, the vector difference of the vertical central axis of the face can be calculated. After obtaining the vector difference, each facial key point can be adjusted according to the starting vector, the vector difference and the face rotation angle.

[0055] Furthermore, several specific implementation methods of adjusting facial key points are given, which are described as follows:

[0056] The first implementation method:

[0057] In one embodiment, the facial key points include proximal facial key points, which refer to the facial key points of the side face close to the front standard posture when the face is deflected; each facial key point is adjusted according to the starting point vector, the vector difference of the vertical central axis of the face and the face rotation angle, including: correcting the vector difference based on the quantity value of the proximal key point; calculating the offset vector of each proximal facial key point according to the starting point vector, the corrected vector difference, each proximal facial key point and the face rotation angle; determining each new proximal facial key point according to each proximal facial key point and the offset vector corresponding to each proximal facial key point; and updating each proximal facial key point to the corresponding new proximal facial key point.

[0058] Specifically, the facial key points may include near-side facial key points and far-side facial key points; near-side facial key points refer to key points of the near-side face, and far-side facial key points refer to key points of the far-side face; wherein, the near-side face refers to the side face close to the front standard posture when the face is deflected, and the far-side face refers to the side face that is not in the front standard posture when the face is deflected. For example, when the face turns to the left, the right side of the face is the near-side face, and the left side of the face is the far-side face; on the contrary, when the face turns to the right, the right side of the face is the far-side face, and the left side of the face is the near-side face. When adjusting the near-side face, the vector difference is first corrected based on the quantity value of the near-side key point; the specific correction process can be to place the vector difference at the quantity value of the near-side key point. Then, the offset vector of each near-side facial key point is calculated based on the starting point vector, the corrected vector difference, each near-side facial key point and the face rotation angle. After obtaining the offset vector of each near-side facial key point, each near-side facial key point is summed with the offset vector corresponding to each near-side facial key point to determine each new near-side facial key point; finally, each near-side facial key point is updated to the corresponding new near-side facial key point. This method can make the calculated new near-side facial key points more accurate.

[0059] It should be noted that when adjusting the near-side face key points, the near-side face key points are usually retracted inward. In addition, when adjusting the near-side face key points, you can adjust the key points of the entire face, or you can only adjust the key points of the entire cheek edge below the forehead. Since the forehead key points and the key points of the facial features are not obviously offset when the face is deflected to form a side face, and makeup materials are rarely drawn in the forehead area, in order to speed up the operation and the entire makeup process, you can choose to only adjust the key points of the entire cheek edge below the forehead.

[0060] Furthermore, an implementation method for calculating the offset vector of the near-side face key point is given, which is specifically described as follows:

[0061] In one embodiment, the offset vector of each near-side facial key point is calculated according to the starting point vector, the corrected vector difference, each near-side facial key point and the face rotation angle, including: calculating the vector sum of the starting point vector and the corrected vector difference, recorded as the first vector; determining each near-side facial key point vector based on each near-side facial key point; calculating the vector difference between the first vector and each near-side facial key point vector, recorded as each second vector; calculating the offset vector of each near-side facial key point according to the first prediction vector coefficient, the face rotation angle and each second vector. The vector of the near-side facial key point refers to the vector formed from the coordinate origin to the near-side facial key point.

[0062] Specifically, the near-side key points of the face (i.e., edge key points) can be set as P1, P2, ..., Pn, where n represents the number of near-side key points of the face; the starting vector and the end vector of the vertical axis of the face are recorded as P FaceAxisStart , P FaceAxisEnd ; Face deflection angle θ, vector difference = P FaceAxisEnd -P FaceAxisStart ; Corrected vector difference P step =1 / n(P FaceAxisEnd -P FaceAxisStart ); first vector = P FaceAxisStart +i*P step , n represents the number of key points near the face; the second vector = P FaceAxisStart +i*P step -P i ; offset vector = k i *sin(θ)(P FaceAxisStart +i*P step -P i ). Then the calculation formula for the i-th new near-side key point of the face is as follows:

[0063]

[0064] in, Represents the coordinates of the i-th new near-side facial key point; k represents the coordinates of the i-th near-side facial key point, 1≤i≤n; i Represents the first prediction vector coefficient, which can also be called the fine-tuning coefficient of the i-th near-side facial key point.

[0065] The second implementation method:

[0066] In one embodiment, the facial key points include far-side facial key points, which refer to the facial key points of the side face that is away from the front standard posture when the face is deflected; each facial key point is adjusted according to the starting point vector, the vector difference and the face rotation angle, including: calculating the offset vector of each far-side facial key point according to the starting point vector, the corrected vector difference, each far-side facial key point and the face rotation angle; determining each new far-side facial key point according to the starting point vector and the offset vector of each far-side facial key point; and updating each far-side facial key point to the corresponding new far-side facial key point.

[0067] When adjusting the far-side face, firstly, the vector difference is corrected based on the quantity value of the near-side key point; the specific correction process can be to put the vector difference at the quantity value of the near-side key point. Then, the offset vector of each far-side face key point is calculated according to the starting vector, the corrected vector difference, each far-side face key point and the face rotation angle. After obtaining the offset vector of each far-side face key point, the starting vector and the offset vector of each far-side face key point are summed to determine each new far-side face key point; finally, each far-side face key point is updated to the corresponding new far-side face key point. The use of this method can make the calculated new far-side face key point more accurate.

[0068] It should be noted that when adjusting the key points of the far side face, the key points of the far side face are usually expanded outwards, and the result is as follows Figure 5 As shown. In addition, when adjusting the key points of the far face, you can adjust the key points of the entire face, or just adjust the key points of the entire cheek edge below the forehead. Since the forehead key points and the key points of the facial features are not obviously offset when the face is deflected to form a side face, and makeup materials are rarely drawn in the forehead area, in order to speed up the entire makeup process, you can choose to adjust only the key points of the entire cheek edge below the forehead.

[0069] Furthermore, an implementation method for calculating the offset vector of the far-side face key point is given, which is specifically described as follows:

[0070] In one embodiment, the offset vector of each far-side face key point is calculated according to the starting point vector, the corrected vector difference, each far-side face key point and the face rotation angle, including: determining each far-side face key point vector based on each far-side face key point; calculating the vector difference between each far-side face key point vector and the first vector, recorded as each third vector; calculating each fourth vector according to the second prediction vector coefficient, the face rotation angle and each third vector; calculating the vector sum of the corrected vector difference and each fourth vector, recorded as the offset vector of each far-side face key point. The vector of the far-side face key point refers to the vector formed from the coordinate origin to the far-side face key point.

[0071] Specifically, following the above embodiment of the near-side key points of the face, the far-side key points of the face (i.e., edge key points) can be set as P1, P2, ..., Pm, where m represents the number of near-side key points of the face; the starting vector and the end vector of the vertical central axis of the face are respectively recorded as P FaceAxisStart , P FaceAxisEnd ; Face deflection angle θ, vector difference = P FaceAxisEnd -P FaceAxisStart ; Corrected vector difference P step =1 / n(P FaceAxisEnd -P FaceAxisStart ); the third vector = P j -(P FaceAxisStart +j*P step ); the fourth vector = (1+k j )*sin(θ)((P j -(P FaceAxisStart +j*P step )); offset vector = j*P step +(1+k j )*sin(θ)((P j -(P FaceAxisStart +j*P step )). Then the calculation formula for the jth new far-side face key point is as follows:

[0072]

[0073] in, Represents the coordinates of the jth new far-side face detection key point; Represents the coordinates of the jth near-side facial key point, 1≤j≤m; 1+k j k represents the second prediction vector coefficient. j Represents the fine-tuning coefficient of the j-th near-side facial landmark.

[0074] Next, an implementation method for applying makeup based on facial key points and makeup materials is given, which is described in detail as follows:

[0075] In one embodiment, makeup is applied to a target image based on each adjusted facial key point and makeup material, including: when a facial key point is a far-side facial key point, a facial mask template is drawn according to each facial key point before adjustment; and makeup is applied to the target image according to the facial mask template, each adjusted far-side facial key point and makeup material.

[0076] Since the adjustment of the far-side facial key points is to expand the facial key points, and then draw a triangle mesh based on the adjusted far-side facial key points, it is easy to cause makeup overflow when drawing makeup based on the triangle mesh and makeup material, that is, drawing the makeup material outside the face area. To avoid this problem, the facial key points before adjustment can be used to draw a face mask template (i.e., mask, such as Figure 6 Then, the target image is made up according to the face mask template, each adjusted face key point and the makeup material.

[0077] In one embodiment, makeup is applied to the target image according to the face mask template, each adjusted far-side face key point, and the makeup material to form a final makeup image, which can be expressed by the following formula:

[0078]

[0079] in, represents the final makeup image, Represents a triangular mesh formed according to each adjusted far-side face key point; Represents the face mask template (i.e., mask), Indicates makeup material; They represent the points of the face image on the coordinate axis respectively.

[0080] In order to facilitate understanding of the virtual makeup method of the present application, a detailed embodiment is given. Figure 7 The specific process of the virtual makeup method is as follows: 1. Obtain the user's target image; 2. Use face point detection on the target image; 3. Calculate the face key points when the face is deflected (i.e., side face), and then adjust or correct the face key points, and use the corrected face key points to draw the face triangle mesh; 4. Use the face key points (i.e., the face key points before correction) to draw the face triangle mesh; 5. Draw the face mask area based on the face triangle mesh drawn based on the face key points before correction; 6. Use the makeup material and the face triangle mesh drawn based on the corrected face key points to draw the makeup; 7. Make the drawn makeup and the face mask area intersect artificially to prevent the makeup from being drawn outside the face area, where this step is applicable to the makeup processing of the far side face, if it is the makeup of the near side face, this step can be omitted; 8. Mix the target image with the makeup, and output the final makeup image.

[0081] It should be understood that although Figure 2 and Figure 7The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Figure 2 and Figure 7 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0082] The above-mentioned embodiment disclosed in the present application describes in detail a virtual makeup method. The above-mentioned method disclosed in the present application can be implemented by various forms of equipment. Therefore, the present application also discloses a virtual makeup device corresponding to the above-mentioned method. A specific embodiment is given below for detailed description.

[0083] See also Figure 8 , is a virtual makeup device disclosed in an embodiment of the present application, mainly comprising:

[0084] The template image module 810 is used to obtain a target image containing a human face.

[0085] The key point and rotation angle determination module 820 is used to determine the face rotation angle and multiple face key points based on the target image; wherein the face rotation angle refers to the angle formed in the horizontal direction when the face is deflected based on the front standard posture.

[0086] The key point adjustment module 830 is used to adjust each face key point based on the face rotation angle.

[0087] The makeup module 840 is used to apply makeup to the target image based on the adjusted key points of each face and the makeup material.

[0088] In one embodiment, the key point adjustment module 830 is used to determine the starting vector and the end point vector of the vertical central axis of the face according to the target image; calculate the vector difference of the vertical central axis of the face according to the starting vector and the end point vector; and adjust each facial key point according to the starting vector, the vector difference and the face rotation angle.

[0089] In one embodiment, the facial key points include proximal facial key points, which refer to the facial key points of the side face close to the front standard posture when the face is deflected; the key point adjustment module 830 is used to correct the vector difference based on the quantity value of the proximal key points; calculate the offset vector of each proximal facial key point according to the starting point vector, the corrected vector difference, each proximal facial key point and the face rotation angle; determine each new proximal facial key point according to each proximal facial key point and the offset vector corresponding to each proximal facial key point; and update each proximal facial key point to the corresponding new proximal facial key point.

[0090] In one embodiment, the facial key points include far-side facial key points, which refer to the facial key points of the side face that are away from the front standard posture when the face is deflected; the key point adjustment module 830 is used to calculate the offset vector of each far-side facial key point based on the starting point vector, the corrected vector difference, each far-side facial key point and the face rotation angle; determine each new far-side facial key point based on the starting point vector and the offset vector of each far-side facial key point; and update each far-side facial key point to the corresponding new far-side facial key point.

[0091] In one embodiment, the key point adjustment module 830 is used to calculate the sum of the vectors of the starting point vector and the corrected vector difference, recorded as the first vector; determine each near-side face key point vector based on each near-side face key point; calculate the vector difference between the first vector and each near-side face key point vector, recorded as each second vector; calculate the offset vector of each near-side face key point based on the first prediction vector coefficient, the face rotation angle and each second vector.

[0092] In one embodiment, the key point adjustment module 830 is used to determine each far-side face key point vector based on each far-side face key point; calculate the vector difference between each far-side face key point vector and the first vector, recorded as each third vector; calculate each fourth vector according to the second prediction vector coefficient, the face rotation angle and each third vector; calculate the vector sum of the corrected vector difference and each fourth vector, recorded as the offset vector of each far-side face key point.

[0093] In one embodiment, the makeup module 840 is used to draw a face mask template according to each face key point before adjustment when the face key point is a far face key point; and apply makeup to the target image according to the face mask template, each far face key point after adjustment and the makeup material.

[0094] For the specific definition of the virtual makeup device, please refer to the definition of the method above, which will not be repeated here. Each module in the above device can be implemented in whole or in part by software, hardware and their combination. The above modules can be embedded in or independent of the processor in the terminal device in hardware form, or stored in the memory of the terminal device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0095] Please refer to Fig. 9 , Fig. 9 It shows a structural block diagram of a terminal device provided in an embodiment of the present application. The terminal device 90 may be a computer device. The terminal device 90 in the present application may include one or more of the following components: a processor 92, a memory 94, and one or more applications, wherein the one or more applications may be stored in the memory 94 and configured to be executed by one or more processors 92, and the one or more applications are configured to execute the method described in the above-mentioned virtual makeup method embodiment.

[0096] The processor 92 may include one or more processing cores. The processor 92 uses various interfaces and lines to connect the various parts of the entire terminal device 90, and executes various functions and processes data of the terminal device 90 by running or executing instructions, programs, code sets or instruction sets stored in the memory 94, and calling data stored in the memory 94. Optionally, the processor 92 can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor 92 can integrate one or more combinations of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 92, but may be implemented separately through a communication chip.

[0097] The memory 94 may include a random access memory (RAM) or a read-only memory (ROM). The memory 94 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 94 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data created by the terminal device 90 during use, etc.

[0098] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the terminal device to which the scheme of the present application is applied. The specific terminal device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0099] In summary, the terminal device provided in the embodiment of the present application is used to implement the corresponding virtual makeup method in the aforementioned method embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0100] See also Fig.10 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable storage medium 1000 stores program codes, which can be called by a processor to execute the method described in the above virtual makeup method embodiment.

[0101] The computer readable storage medium 1000 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 1000 includes a non-transitory computer-readable storage medium. The computer readable storage medium 1000 has storage space for program code 1002 that performs any method step in the above method. These program codes can be read from or written to one or more computer program products. The program code 1002 can be compressed, for example, in an appropriate form.

[0102] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0103] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A virtual makeup method, characterized in that: The method comprises: Get a target image containing a face; Determining a face rotation angle and a plurality of face key points on the target image; wherein the face rotation angle refers to an angle formed in the horizontal direction when the face is deflected based on a frontal standard posture; Adjusting each of the facial key points based on the facial rotation angle; Applying makeup to the target image based on the adjusted facial key points and makeup materials; The adjusting each of the facial key points based on the facial rotation angle includes: Determine the starting point vector and the ending point vector of the vertical median axis of the face according to the target image; Calculate the vector difference of the vertical median axis of the face according to the starting point vector and the end point vector; Adjusting each of the facial key points according to the starting point vector, the vector difference and the facial rotation angle; The facial key points include near-side facial key points, which are facial key points on the side of the face close to the frontal standard posture when the face is deflected; and adjusting each of the facial key points according to the starting point vector, the vertical mid-axis vector difference of the face, and the face rotation angle, comprises: Correcting the vector difference based on the quantity value of the near side key point; Calculate the offset vector of each of the near-side facial key points according to the starting point vector, the corrected vector difference, each of the near-side facial key points and the face rotation angle; Determine each new near-side facial key point according to each of the near-side facial key points and the offset vector corresponding to each of the near-side facial key points; Updating each of the near side facial key points to a corresponding new near side facial key point; The step of calculating the offset vector of each of the near-side facial key points according to the starting point vector, the corrected vector difference, each of the near-side facial key points and the face rotation angle comprises: Calculate the sum of the vectors of the starting point vector and the corrected vector difference, and record it as the first vector; Determine each proximal face key point vector based on each proximal face key point; Calculate the vector difference between the first vector and each near-side facial key point vector, and record them as second vectors; An offset vector of each of the near-side facial key points is calculated according to a first prediction vector coefficient, the face rotation angle and each of the second vectors, wherein the prediction vector coefficient is related to a fine-tuning coefficient of the facial key point.

2. The method according to claim 1, characterized in that The facial key points also include far-side facial key points, which refer to facial key points on the side of the face that are away from the front standard posture when the face is deflected; adjusting each of the facial key points according to the starting point vector, the vector difference and the face rotation angle includes: Calculate the offset vector of each of the far-side facial key points according to the starting point vector, the corrected vector difference, each of the far-side facial key points and the face rotation angle; Determine each new far-side facial key point according to the starting point vector and the offset vector of each far-side facial key point; Each of the far-side facial key points is updated to a corresponding new far-side facial key point.

3. The method according to claim 2, characterized in that The step of calculating the offset vector of each of the far-side face key points according to the starting point vector, the corrected vector difference, each of the far-side face key points and the face rotation angle comprises: Determine each far-side facial key point vector based on each far-side facial key point; Calculate the vector difference between each far-side facial key point vector and the first vector, and record them as third vectors; Calculate each fourth vector according to the second prediction vector coefficient, the face rotation angle and each third vector; The vector sum of the corrected vector difference and each of the fourth vectors is calculated and recorded as an offset vector of each of the far-side facial key points.

4. The method according to any one of claims 2 to 3, characterized in that: Applying makeup to the target image based on each adjusted facial key point and makeup material, including: When the facial key point is a far-side facial key point, a facial mask template is drawn according to each facial key point before adjustment; The target image is made up according to the face mask template, each adjusted far-side face key point and the makeup material.

5. A virtual makeup device, characterized in that: The device comprises: A template image module is used to obtain a target image containing a face; A key point and rotation angle determination module, used to determine the face rotation angle and multiple face key points based on the target image; wherein the face rotation angle refers to the angle formed in the horizontal direction when the face is deflected based on the front standard posture; A key point adjustment module, used for adjusting each of the key points of the face based on the face rotation angle; A makeup module, used for applying makeup to the target image based on the adjusted key points of each face and the makeup material; The adjusting each of the facial key points based on the facial rotation angle includes: Determine the starting point vector and the ending point vector of the vertical median axis of the face according to the target image; Calculate the vector difference of the vertical median axis of the face according to the starting point vector and the end point vector; Adjusting each of the facial key points according to the starting point vector, the vector difference and the facial rotation angle; The facial key points include near-side facial key points, which are facial key points on the side of the face close to the frontal standard posture when the face is deflected; and adjusting each of the facial key points according to the starting point vector, the vertical mid-axis vector difference of the face, and the face rotation angle, comprises: Correcting the vector difference based on the quantity value of the near side key point; Calculate the offset vector of each of the near-side facial key points according to the starting point vector, the corrected vector difference, each of the near-side facial key points and the face rotation angle; Determine each new near-side facial key point according to each of the near-side facial key points and the offset vector corresponding to each of the near-side facial key points; Updating each of the near side facial key points to a corresponding new near side facial key point; The step of calculating the offset vector of each of the near-side facial key points according to the starting point vector, the corrected vector difference, each of the near-side facial key points and the face rotation angle comprises: Calculate the sum of the vectors of the starting point vector and the corrected vector difference, and record it as the first vector; Determine each proximal face key point vector based on each proximal face key point; Calculate the vector difference between the first vector and each near-side facial key point vector, and record them as second vectors; An offset vector of each of the near-side facial key points is calculated according to a first prediction vector coefficient, the face rotation angle and each of the second vectors, wherein the prediction vector coefficient is related to a fine-tuning coefficient of the facial key point.

6. A terminal device, characterized in that: include: Memory; One or more processors coupled to the memory; One or more applications, wherein the one or more applications are stored in a memory and configured to be executed by one or more processors, and the one or more applications are configured to execute the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 4.

Images