Face key point prediction method, app, terminal device and storage medium
By employing a lightweight neural network on a mobile terminal and combining it with a coarse key point coordinate set and the image to be predicted, the problem of limited computing performance on mobile terminals is solved, achieving high-precision prediction of key points in the facial feature region and improving the effect of makeup placement.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- FOSHAN HUYA HUXIN TECH CO LTD
- Filing Date
- 2023-02-17
- Publication Date
- 2026-05-12
AI Technical Summary
Mobile devices have limited computing power, making it difficult to improve the accuracy of predicting key points in specific facial features while reducing computational load, resulting in poor makeup placement.
A lightweight neural network is used in conjunction with the fusion processing of a coarse keypoint coordinate set and the image to be predicted. Fine keypoint prediction is then performed through the first lightweight neural network, which reduces the amount of computation and improves the prediction accuracy.
High-precision prediction of facial features on mobile devices was achieved, reducing computational load and improving the stability of makeup placement and user experience.
Smart Images
Figure CN116152896B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of face recognition, and in particular to a method for predicting facial landmarks, an APP, a terminal device, and a computer-readable storage medium. Background Technology
[0002] Facial landmark prediction refers to predicting the location points of specific parts of a face. By using these location points, the position or size of the specific part can be determined. Facial landmark prediction can be applied to a variety of scenarios. Taking video makeup placement as an example, video makeup placement refers to placing various makeup elements on specific parts of a face that appear in a video, such as placing specific virtual makeup or virtual decorations on specific parts of the face, so that the face can show the effect that the user needs in the video.
[0003] Currently, the demand for makeup placement on specific facial features is mainly found in mobile terminals. Mobile terminals achieve video makeup placement by predicting key points of specific facial features. However, due to the limitations of mobile terminal computing performance, it is difficult to improve the accuracy of key point prediction for specific facial features while adapting to the computing performance of mobile terminals. Summary of the Invention
[0004] To address the aforementioned technical problems, this application provides a facial landmark prediction method, an app, a terminal device, and a computer-readable storage medium. The technical solution is as follows:
[0005] According to a first aspect of this application, a method for predicting facial landmarks is provided, the method comprising:
[0006] Obtain a rough key point coordinate set of the face region in the image frame, and locate the target facial feature region in the face region based on the rough key point coordinate set to obtain the image to be predicted containing the target facial feature region. The rough key point coordinate set is used to characterize the facial structure information.
[0007] The coarse key point coordinate set and the image to be predicted are input into a first lightweight neural network, and the first lightweight neural network is used to perform fusion processing to obtain a fine key point coordinate set, which includes the coordinates of several target facial features.
[0008] By merging the coordinates of each coarse keypoint and each fine keypoint, the keypoint prediction result is obtained.
[0009] According to a second aspect of this application, an APP is provided, which deploys a first lightweight neural network as described in the first aspect, and also deploys a second lightweight neural network.
[0010] The second lightweight neural network is used to take an image frame as input data and output a coarse key point coordinate set of the face region in the image frame. The APP is used to obtain the coarse key point coordinate set output by the second lightweight neural network, and locate the target facial feature region in the face region based on the coarse key point coordinate set to obtain a prediction image containing the target facial feature region. The coarse key point coordinate set is used to characterize facial structure information. The APP is also used to input the coarse key point coordinate set and the prediction image into the first lightweight neural network.
[0011] The first lightweight neural network is used to perform fusion processing to obtain a fine key point coordinate set, which includes the coordinates of several target facial features.
[0012] The app is also used to merge the coordinates of each coarse key point and the coordinates of each fine key point to obtain the key point prediction result.
[0013] The technical solution provided in this application enables fine key point prediction of the target facial features region on a mobile terminal using a lightweight neural network, significantly reducing the computational load of the mobile terminal. While reducing the computational load, to ensure prediction accuracy, the lightweight neural network combines a coarse key point coordinate set of the face region with the image to be predicted containing the target facial features region to perform fine key point prediction of the target facial features region. The lightweight neural network performs fusion processing of the coarse key point coordinate set and the image to be predicted. The coarse key point coordinate set of the face region can provide structural information of the face, assisting the lightweight neural network in prediction. The lightweight neural network can make full use of the obtained coarse key point coordinates, avoid computational waste and reduce additional computational load, thereby improving prediction accuracy while further reducing the computational load of the mobile terminal. It can improve the key point prediction accuracy of the target facial features region in the face while adapting to the computing performance of the mobile terminal.
[0014] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings.
[0016] Figure 1 This is a schematic diagram illustrating facial landmark prediction using related technologies;
[0017] Figure 2This is a video illustration illustrating the placement of makeup elements using related technologies;
[0018] Figure 3 This is a flowchart illustrating a facial landmark prediction method according to an embodiment of this application;
[0019] Figure 4 This is a schematic diagram of eye region orientation correction according to an embodiment of this application;
[0020] Figure 5 This is a schematic diagram of the eye region mirrored in one embodiment of this application;
[0021] Figure 6 This is a schematic diagram of an expanded eye region image according to an embodiment of this application;
[0022] Figure 7 This is a schematic diagram of a first lightweight neural network structure according to an embodiment of this application;
[0023] Figure 8 This is a flowchart illustrating a facial landmark prediction method according to another embodiment of this application;
[0024] Figure 9 This is a flowchart illustrating a facial landmark prediction method according to another embodiment of this application;
[0025] Figure 10 This is a schematic diagram of face segmentation information according to an embodiment of this application;
[0026] Figure 11 This is a schematic diagram of the APP structure according to an embodiment of this application;
[0027] Figure 12 This is a schematic diagram of the structure of a terminal device according to an embodiment of this application. Detailed Implementation
[0028] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art should fall within the scope of protection of this application.
[0029] Please see Figure 1 First, let's introduce facial landmark prediction. Facial landmark prediction refers to predicting the location points of specific parts of a face. By using these location points, we can determine the position or size of that specific part. Facial landmark prediction can be applied to a variety of scenarios.
[0030] Please see Figure 2Taking the application of facial landmark prediction to video makeup placement as an example, video makeup placement refers to placing various makeup elements on specific parts of the face in a video, such as placing specific virtual makeup or virtual decorations on specific parts of the face, so that the face can show the effect required by the user in the video.
[0031] Currently, when applying makeup placement to specific facial features in live streaming scenarios, there is a real-time requirement; users expect to see the makeup placement result immediately. Therefore, the demand for makeup placement to specific facial features primarily exists on mobile devices, where it achieves video makeup placement through key point prediction of specific facial features. However, due to the limitations of mobile device computing power, it is difficult to improve the accuracy of key point prediction for specific facial features while adapting to the limitations of mobile device computing power. Under the influence of the relatively low key point prediction accuracy of specific facial features on mobile devices, various problems arise in makeup placement to specific facial features, such as… Figure 2 The problems shown include: makeup not adhering properly when trembling eyebrows, eye sockets appearing erratic when concealing, makeup not adhering properly when concealing, and makeup being unstable and sticking to the hands when concealing.
[0032] It should be noted that the above description of the application of facial landmark prediction to makeup placement in live video is merely an illustrative example. In practical applications, in addition to the above-mentioned scenarios, the embodiments of this application can also be applied to other scenarios, and there are no specific limitations on these.
[0033] It should be noted that the above description of the application of facial landmark prediction to mobile terminals is only an illustrative example. In practical applications, in addition to mobile terminals, the embodiments of this application can also be applied to other terminals such as PCs, and there are no specific limitations on this.
[0034] To address the aforementioned problems, one embodiment of this application provides a facial landmark prediction method that can improve the prediction accuracy of landmarks in the target facial feature region while adapting to the computing performance of mobile terminals. For example... Figure 3 As shown, the method includes the following steps:
[0035] S301. Obtain the rough key point coordinate set of the face region in the image frame, and locate the target facial feature region in the face region based on the rough key point coordinate set to obtain the image to be predicted containing the target facial feature region. The rough key point coordinate set is used to characterize the face structure information.
[0036] As an example, an image frame can be input into a second lightweight neural network, and the second lightweight neural network can process and output a set of coarse key point coordinates of the face region in the image frame. It is worth noting that the network used to obtain the coarse key point coordinate set is only an example, and other networks may be used in actual applications. No specific limitation is made in this regard.
[0037] Please see Figure 1 As shown, as an example, the rough keypoint coordinate set can be the coordinates of 106 points on a face, or other coordinates, without limitation.
[0038] As an example, the target facial feature can be either the eyes or the mouth; there is no limitation on this.
[0039] As an example, the orientation of the target facial features can be corrected to vertical based on a coarse keypoint coordinate set, resulting in an image to be detected containing the vertical target facial features. The fine keypoint coordinates can then be the coordinates of the vertical target facial features. See also... Figure 4 As shown below, examples illustrate the above correction methods:
[0040] Taking the eyes as the target facial feature as an example, one way to locate the target facial feature region in a face is to obtain a bounding map of the eye region using the maximum and minimum coordinate values from the coarse keypoint coordinate set of the eye region. The angle between the line connecting the corner points P1(x1,y1) and P2(x2,y2) of the eye and the horizontal axis can be calculated using the coarse keypoint coordinate set of the eye region in this bounding map. The calculation formula is as follows:
[0041]
[0042] The rotation angle t can be obtained by formula (1), and then the block diagram can be rotated according to the rotation angle t to obtain a block diagram in which the eye region is oriented vertically, that is, the corner of the eye is kept horizontal. This results in the image to be detected containing the vertical eye region, which can ensure that the orientation of the eye region in the image to be detected is vertical, which can greatly reduce the difficulty of the input image (reduce the distribution) and thus improve the prediction accuracy.
[0043] It is worth noting that the above description of the orientation correction method and calculation formula for the target facial features area is only an illustrative example. Other correction methods and calculation formulas are not excluded in actual applications, and no specific limitations are made here.
[0044] Please see Figure 5 As shown, taking the eyes as an example, the eye region of one eye can be mirrored to obtain the image to be detected. For example, the eye region corresponding to the right eye can be mirrored to become the left eye, or the eye region corresponding to the left eye can be mirrored to become the right eye. Unifying the eye regions into one form can reduce the size of the user software package and improve the training and prediction efficiency of the network.
[0045] It is worth noting that the above description of the uniform form of the eye area is only an example, and other methods are not excluded in actual application, but no specific limitation is made.
[0046] Please see Figure 6 As shown, taking the eyes as an example of the target facial feature, in the eyebrow-shaking scenario, the image to be detected is obtained according to a first expansion ratio; in the non-eyebrow-shaking scenario, the image to be detected is obtained according to a second expansion ratio. The first expansion ratio is greater than the second expansion ratio. Eyebrow shaking refers to the shaking of the eyebrows relative to the eyes in the eye region. The expansion ratio represents the ratio of the image to be detected to the image of the eye region. This allows for adaptive adjustment of the expansion ratio in both eyebrow-shaking and non-eyebrow-shaking scenarios, ensuring the accuracy of keypoint prediction as much as possible in both scenarios. See [link / reference]. Figure 6 The following example illustrates how to adjust the aforementioned expansion ratio:
[0047] First, define an outer bounding box for the eye area and enclose the eye area within this bounding box. Then, determine whether it is a trembling eyebrow scenario: if it is not a trembling eyebrow scenario, expand the outer bounding box according to the second expansion ratio to obtain the expanded box for the non-trembling eyebrow scenario; if it is a trembling eyebrow scenario, expand the outer bounding box according to the first expansion ratio, which is greater than the second expansion ratio, to obtain the upward expansion box for the trembling eyebrow scenario, so as to surround the area of the eyes and eyebrows as much as possible.
[0048] It should be noted that the above description of the methods for adjusting the external expansion ratio is only an example. Other adjustment methods are not excluded in actual applications, and no specific limitations are made for them.
[0049] S302. Input the coarse key point coordinate set and the image to be predicted into the first lightweight neural network, and use the first lightweight neural network to perform fusion processing to obtain the fine key point coordinate set, which includes the coordinates of several target facial features.
[0050] As an example, the first lightweight neural network can be trained on a sample image with an added occlusion patch using a preset probability as input. The occlusion patch represents a patch obscuring the facial features of a target. For instance, an occlusion patch can be added to the image with a preset probability of p = 0.5 to obtain a sample image. This sample image can then be used to train the first lightweight neural network, without changing the training label data. This enables the first lightweight neural network to additionally learn to predict invisible key points of the occluded portion based on visible points. The preset probability is not limited to 0.5; other values are not specified.
[0051] As an example, the first lightweight neural network is trained based on a distance loss function. This distance loss function characterizes the relationship between the distances between keypoints in the sample image and the distances between predicted keypoints, improving the first lightweight neural network's structural perception of the predicted keypoints and making the prediction results more reliable. The distance loss function is illustrated below:
[0052] As an example, the distance loss function could be the adjacent edge distance loss function, Loss. edge The adjacent edge list L of all points in the target facial region can be represented by the following formula:
[0053]
[0054] Where N is the total number of adjacent edge lists L of the key point. Let represent the edge connecting keypoints on the sample image of the i-th edge. This represents the edge connected to the predicted keypoint of the i-th edge, and the dis() function represents the edge length distance function.
[0055] As an example, the distance loss function can also be the centroid distance loss function (Loss). center It can be expressed by the following formula:
[0056]
[0057]
[0058]
[0059] Where M is the total number of key points. This represents the coordinates of the keypoints in the i-th sample image. Let C represent the coordinates of the i-th predicted keypoint. gt C represents the center point of all keypoints on all sample images. pred This represents the center point of all predicted key points.
[0060] It is worth noting that the above introduction to the design of distance loss functions is only an illustrative example, and the design methods are not excluded in practical applications, so no specific limitations are made.
[0061] It is worth noting that the above description of the first lightweight neural network training method is only an illustrative example. Other training methods are not excluded in practical applications, and no specific limitations are made here.
[0062] As an example, the first lightweight neural network could be ShuffleNet v2, or any other network; there are no restrictions on this.
[0063] Please see Figure 7 Taking ShuffleNet v2 as an example, the structure of the first lightweight neural network will be described exemplarily:
[0064] The image to be predicted and the coarse keypoint coordinate set are input into ShuffleNet v2. The image features of the image to be predicted are extracted and fused with the coarse keypoint coordinate set. In linear layer 1, the image features and the features of the coarse keypoint coordinates are concatenated to obtain the comprehensive features. Through the processing of linear layer 2, the fine keypoint coordinate set is obtained, and the loss is calculated.
[0065] It is worth noting that the above description of the structure of the first lightweight neural network is merely an illustrative example, and the structure may vary in practical applications; therefore, no specific limitations are imposed.
[0066] S303. Merge the coordinates of each coarse key point and each fine key point to obtain the key point prediction result.
[0067] As an example, the keypoint prediction result can be processed by a sliding window smoothing algorithm, where the smoothing parameter of the sliding window smoothing algorithm is greater than or equal to a preset value; the image to be predicted can be smoothed by a Gaussian filter operator, which can minimize the jitter in the keypoint prediction result and obtain a stable and non-lag-prone effect. The sliding window smoothing algorithm and the Gaussian filter operator are illustrated below:
[0068] As an example, the sliding window smoothing algorithm can be expressed by the following formula:
[0069] P = γ * P i +(1-γ)*P i-1 (6)
[0070] Among them, P i P is the keypoint prediction result for the current frame. i-1 The key point prediction result is from the previous frame. γ is a smoothing parameter. The larger the value of γ, the lower the smoothness. You can set γ to be greater than or equal to a preset value to make the smoothness as low as possible. For example, you can set γ to 0.99.
[0071] As an example, the Gaussian filter operator can be a 3x3 operator, i.e.:
[0072]
[0073] It is worth noting that the above introduction to the sliding window smoothing algorithm and the design method of the Gaussian filter operator is only an illustrative example. Other design methods are not excluded in practical applications, and no specific limitations are made here.
[0074] The technical solution provided in this application enables fine key point prediction of the target facial features region on a mobile terminal using a lightweight neural network, significantly reducing the computational load of the mobile terminal. While reducing the computational load, to ensure prediction accuracy, the lightweight neural network combines a coarse key point coordinate set of the face region with the image to be predicted containing the target facial features region to perform fine key point prediction of the target facial features region. The lightweight neural network performs fusion processing of the coarse key point coordinate set and the image to be predicted. The coarse key point coordinate set of the face region can provide structural information of the face, assisting the lightweight neural network in prediction. The lightweight neural network can make full use of the obtained coarse key point coordinates, avoid computational waste and reduce additional computational load, thereby improving prediction accuracy while further reducing the computational load of the mobile terminal. It can improve the key point prediction accuracy of the target facial features region in the face while adapting to the computing performance of the mobile terminal.
[0075] Please see Figure 8 To ensure that the orientation of the face region in the image frame input to the network is vertical, thereby reducing data distribution differences and improving the prediction accuracy of the network, this application provides another embodiment of a face key point prediction method, such as... Figure 8 As shown, the method includes the following steps:
[0076] S801. Correct the orientation of the face region in the image frame to vertical;
[0077] S802. Obtain the rough key point coordinate set of the face region in the image frame, and locate the target facial feature region in the face region based on the rough key point coordinate set to obtain the image to be predicted containing the target facial feature region. The rough key point coordinate set is used to characterize the face structure information.
[0078] S803. Input the coarse key point coordinate set and the image to be predicted into the first lightweight neural network, and use the first lightweight neural network to perform fusion processing to obtain the fine key point coordinate set, which includes the coordinates of several target facial features.
[0079] In this embodiment, S802 and S803 are the same as those described above. Figure 3 S301 and S302 in the illustrated embodiment are similar, and will not be described in detail here.
[0080] S804. Merge the coordinates of each coarse key point and each fine key point, and restore the orientation of the merged face region to the orientation of the face region in the image frame to obtain the key point prediction result.
[0081] In this embodiment, S804 is the same as the aforementioned Figure 3Compared to S303 in the illustrated embodiment, the process is similar except that the orientation of the merged face region is restored to the orientation of the face region in the image frame. The details will not be repeated here.
[0082] Please see Figure 9 To avoid performing operations such as virtual makeup placement on the target facial features when they are obscured, thereby improving the user experience, this application provides another embodiment of a facial landmark prediction method, such as... Figure 9 As shown, the method includes the following steps:
[0083] S901. Obtain the rough key point coordinate set of the face region in the image frame, and locate the target facial feature region in the face region based on the rough key point coordinate set to obtain the image to be predicted containing the target facial feature region. The rough key point coordinate set is used to characterize the face structure information.
[0084] S902. Input the coarse key point coordinate set and the image to be predicted into the first lightweight neural network, and use the first lightweight neural network to perform fusion processing to obtain the fine key point coordinate set, which includes the coordinates of several target facial features.
[0085] S903. Merge the coordinates of each coarse key point and each fine key point to obtain the key point prediction result.
[0086] In this embodiment, S801, S802, and S803 are the same as those described above. Figure 3 S301, S302 and S303 in the illustrated embodiment are similar, and will not be described in detail here.
[0087] S904. Using a segmentation algorithm, determine the face pixels and occlusion pixels in the image frame to obtain face segmentation information. This face segmentation information can be used to combine with the key point prediction results to apply virtual makeup to the target facial features. An example of the segmentation method is given below:
[0088] Please see Figure 10 As an example, a segmentation algorithm can be used to obtain the segmentation result G of the face region in the image frame, where G(occlusion pixels) = 0 and G(face pixels) = 1. Then, when placing virtual makeup on the target facial features, the placement result is R = G * makeup + original image. By using the occlusion segmentation result, the makeup will not be placed on the occlusion but will only appear in the actual position of the face, improving the user experience.
[0089] It is worth noting that the above description of the segmentation methods is only an illustrative example, and other segmentation methods are not excluded in practical applications, but no specific limitation is made.
[0090] Corresponding to the above method embodiments, this application also provides an APP, see [link to APP]. Figure 11 As shown, the APP is equipped with a first lightweight neural network 1101 that implements the facial key point prediction method described in any of the embodiments above, and a second lightweight neural network 1102.
[0091] The second lightweight neural network 1102 is used to take an image frame as input data and output a coarse key point coordinate set of the face region in the image frame. The APP is used to obtain the coarse key point coordinate set output by the second lightweight neural network 1102, and locate the target facial feature region in the face region based on the coarse key point coordinate set to obtain a prediction image containing the target facial feature region. The coarse key point coordinate set is used to characterize facial structure information. The APP is also used to input the coarse key point coordinate set and the prediction image into the first lightweight neural network 1101.
[0092] The first lightweight neural network 1101 is used to perform fusion processing to obtain a fine key point coordinate set, which includes the coordinates of several target facial feature regions.
[0093] The app is also used to merge the coordinates of each coarse key point and the coordinates of each fine key point to obtain the key point prediction result.
[0094] This application also provides a terminal device, such as Figure 12 As shown, the terminal device includes:
[0095] Processor 1201;
[0096] Memory 1202 is used to store processor-executable instructions;
[0097] The processor 1201 is configured to implement the facial landmark prediction method described in any of the embodiments above.
[0098] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the facial landmark prediction method described in any of the embodiments above.
[0099] The above description is only a specific embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A method for predicting facial landmarks, characterized in that, include: Obtain a rough key point coordinate set of the face region in the image frame, and locate the target facial feature region in the face region based on the rough key point coordinate set to obtain the image to be predicted containing the target facial feature region. The rough key point coordinate set is used to characterize the facial structure information. The step of obtaining the image to be predicted containing the target facial features region includes at least: obtaining the image to be detected containing the target facial features region based on the coarse key point coordinate set; wherein, the target facial features include the eyes, and in the eyebrow shaking scenario, the image to be detected is obtained according to a first expansion ratio; in the non-eyebrow shaking scenario, the image to be detected is obtained according to a second expansion ratio; wherein, the first expansion ratio is greater than the second expansion ratio, eyebrow shaking refers to the shaking of the eyebrows relative to the eyes in the eye region, and the expansion ratio represents the ratio of the image to be detected to the image of the eye region; The coarse key point coordinate set and the image to be predicted are input into a first lightweight neural network, and the first lightweight neural network is used to perform fusion processing to obtain a fine key point coordinate set, which includes the coordinates of several target facial features. By merging the coordinates of each coarse key point and each fine key point, a key point prediction result is obtained, wherein the key point prediction result is used to place virtual makeup on the target facial features area.
2. The method according to claim 1, characterized in that, Before obtaining the rough set of key point coordinates of the face region in the image frame, the method further includes: Correct the orientation of the face region in the image frame to vertical; The steps to obtain keypoint prediction results include: The orientation of the merged face region is restored to the orientation of the face region in the image frame to obtain the key point prediction result.
3. The method according to claim 2, characterized in that, The steps to obtain the image to be predicted, which contains the target facial features, include: Based on the coarse key point coordinate set, the orientation of the target facial features region is corrected to vertical, resulting in a detection image containing the vertical target facial features region. The fine key point coordinates are the coordinates of the vertical target facial features region.
4. The method according to claim 3, characterized in that, The target facial features include the eyes. The step of obtaining a detection image containing the vertical target facial feature region includes: The eye area of one of the eyes is mirrored to obtain the image to be detected.
5. The method according to claim 1, characterized in that, The first lightweight neural network is generated by training on sample images with images whose occlusion color blocks have been increased with a preset probability as input, where the occlusion color blocks refer to color blocks that occlude the facial features of the target.
6. The method according to claim 5, characterized in that, The first lightweight neural network is trained based on a distance loss function, which represents the relationship between the distance between key points on the sample image and the distance between the predicted key points.
7. The method according to claim 1, characterized in that, The method further includes: The face pixels and occlusion pixels in the image frame are determined by the segmentation algorithm to obtain face segmentation information. The face segmentation information can be used to combine the key point prediction results to place virtual makeup on the target facial features.
8. The method according to claim 1, characterized in that, The key point prediction result is processed by a sliding window smoothing algorithm, and the smoothing parameter of the sliding window smoothing algorithm is greater than or equal to a preset value; the image to be predicted is smoothed by a Gaussian filter operator.
9. An APP, characterized in that, The APP is equipped with a first lightweight neural network as described in any one of claims 1 to 8, and also with a second lightweight neural network. The second lightweight neural network is used to take an image frame as input data and output a set of coarse key point coordinates of the face region in the image frame. The APP is used to obtain the set of coarse key point coordinates output by the second lightweight neural network and locate the target facial feature region in the face region based on the set of coarse key point coordinates to obtain an image to be predicted containing the target facial feature region. The set of coarse key point coordinates is used to characterize facial structure information. The coarse key point coordinate set and the image to be predicted are input into a first lightweight neural network. The step of obtaining the image to be predicted containing the target facial features region includes at least: obtaining the image to be detected containing the target facial features region based on the coarse key point coordinate set; wherein, the target facial features include the eyes, and in the eyebrow shaking scenario, the image to be detected is obtained according to a first expansion ratio; in the non-eyebrow shaking scenario, the image to be detected is obtained according to a second expansion ratio; wherein, the first expansion ratio is greater than the second expansion ratio, eyebrow shaking refers to the shaking of the eyebrows relative to the eyes in the eye region, and the expansion ratio represents the ratio of the image to be detected to the image of the eye region; The first lightweight neural network is used to perform fusion processing to obtain a fine key point coordinate set, which includes the coordinates of several target facial features. The app is also used to merge the coordinates of each coarse key point and the coordinates of each fine key point to obtain the key point prediction result, wherein the key point prediction result is used to place virtual makeup on the target facial features area.