A method, device and equipment for processing key points of human face

Abstract

The embodiments of the present specification disclose a method, device and equipment for processing facial key points, including: inputting a facial image of a current frame into a pre-trained facial key point model, predicting basic facial key points of multiple preset areas, and predicting supplementary facial key points of a specified area in the multiple preset areas; determining a basic facial key point set and a supplementary facial key point set updated according to facial images of frames before the current frame, the basic facial key point set and the supplementary facial key point set being used for image processing; performing facial motion detection based on the basic facial key points and the supplementary facial key points predicted for the facial image of the previous frame and the facial image of the current frame respectively; based on the result of the facial motion detection, if it is determined that facial motion has occurred in the specified area, updating the supplementary facial key point set based on the supplementary facial key points predicted for the facial image of the current frame, so as to update the image processing accordingly.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device and equipment for processing key points of a face. Background Art

[0002] Facial key point detection is an important link in the process of facial image processing. After completing facial key point detection, the input face detection frame diagram is processed to locate the key facial feature points on the face detection frame diagram, including eyes, nose tip, mouth corners, and contour points of various parts of the face. It is widely used in technical fields such as face recognition, beauty, expression analysis, and face tracking.

[0003] In the existing key point detection schemes, most of them have the problem of insufficient key points under the premise of limited model configuration, which results in that the line segments connected by the key points cannot accurately fit the various parts of the face, affecting the subsequent image processing effect. At the same time, since the key points located by the algorithm will have a certain error, when processing multiple consecutive frames of face images, random errors often manifest as jitters in the output key points, resulting in unstable subsequent image processing effects.

[0004] Based on this, a more accurate key point prediction method is now needed for multiple consecutive frames of face images. Summary of the invention

[0005] One or more embodiments of this specification provide a method, device and apparatus for processing key points of a face, which are used to solve the following technical problems:

[0006] In the existing key point detection schemes, most of them have the problem of insufficient key points under the premise of limited model configuration, which results in that the line segments connected by the key points cannot accurately fit the various parts of the face, affecting the subsequent image processing effect. At the same time, since the key points located by the algorithm will have a certain error, when processing multiple consecutive frames of face images, random errors often manifest as jitters in the output key points, resulting in unstable subsequent image processing effects.

[0007] One or more embodiments of this specification adopt the following technical solutions:

[0008] One or more embodiments of this specification provide a method for processing key points of a face, wherein multiple frames of face images are continuously collected for the same face, and the method includes:

[0009] Inputting the face image of the current frame into a pre-trained face key point model, predicting the basic face key points of multiple preset areas, and predicting the face supplementary key points of a designated area among the multiple preset areas;

[0010] Determine a basic face key point set and a supplementary face key point set obtained by updating face images of frames before a current frame, wherein the basic face key point set and the supplementary face key point set are used for image processing;

[0011] Performing face motion detection based on the face basic key points and face supplementary key points predicted for the face image of the previous frame and the face image of the current frame respectively;

[0012] According to the result of the face motion detection, if it is determined that face motion has occurred in the designated area, the face supplementary key point set is updated according to the face supplementary key points predicted for the face image of the current frame, so as to update the image processing accordingly.

[0013] One or more embodiments of this specification provide a facial key point processing device, which continuously collects multiple frames of facial images for the same face, and the device includes:

[0014] A prediction unit, inputting a face image of a current frame into a pre-trained face key point model, predicting basic face key points of a plurality of preset areas, and predicting supplementary face key points of a designated area among the plurality of preset areas;

[0015] An updating unit, which determines a basic face key point set and a supplementary face key point set obtained by updating face images of frames before a current frame, wherein the basic face key point set and the supplementary face key point set are used for image processing;

[0016] A detection unit performs face motion detection based on the face basic key points and face supplementary key points predicted from the face image of the previous frame and the face image of the current frame respectively;

[0017] The processing unit updates the face supplement key point set according to the face supplement key points predicted for the face image of the current frame if it is determined that face motion has occurred in the designated area based on the result of the face motion detection, so as to update the image processing accordingly.

[0018] One or more embodiments of this specification provide a facial key point processing device, including:

[0019] at least one processor; and,

[0020] a memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0022] Inputting the face image of the current frame into a pre-trained face key point model, predicting the basic face key points of multiple preset areas, and predicting the face supplementary key points of a designated area among the multiple preset areas;

[0023] Determine a basic face key point set and a supplementary face key point set obtained by updating face images of frames before a current frame, wherein the basic face key point set and the supplementary face key point set are used for image processing;

[0024] Performing face motion detection based on the face basic key points and face supplementary key points predicted for the face image of the previous frame and the face image of the current frame respectively;

[0025] According to the result of the face motion detection, if it is determined that face motion has occurred in the designated area, the face supplementary key point set is updated according to the face supplementary key points predicted for the face image of the current frame, so as to update the image processing accordingly.

[0026] One or more embodiments of this specification provide a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to:

[0027] Inputting the face image of the current frame into a pre-trained face key point model, predicting the basic face key points of multiple preset areas, and predicting the face supplementary key points of a designated area among the multiple preset areas;

[0028] Determine a basic face key point set and a supplementary face key point set obtained by updating face images of frames before a current frame, wherein the basic face key point set and the supplementary face key point set are used for image processing;

[0029] Performing face motion detection based on the face basic key points and face supplementary key points predicted for the face image of the previous frame and the face image of the current frame respectively;

[0030] According to the result of the face motion detection, if it is determined that face motion has occurred in the designated area, the face supplementary key point set is updated according to the face supplementary key points predicted for the face image of the current frame, so as to update the image processing accordingly.

[0031] At least one of the above technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects: the embodiments of this specification predict the basic face key points and supplementary face key points of the face image of the current frame through the face key point model. The supplementary face key points can make up for the problem of insufficient number of basic face key points, making subsequent image processing more accurate. At the same time, the embodiments of this specification place the basic face key points in multiple preset areas in the face image, and place the supplementary face key points in the specified areas of multiple preset areas in the face image, and determine the basic face key point set and the supplementary face key point set in each preset area. At this time, the basic face key point set and the supplementary face key point set are both updated according to the face images of each frame before the current frame, which can obtain a lower delay while obtaining a better smoothing effect. In addition, in order to prevent facial motion (i.e., image jitter) in the facial image of the current frame, which would make subsequent image processing effects unstable, the embodiments of the present specification perform facial motion detection based on the basic facial key points and supplementary facial key points predicted for the facial image of the previous frame and the facial image of the current frame, respectively. If it is determined that facial motion has occurred in the designated area, the set of supplementary facial key points is updated based on the supplementary facial key points predicted for the facial image of the current frame. During subsequent image processing, the processing effect of the designated area can be made more stable. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art description. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. In the drawings:

[0033] Figure 1 A flowchart of a method for processing key points of a face provided in one or more embodiments of this specification;

[0034] Figure 2 A schematic diagram of a 106-point key point mouth definition structure provided for one or more embodiments of this specification;

[0035] Figure 3 A schematic diagram of a model training process provided for one or more embodiments of this specification;

[0036] Figure 4 A schematic diagram of the effects including newly added key points provided for one or more embodiments of this specification;

[0037] Figure 5 A schematic diagram of a key point detection process provided for one or more embodiments of this specification;

[0038] Figure 6 A schematic diagram of face area grouping provided for one or more embodiments of this specification;

[0039] Figure 7 A schematic diagram of a motion group key point associated with the entire face provided by one or more embodiments of this specification;

[0040] Figure 8 A schematic diagram of key points of eye movement groups associated with one or more embodiments of the present specification;

[0041] Fig. 9 A schematic diagram of key points of a motion group associated with a mouth provided for one or more embodiments of the present specification;

[0042] Fig.10 A schematic diagram of a flow chart of motion detection of a motion group provided for one or more embodiments of this specification;

[0043] Fig.11 A schematic diagram of a process for invisible point beautification optimization provided for one or more embodiments of this specification;

[0044] Fig.12 A schematic diagram of the adjacent relationship of the mouth area provided for one or more embodiments of this specification;

[0045] Fig.13 A schematic diagram of the structure of a facial key point processing device provided in one or more embodiments of this specification;

[0046] Fig.14 A schematic diagram of the structure of a facial key point processing device provided in one or more embodiments of this specification. DETAILED DESCRIPTION

[0047] The embodiments of this specification provide a method, device and equipment for processing key points of a face.

[0048] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0049] Figure 1A flowchart of a facial key point processing method provided for one or more embodiments of this specification, the process can be executed by a related image processing platform, the platform can be applied in different scenarios, such as face recognition, beauty, expression analysis, face tracking, to more accurately predict facial key points, and then perform corresponding processing on the face image. Certain input parameters or intermediate results in the process allow manual intervention and adjustment to help improve accuracy.

[0050] It should be noted that the embodiments of this specification may continuously capture multiple frames of facial images for the same face.

[0051] The method steps of the embodiment of this specification are as follows:

[0052] S102, inputting the face image of the current frame into a pre-trained face key point model, predicting basic face key points of multiple preset areas, and predicting supplementary face key points of a designated area among the multiple preset areas.

[0053] In the embodiments of this specification, the preset area can be an area preset in the face image, for example, the face image is divided into a mouth area, a nose area, an eye area, an eyebrow area and a cheek area, the eye area includes a left eye area and a right eye area, and the eyebrow area includes a left eyebrow area and a right eyebrow area. It should be noted that the embodiments of this specification are not limited to the above method when presetting the partitions, and can also be divided in other ways according to actual conditions.

[0054] The basic key points of the face can adopt the existing key point solution, for example, 68 key points of the face or 106 key points of the face. The model configuration of the embodiment of this specification is limited and is not suitable for the method of too many key points, for example, the existing method of 1000 dense key points.

[0055] The embodiments of this specification can distribute the basic key points of the face in each preset area; the supplementary key points of the face are for the specified areas in multiple preset areas. For example, the specified area is set as the mouth area, and 30 supplementary key points of the face can be added to the mouth area. In addition, the specified area in the embodiments of this specification can also be multiple, and the supplementary key points of the face can be set in different numbers according to different preset areas. These can be set according to actual needs. For example, the specified area is set as the mouth area and the nose area, and 30 supplementary key points of the face are added in the mouth area, and 25 supplementary key points of the face are added in the nose area.

[0056] The above-mentioned facial key point model can be trained by the following steps in the embodiment of this specification:

[0057] A first face training image set with basic face key points annotated in multiple preset areas is obtained, and a deep network model consisting of a backbone network and a basic prediction head is trained based on the first face training image set to obtain a basic key point model. Then, a second face training image set with supplementary face key points annotated in a specified area is obtained. Then, the trained network parameters in the basic key point model are fixed, and a supplementary prediction head is added to the basic key point model. The supplementary prediction head is trained based on the second face training image set, and the basic key point model trained by the supplementary prediction head is used as the face key point model.

[0058] It can be seen from the above training process that the facial key point model of the embodiment of the present specification can be used to identify the key points of a specified part of the face. For example, the facial key point model is used to identify the mouth so that the lip color can be changed for the user later. At this time, only a second face training image set with supplementary facial key points related to the mouth needs to be marked in the mouth area separately. The facial key point model obtained subsequently will be more accurate in identifying the mouth.

[0059] After the face image of the current frame is input into the pre-trained face key point model in the embodiment of the present specification, the face key point model is processed to predict whether the key points in the basic face key points and / or the supplementary face key points are visible or not, wherein the key points that are invisible may be blocked by foreign objects or self-occluded by the shooting angle, and cannot be displayed normally. Then, the area to be processed is determined in multiple preset areas, and the area to be processed can be preset. For example, for the lip color enhancement function, the area to be processed can be set as the mouth area, and the adjacent relationship between the key points to be processed in the area to be processed is determined according to the basic face key points and / or the supplementary face key points. The adjacent relationship can be determined according to the preset rules. The rule can be that the basic face key points and / or the supplementary face key points in the same preset area and preset range have an adjacent relationship, and the preset range can be set according to actual needs.

[0060] Then, according to the prediction results of the adjacent relationship and whether the key points are visible or not, the locally adjacent invisible key points to be processed are divided into the same set to be processed. In addition, if the number of key points to be processed contained in the set to be processed is greater than the set threshold, it is determined that there is occlusion on the face in the face image, and the set to be processed is used as the occlusion set. The set threshold here can be set according to the actual situation. For example, the face in the face image wears a mask, and the number of key points to be processed occupied by the mask area is determined. If the number is greater than the set threshold, it is determined that there is occlusion on the face in the face image. For another example, the face in the face image is partially occluded by fingers, and the number of key points to be processed occupied by the occluded area is determined. If the number is not greater than the set threshold, it cannot be determined that there is occlusion on the face in the face image.

[0061] Furthermore, the image processing in the embodiments of this specification includes performing face display effect processing on the face image.

[0062] The embodiments of this specification determine the connected domain surrounded by the key points to be processed in the occlusion set; when performing face display effect processing on the face image, the connected domain is excluded so that the face display effect processing is not performed in the connected domain. Since the connected domain is the part that is determined to be possibly occluded, this part may have an impact on image processing, so the embodiments of this specification need to exclude the connected domain before processing. For example, the face in the face image wears a mask, and it is determined that there is occlusion on the face in the face image. When determining the connected domain surrounded by the key points to be processed in the occlusion set, that is, the mask area, when performing face display effect processing on the face image, the mask area needs to be excluded.

[0063] S104, determining a basic face key point set and a supplementary face key point set obtained by updating face images of frames before the current frame, and the basic face key point set and the supplementary face key point set are used for image processing.

[0064] In the embodiment of the present specification, the basic face key points in the basic face key point set and the face supplementary face key points in the face supplementary face key point set can be determined based on the face images of each frame before the current frame. The basic face key point set and the face supplementary face key point set here can identify the basic face key points and face supplementary face key points currently output.

[0065] Furthermore, when determining the above-mentioned basic face key point set and supplementary face key point set, whether there is jitter in the face image of the current frame is not considered, that is, whether the face in the face image moves is not considered. If the face in the face image moves, the basic face key points and supplementary face key points of the face image of the current frame can be output.

[0066] It should be noted that the basic face key point set and the supplementary face key point set do not necessarily belong to the face image of the previous frame. If the face in the previous frame face image does not move, the basic face key point set and the supplementary face key point set may belong to the face image of the previous two frames.

[0067] The following steps are to perform face motion detection on the face image of the current frame.

[0068] S106, performing face motion detection according to the face basic key points and face supplementary key points predicted for the face image of the previous frame and the face image of the current frame respectively.

[0069] The embodiments of the present specification can calculate the distance parameters between at least some of the corresponding basic face key points predicted for the face image of the previous frame and the face image of the current frame, and / or the distance parameters between at least some of the corresponding supplementary face key points predicted for the face image of the previous frame and the face image of the current frame, wherein the distance parameters about the basic face key points can be the position deviation values ​​of the same basic face key points in the face image of the previous frame and the face image of the current frame, and the distance parameters about the supplementary face key points can be the position deviation values ​​of the same supplementary face key points in the face image of the previous frame and the face image of the current frame. For example, a basic face key point of the mouth is selected, and the distance parameter between the corresponding basic face key point predicted by the face image of the previous frame and the face image of the current frame is calculated, and / or a supplementary face key point of the mouth is selected, and the distance parameter between the corresponding supplementary face key point predicted by the face image of the previous frame and the face image of the current frame is calculated.

[0070] Finally, face motion detection is performed based on the calculated distance parameter and the set distance threshold.

[0071] It should be noted that the face motion detection involved in the embodiments of this specification can be a detection performed on different preset areas, that is, only the basic face key points of the preset area where face motion occurs can be updated. If there are supplementary face key points in the area, the supplementary face key points of the area can also be updated. The updating method is to output the basic face key points or supplementary face key points of the face image of the current frame.

[0072] In the embodiments of this specification, the distance threshold can be set as follows:

[0073] A low threshold and a high threshold are generated according to the size of a specified part in the same face. The size of the specified part can be set according to actual conditions, and the distance threshold proposed in the embodiments of this specification is a size related to the face in the current face image. For example, the interocular width distance in the eye spacing is selected, and the straight-line distance from the outer corner of the eye to the inner corner of the eye on the right eye or the left eye. The low threshold can be set to 1.5% of the interocular width distance, and the high threshold can be set to 3% of the interocular width distance.

[0074] At the same time, the distance threshold can be initially set to a low threshold. For several frames of face images before the current frame, if the number of face images that are determined to have undergone face motion is greater than the set threshold, the distance threshold is still set to a low threshold. Otherwise, the distance threshold is set to a high threshold. The set threshold can be set according to actual conditions and temporarily adjusted according to different scenes. In the above process, the threshold is low when there is continuous motion and is updated frequently, and the threshold is high when it is stationary to prevent serious jitter of the basic key points of the face and the supplementary key points of the face.

[0075] It should be noted that different distance thresholds can be set for different preset areas. For example, after the above judgment, the mouth area is set as a low threshold and the eye area is set as a high threshold.

[0076] In the embodiment of the present specification, performing face motion detection may specifically include: performing global motion detection on the entire face, the global motion detection may select key points of the relevant global area for detection, the key points of the global area may include some basic face key points of the cheek area, if the designated area is the cheek area, the key points of the global area may also include some supplementary face key points of the cheek area, the key points of the global area may also select some basic face key points and some supplementary face key points from each preset area. Then, according to the result of the global motion detection, if it is determined that no global face motion occurs, local motion detection is performed on one or more local face areas to determine whether face motion occurs in the local face area, the one or more local face areas at least include the designated area, and then S108 may be executed.

[0077] It should be noted that the local area in the embodiments of this specification may refer to multiple preset areas, or may be a newly planned area in the face image. When performing local motion detection, key points of the relevant local area may be selected for detection, and the key points of the local area may include some basic key points of the mouth area, eye area and eyebrow area. If the designated area includes one or more of the mouth area, eye area and eyebrow area, the key points of the local area may also include one or more supplementary key points of the mouth area, eye area and eyebrow area.

[0078] Further, in the embodiment of the present specification, if it is determined that global face motion has occurred based on the result of global motion detection, local motion detection will no longer be performed on one or more local face regions. Then, based on at least some of the key points that reflect global face motion predicted for the face image of the current frame, the basic face key point set and / or the supplementary face key point set are updated. If at least some of the key points that reflect global face motion do not include the specified area, there is no need to update the supplementary key point set.

[0079] The above steps are to first judge the global face movement by at least part of the key points reflecting the global face movement. If the global face occurs, the basic face key point set and / or the face supplementary key point set are updated only based on at least part of the key points reflecting the global face movement predicted for the face image of the current frame, without the need to update the basic face key points and face supplementary key points of the local face area.

[0080] Furthermore, when performing face motion detection based on the basic face key points and supplementary face key points predicted for the face image of the previous frame and the face image of the current frame, the embodiments of this specification may select some key points from the basic face key points and supplementary face key points predicted for the face image of the current frame; then, based on the selected key points, determine a global key point subset reflecting the global face for global motion detection, and one or more local key point subsets reflecting different local face regions for local motion detection of the corresponding local face regions. In this process, the embodiments of this specification perform motion detection on both the global and local sides at the same time.

[0081] S108, based on the result of face motion detection, if it is determined that face motion has occurred in the designated area, the face supplement key point set is updated based on the face supplement key points predicted for the face image of the current frame, so as to update the image processing accordingly.

[0082] Further, according to the result of face motion detection, if it is determined that face motion has occurred in the designated area, the face supplementary key point set is updated according to the face supplementary key points predicted for the face image of the current frame, specifically including:

[0083] As a result of local motion detection, if it is determined that facial motion has occurred in the specified area, the following contents are executed: based on the key points reflecting the specified area among the selected key points, the set of supplementary facial key points is updated; if the key points reflecting the specified area among the selected key points also include the basic facial key points, the set of basic facial key points is also updated accordingly.

[0084] As a result of the local motion detection, if it is determined that face motion has occurred in areas other than the specified area, the following is performed: based on the key points reflecting other areas among the selected key points, the basic face key point set is updated.

[0085] It should be noted that common beautification effects, such as stickers, face slimming, and makeup, all require accurate positioning of facial key points, and then image processing is performed in the area determined by the key points to generate the beautification effect. Therefore, beauty applications have high requirements for the accuracy of key points. Existing key point schemes, such as 68 points and 106 points, have insufficient points in local areas, and the line segments connecting the key points cannot accurately fit the local areas, affecting the beautification effect.

[0086] The existing 1000-point key point solution requires preliminary positioning of the approximate position of the eyes, ears, mouth and nose, and then extracting part of the image of the eyes, ears, mouth and nose area to run the pre-trained models of the eyes, ears, mouth and nose. This solution requires training a large number of models, which is time-consuming and not conducive to running on the IOT device side. At the same time, since the key points located by the algorithm will have certain errors, when processing continuous multi-frame video streams (continuous multi-frame face images), random errors often manifest as jitter of the output key points, resulting in unstable subsequent beauty effects. The existing key point post-processing solutions, whether filtering or directly updating based on the threshold, have the problem of high response delay when the smoothing effect is good. In addition, when there is a small part of occlusion on the face, the existing beauty algorithm does not make a judgment and often draws the beauty effect on the occlusion. The following is a detailed introduction to the existing 106 or 68 key points, 1000 key points, key point post-processing and cutout beauty:

[0087] Regarding 106 or 68 key points, these two key point definitions are the most common key point definitions. Most dense key point related technologies use these two key point definitions. Figure 2 The schematic diagram of the mouth definition structure of 106 key points is shown. Even if there are 106 points with a large number of points, when the positions are accurate, the area formed by the key points cannot accurately fit the actual area with a certain semantics, thus affecting the related beautification effect.

[0088] Regarding 1000 key points, 1000 dense key points can output a large number of key points, which has a good fit effect on the contours of the face. This solution first preliminarily locates the approximate position of the eyes, ears, mouth and nose, and then extracts part of the image of the eyes, ears, mouth and nose area to run the pre-trained models of the eyes, ears, mouth and nose, and further outputs very dense key point information of each part. Therefore, this solution requires training a large number of models, which is time-consuming and not conducive to running on the IoT device side.

[0089] Regarding post-processing of key points, various filtering algorithms such as Kalman filtering are commonly used, or weighted averaging is directly used to smooth the key points of multiple frames to solve the problem of key point jitter. However, these solutions have the problem of high response delay when the smoothing effect is good. In addition, there are also solutions that calculate whether the offset between each key point of the current frame and each key point of the previous frame exceeds the threshold, and update the key point after exceeding the threshold. There are also problems such as high response delay when the threshold is large, obvious jitter when the threshold is small, and jitter of some points when the face is still.

[0090] Regarding image beautification, some commonly used beautification algorithms are based on the location of key points to determine the local area, and use the corresponding beautification image processing algorithm in this area to obtain the beautification effect, such as the lipstick beautification effect: according to the key points of the mouth area, the corresponding key points are connected to determine the upper and lower lip areas, and the lipstick color is added to the upper and lower lip areas. However, when there is a certain occlusion in the local area, the filled-in beauty effect will be drawn on the occlusion, resulting in a bad visual effect.

[0091] Based on the above problems, the embodiments of this specification can add key points in the designated area so that the designated area can be an area that needs to be processed in detail. The embodiments of this specification can improve the fitting effect of key points in the local area with minimal increase in calculation amount based on a large amount of 106 or 68 key point data and a small amount of newly annotated local area key point data.

[0092] At the same time, the embodiment of this specification divides the area and selects stable points where the face does not move, and then combines high and low thresholds to make motion judgments. Among them, the method of updating key points in different areas can achieve lower delays while achieving better smoothing effects.

[0093] In addition, the embodiment of this specification adds the visible and invisible information of each key point to the face key point model. In the subsequent face image processing algorithm, such as the beauty algorithm, the visible part of the local area is further accurately extracted according to the visible and invisible information of the key points, and the beauty effect is accurately drawn to obtain a better effect. At the same time, the above scheme of the embodiment of this specification can be applied to fun face scanning.

[0094] It should be noted that face key point detection is an important link in the face image processing process. The face key point positioning task locates the key facial feature points on the face detection frame diagram input, including eyes, nose tip, mouth corners, and contour points of various parts of the face. It is widely used in technical fields such as face recognition, beauty, expression analysis, and face tracking. The face key point detection of the embodiment of this specification can adopt a deep learning-based solution, with the input being a picture containing a face, which is input into the network after image preprocessing to predict the position of each face key point.

[0095] At the same time, the embodiments of this specification can be applied to face recognition products, and the output key points can be used for exaggerated expression recognition and beauty effect generation. Common beauty effects, such as stickers, face thinning, makeup, etc., all require accurate positioning of facial key points, and then image processing is performed through the area determined by the key points to generate beauty effects. Therefore, beauty applications have high requirements for the accuracy of key points. At the same time, since the key points positioned by the algorithm will have certain errors, when processing video streams, random errors often manifest as jitters of the output key points, resulting in unstable subsequent beauty effects. In addition, when there is a small part of the face occluded, the existing beauty algorithm does not make a judgment and often draws the beauty effect on the occluder. The embodiments of this specification can be aimed at the high requirements for key point positioning accuracy and stability of beauty, and innovatively propose non-uniform key point design, a grouped key point smoothing strategy, and accurate extraction of the visible part of the local area based on the visible and invisible information of the key points, which are used to improve the beauty effect.

[0096] Specifically, the embodiments of this specification can be implemented by implementing the following specific steps.

[0097] The overall solution of the embodiment of this specification includes two parts: model training and key point detection. Figure 3 The flowchart of the model training shown in the figure takes the mouth area as an example, adds key points, that is, the mouth area is the designated area, and the added key points are the key points added to the face. The steps are:

[0098] S1101, use face images and annotated key point data to train a deep network model, the model consists of a backbone network and a prediction head, the model input is a face image, the output is 106 pre-defined face key points, and the information of whether each point is visible is output.

[0099] S1102, for lipstick beautification, a group of 32 key points are additionally defined in the mouth area as a supplement, and a batch of image data containing only the newly added mouth key points is marked.

[0100] S1103, based on the key point model trained in S1101, fix all network parameters and add a prediction head. The prediction head outputs the positions of 32 newly added key points. Use the data annotated in S1102 to train the network parameters of the newly added prediction head separately to obtain a new key point model, which is the facial key point model for the mouth area.

[0101] For the above facial key point model, the output effect can be seen in Figure 4 The schematic diagram of the effect of adding new key points is shown. The line segments in the figure are the mouth key points (basic face key points) included in the 106 key points, and the scattered points are the added mouth key points (supplementary face key points).

[0102] The key point detection of the embodiments of this specification can refer to Figure 5 The schematic diagram of the key point detection process shown in FIG. 1 includes the following steps:

[0103] S1201, run the face key point model for the face position in the t-th frame video image to obtain a total of 106+32=138 face key points, and divide all the key points into 7 area groups according to the left and right eyes, left and right eyebrows, nose, mouth, and cheeks.

[0104] S1202, select a part of key points and divide them into 6 motion groups, one group of key points is associated with the entire face, and the remaining five groups are respectively associated with several face regions: left and right eyes, left and right eyebrows, and mouth.

[0105] S1203, determine whether the face is moving based on a group of key points associated with the entire face. If so, based on the key points output in the t-1 frame, use the key points of the area group in the t-th frame to update, output the key points detected in S1201 as the key points of the t-th frame, and set t=t+1 to enter step S1201; if not, continue.

[0106] S1204, judging whether the associated face region groups are in motion according to the key points associated with the several face region groups, and if so, updating the key points of the region group in the tth frame based on the key points outputted in the t-1 frame.

[0107] S1205, output the key points updated in step S1204 as the key points of the t-th frame, set t=t+1, and re-enter step S1201.

[0108] For the above key point detection, the output effect can be seen in Figure 6 The figure shows the grouping of face regions, which respectively show the left and right eyes, left and right eyebrows, nose, mouth, and cheek regions. Figure 7 Schematic diagram of the key points of the motion group associated with the entire face, in which the key points are marked at the corners of the eyes and the corners of the mouth. Figure 8 A schematic diagram showing the key points of the associated eye movement group, and Fig. 9 A schematic diagram showing a group of key points associated with the motion of the mouth.

[0109] In this embodiment of the specification, when judging whether the key point of the motion group is moving, you can refer to Fig.10 The schematic diagram of the flow of motion detection of the motion group shown is as follows:

[0110] S1301, setting a low threshold and a high threshold, wherein the low threshold is 1.5% of the eye distance multiplied by the eye distance, and the high threshold is 3% of the eye distance multiplied by the eye distance, and for each key point of the motion group, setting a current threshold, which is initially the low threshold;

[0111] S1302, for a group of key points, calculate the average distance between the t-th frame and the t-1-th frame, and when the distance is greater than the current threshold of the group, mark the t-th frame of the motion group as motion, otherwise mark it as non-motion.

[0112] It should be noted that the average distance is the distance parameter mentioned above.

[0113] S1303, for each motion group, count the number of frames marked as motion from frame tN to frame t-1, if the number of frames is greater than M, set the current threshold to a low threshold, otherwise to a high threshold, where N and M are both positive integers greater than 1.

[0114] The visible and invisible information of key points obtained based on S1101 in the embodiments of this specification can be used to optimize the beauty effect.

[0115] See also Fig.11 The flowchart of invisible point beautification optimization is shown as follows:

[0116] S1401, determining key points of a processing area according to a required beautification effect, and pre-defining adjacent relationships between key points of the area.

[0117] S1402: Based on the invisible information outputted in step S1101, locally adjacent invisible points are divided into a group.

[0118] The locally adjacent invisible points may refer to adjacent invisible points within a preset range.

[0119] S1403, for each group of points whose number is greater than a certain threshold, determine it as an occlusion area, calculate the connected domain surrounded by the group of key points, and output the outer boundary of the connected domain.

[0120] S1404, when drawing the beauty effect, exclude each blocked communication domain.

[0121] For the adjacent relationship between the above predefined points, the output effect can be found in Fig.12 A schematic diagram of the adjacent relationship of the mouth area is shown, in which the key points connected by line segments are pre-defined adjacent points.

[0122] It should be noted that step S1101 of the embodiment of this specification may also adopt other key point definitions, such as 68 key points, 96 key points, etc.

[0123] In step S1102 of the embodiment of this specification, it is also possible to define additional eye key points for contact lens beautification, and it is also possible to define additional eyebrow key points for eyebrow beautification.

[0124] In step S1103 of the embodiment of this specification, multiple prediction heads may be added to train the newly added mouth, eye, and eyebrow key points respectively.

[0125] The region group in step S1201 of the embodiment of this specification can also be adjusted according to beauty requirements. For example, the cheek and nose key points can be combined into one group.

[0126] The threshold of step S1301 in the embodiment of this specification can be adjusted, and multiple threshold levels can also be added.

[0127] The embodiments of this specification can improve the fitting effect of key points in local areas with minimal increase in calculation amount by supplementing key points based on a large amount of 106 or 68 key point data, plus a small amount of newly annotated local area key point data, with minimal increase in calculation amount. At the same time, the embodiments of this specification can obtain lower latency while obtaining better smoothing effects by dividing the areas and selecting stable points, combining high and low thresholds for motion judgment, and updating key points in different areas. In addition, in the subsequent beauty algorithm, based on the visible and invisible information of the key points, the visible part of the local area can be further accurately extracted, and the beauty effect can be accurately drawn to obtain better results.

[0128] Fig.13 A structural schematic diagram of a facial key point processing device is provided for one or more embodiments of this specification. Multiple frames of facial images are continuously collected for the same face. The device includes: a prediction unit 1302, an update unit 1304, a detection unit 1306 and a processing unit 1308.

[0129] The prediction unit 1302 inputs the face image of the current frame into a pre-trained face key point model, predicts the basic face key points of multiple preset areas, and predicts the face supplementary key points of a designated area among the multiple preset areas;

[0130] The updating unit 1304 determines a basic face key point set and a supplementary face key point set obtained by updating the face images of each frame before the current frame, and the basic face key point set and the supplementary face key point set are used for image processing;

[0131] The detection unit 1306 performs face motion detection based on the face basic key points and face supplementary key points predicted for the face image of the previous frame and the face image of the current frame respectively;

[0132] Based on the result of face motion detection, if the processing unit 1308 determines that face motion has occurred in the designated area, the face supplementary key point set is updated according to the face supplementary key points predicted for the face image of the current frame, so as to update the image processing accordingly.

[0133] Furthermore, before the prediction unit 1302 inputs the face image of the current frame into the pre-trained face key point model, the apparatus further includes:

[0134] A first acquisition unit acquires a first face training image set in which basic key points of faces are marked in a plurality of preset areas;

[0135] A first training unit, training a deep network model composed of a backbone network and a basic prediction head according to a first face training image set, to obtain a basic key point model;

[0136] A second acquisition unit acquires a second face training image set with supplementary face key points annotated in a designated area;

[0137] The second training unit fixes the trained network parameters in the basic key point model, adds a supplementary prediction head to the basic key point model, trains the supplementary prediction head according to the second face training image set, and uses the basic key point model trained by the supplementary prediction head as the face key point model.

[0138] Further, the detection unit 1306 performs face motion detection based on the predicted basic face key points and supplementary face key points of the face image of the previous frame and the face image of the current frame, specifically including:

[0139] Calculating distance parameters between at least some of the corresponding basic face key points predicted for the face image of the previous frame and the face image of the current frame, and / or distance parameters between at least some of the corresponding supplementary face key points predicted for the face image of the previous frame and the face image of the current frame;

[0140] Face motion detection is performed based on the calculated distance parameter and the set distance threshold.

[0141] Furthermore, the distance threshold is set as follows:

[0142] Generate a low threshold and a high threshold according to the size of a specified part in the same face;

[0143] Initially, the distance threshold is set to a low threshold;

[0144] For several frames of face images before the current frame, if the number of face images determined to have undergone face motion is greater than the set threshold, the distance threshold is still set to the low threshold, otherwise, the distance threshold is set to the high threshold.

[0145] Furthermore, the detection unit 1306 performs face motion detection, which specifically includes:

[0146] Perform global motion detection on the face;

[0147] According to the result of global motion detection, if it is determined that no global face motion has occurred, local motion detection is performed on one or more local face areas to determine whether face motion has occurred in the local face areas, and the one or more local face areas at least include a designated area.

[0148] Furthermore, the device also includes:

[0149] The local detection unit, based on the result of the global motion detection, if it is determined that global face motion has occurred, then no local motion detection is performed on one or more local face regions this time;

[0150] The set updating unit updates the basic face key point set and / or the supplementary face key point set according to at least part of the key points reflecting the global face motion predicted for the face image of the current frame.

[0151] Further, the detection unit 1306 performs face motion detection based on the predicted basic face key points and supplementary face key points of the face image of the previous frame and the face image of the current frame, specifically including:

[0152] Selecting some key points from the basic face key points and the supplementary face key points predicted for the face image of the current frame respectively;

[0153] Based on the selected key points, a global key point subset reflecting the entire face is determined for global motion detection, and one or more local key point subsets reflecting different local face areas are determined for local motion detection of corresponding local face areas.

[0154] Further, the processing unit 1308 executes the face motion detection result. If it is determined that the face motion occurs in the designated area, the face supplement key point set is updated according to the face supplement key points predicted for the face image of the current frame, specifically including:

[0155] The result of local motion detection is that if it is determined that face motion has occurred in the specified area, the following is executed:

[0156] The face supplementary key point set is updated according to the key points reflecting the designated area among the selected key points;

[0157] If the key points reflecting the designated area in the selected key points also include basic face key points, the basic face key point set is also updated accordingly.

[0158] Further, after the prediction unit 1302 inputs the face image of the current frame into the pre-trained face key point model, the apparatus further includes:

[0159] Predicting visible units, by processing the facial key point model, predicting whether the key points in the basic facial key points and / or the supplementary facial key points are visible or not;

[0160] An adjacent relationship determination unit is configured to determine an area to be processed from among a plurality of preset areas, and to determine an adjacent relationship between key points to be processed in the area to be processed based on basic key points of the face and / or supplementary key points of the face;

[0161] The key point division unit divides the locally adjacent invisible key points to be processed into the same set to be processed according to the adjacent relationship and the prediction result of whether the key points are visible or not;

[0162] The occlusion determination unit determines that there is occlusion on the face in the face image if the number of key points to be processed contained in the set to be processed is greater than a set threshold, and uses the set to be processed as the occlusion set.

[0163] Furthermore, the image processing includes performing face display effect processing on the face image;

[0164] The device also includes:

[0165] A connected domain determining unit, which determines a connected domain surrounded by key points to be processed in the occlusion set;

[0166] The effect processing unit excludes the communication domain when performing face display effect processing on the face image, so that the face display effect processing is not performed in the communication domain.

[0167] Fig.14 A structural schematic diagram of a facial key point processing device is provided for one or more embodiments of this specification, including:

[0168] at least one processor; and,

[0169] a memory communicatively connected to at least one processor; wherein,

[0170] The memory stores instructions executable by at least one processor, the instructions being executed by at least one processor to enable the at least one processor to:

[0171] Inputting the face image of the current frame into a pre-trained face key point model, predicting the basic face key points of multiple preset areas, and predicting the face supplementary key points of a specified area in the multiple preset areas;

[0172] Determine a basic face key point set and a supplementary face key point set obtained by updating face images of frames before the current frame, wherein the basic face key point set and the supplementary face key point set are used for image processing;

[0173] Performing face motion detection based on the face basic key points and face supplementary key points predicted for the face image of the previous frame and the face image of the current frame respectively;

[0174] Based on the result of face motion detection, if it is determined that face motion has occurred in the designated area, the face supplementary key point set is updated based on the face supplementary key points predicted for the face image of the current frame, so as to update the image processing accordingly.

[0175] One or more embodiments of this specification provide a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured as follows:

[0176] Inputting the face image of the current frame into a pre-trained face key point model, predicting the basic face key points of multiple preset areas, and predicting the face supplementary key points of a specified area in the multiple preset areas;

[0177] Determine a basic face key point set and a supplementary face key point set obtained by updating face images of frames before the current frame, wherein the basic face key point set and the supplementary face key point set are used for image processing;

[0178] Performing face motion detection based on the face basic key points and face supplementary key points predicted for the face image of the previous frame and the face image of the current frame respectively;

[0179] Based on the result of face motion detection, if it is determined that face motion has occurred in the designated area, the face supplementary key point set is updated based on the face supplementary key points predicted for the face image of the current frame, so as to update the image processing accordingly.

[0180] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0181] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0182] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0183] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0184] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification may be in the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the embodiments of this specification may be in the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0185] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0186] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0187] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0188] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0189] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0190] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0191] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0192] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0193] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0194] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0195] The above are only one or more embodiments of this specification and are not intended to limit this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification shall be included in the scope of the claims of this specification.

Claims

1. A method for processing facial key points, wherein multiple frames of facial images are continuously collected for the same face, the method comprising: Inputting the face image of the current frame into a pre-trained face key point model, predicting the basic face key points of multiple preset areas, and predicting the face supplementary key points of a designated area among the multiple preset areas; Determine a basic face key point set and a supplementary face key point set obtained by updating face images of frames before a current frame, wherein the basic face key point set and the supplementary face key point set are used for image processing; Performing face motion detection based on the face basic key points and face supplementary key points predicted for the face image of the previous frame and the face image of the current frame respectively; If it is determined that face motion has occurred in the designated area according to the face motion detection result, the face supplementary key point set is updated according to the face supplementary key points predicted for the face image of the current frame, so as to update the image processing accordingly; Before inputting the face image of the current frame into the pre-trained face key point model, the method further includes: Acquire a first face training image set in which basic key points of faces are marked in the plurality of preset areas; According to the first face training image set, training a deep network model consisting of a backbone network and a basic prediction head to obtain a basic key point model; Acquire a second face training image set in which supplementary face key points are annotated in the designated area alone; The trained network parameters in the basic key point model are fixed, and a supplementary prediction head is added to the basic key point model. The supplementary prediction head is trained according to the second face training image set, and the basic key point model trained by the supplementary prediction head is used as the face key point model.

2. The method according to claim 1, wherein the face motion detection is performed based on the basic face key points and the supplementary face key points predicted from the face image of the previous frame and the face image of the current frame, and specifically comprises: Calculating distance parameters between at least some of the corresponding basic face key points predicted for the face image of the previous frame and the face image of the current frame, and / or distance parameters between at least some of the corresponding supplementary face key points predicted for the face image of the previous frame and the face image of the current frame; Face motion detection is performed based on the calculated distance parameter and the set distance threshold.

3. According to the method of claim 2, the distance threshold is set as follows: generating a low threshold and a high threshold according to the size of the designated part of the same face; Initially, the distance threshold is set to the low threshold; For several frames of face images before the current frame, if the number of face images determined to have undergone face motion is greater than the set threshold, the distance threshold is still set to the low threshold; otherwise, the distance threshold is set to the high threshold.

4. The method according to claim 1 or 2, wherein the face motion detection comprises: Perform global motion detection on the face; According to the result of the global motion detection, if it is determined that no global facial motion has occurred, local motion detection is performed on one or more local facial areas to determine whether facial motion has occurred in the local facial areas, and the one or more local facial areas at least include the designated area.

5. The method according to claim 4, further comprising: If it is determined, based on the result of the global motion detection, that global face motion has occurred, then no local motion detection is performed on the one or more local face regions this time; The basic face key point set and / or the supplementary face key point set are updated according to at least part of the key points reflecting the global face motion predicted for the face image of the current frame.

6. The method according to claim 4, wherein the face motion detection is performed based on the basic face key points and the supplementary face key points predicted from the face image of the previous frame and the face image of the current frame, and specifically comprises: Selecting some key points from the basic face key points and the supplementary face key points predicted for the face image of the current frame respectively; Based on the selected key points, a global key point subset reflecting the entire face is determined for performing the global motion detection, and one or more local key point subsets reflecting different local face areas are determined for performing the local motion detection of the corresponding local face areas.

7. The method according to claim 6, wherein if it is determined that face motion has occurred in the designated area according to the result of the face motion detection, the face supplementary key points set is updated according to the face supplementary key points predicted for the face image of the current frame, specifically comprising: If the result of the local motion detection determines that face motion has occurred in the designated area, then: updating the face supplementary key point set according to the key points reflecting the designated area among the selected key points; If the key points reflecting the designated area in the selected key points also include basic face key points, the basic face key point set is also updated accordingly.

8. The method according to claim 1, after inputting the face image of the current frame into the pre-trained face key point model, the method further comprises: Predicting whether the key points in the basic face key points and / or the supplementary face key points are visible or not by processing the face key point model; Determine a to-be-processed area among the plurality of preset areas, and determine the adjacent relationship between the to-be-processed key points in the to-be-processed area according to the basic face key points and / or the supplementary face key points; According to the adjacent relationship and the prediction result of whether the key point is visible or not, the locally adjacent invisible key points to be processed are divided into the same set to be processed; If the number of the key points to be processed contained in the set to be processed is greater than a set threshold, it is determined that there is occlusion on the face in the face image, and the set to be processed is used as an occlusion set.

9. The method according to claim 8, wherein the image processing comprises performing face display effect processing on the face image; The method further comprises: Determine a connected domain surrounded by the key points to be processed in the occlusion set; When performing face display effect processing on the face image, the communication domain is excluded so that the face display effect processing is not performed in the communication domain.

10. A facial key point processing device, which continuously collects multiple frames of facial images for the same face, the device comprising: A prediction unit, inputting a face image of a current frame into a pre-trained face key point model, predicting basic face key points of a plurality of preset areas, and predicting supplementary face key points of a designated area among the plurality of preset areas; An updating unit, which determines a basic face key point set and a supplementary face key point set obtained by updating face images of frames before a current frame, wherein the basic face key point set and the supplementary face key point set are used for image processing; A detection unit performs face motion detection based on the face basic key points and face supplementary key points predicted from the face image of the previous frame and the face image of the current frame respectively; A processing unit, based on the result of the face motion detection, if it is determined that face motion has occurred in the designated area, updates the set of face supplementary key points according to the face supplementary key points predicted for the face image of the current frame, so as to update the image processing accordingly; Before the prediction unit performs the step of inputting the face image of the current frame into the pre-trained face key point model, the device further includes: A first acquisition unit acquires a first face training image set annotated with basic face key points in the plurality of preset areas; A first training unit, training a deep network model consisting of a backbone network and a basic prediction head according to the first face training image set to obtain a basic key point model; A second acquisition unit acquires a second face training image set in which supplementary face key points are annotated in the designated area alone; The second training unit fixes the trained network parameters in the basic key point model, and adds a supplementary prediction head to the basic key point model, trains the supplementary prediction head according to the second face training image set, and uses the basic key point model trained by the supplementary prediction head as the face key point model.

11. The device according to claim 10, wherein the detection unit performs the face motion detection based on the predicted basic face key points and supplementary face key points of the face image of the previous frame and the face image of the current frame, specifically comprising: Calculating distance parameters between at least some of the corresponding basic face key points predicted for the face image of the previous frame and the face image of the current frame, and / or distance parameters between at least some of the corresponding supplementary face key points predicted for the face image of the previous frame and the face image of the current frame; Face motion detection is performed based on the calculated distance parameter and the set distance threshold.

12. The device according to claim 11, wherein the distance threshold is set as follows: generating a low threshold and a high threshold according to the size of the designated part of the same face; Initially, the distance threshold is set to the low threshold; For several frames of face images before the current frame, if the number of face images determined to have undergone face motion is greater than the set threshold, the distance threshold is still set to the low threshold; otherwise, the distance threshold is set to the high threshold.

13. The device according to claim 10 or 11, wherein the detection unit performs face motion detection, specifically comprising: Perform global motion detection on the face; According to the result of the global motion detection, if it is determined that no global facial motion has occurred, local motion detection is performed on one or more local facial areas to determine whether facial motion has occurred in the local facial areas, and the one or more local facial areas at least include the designated area.

14. The apparatus according to claim 13, further comprising: A local detection unit, based on the result of the global motion detection, if it is determined that global face motion has occurred, then no local motion detection is performed on the one or more local face regions this time; A set updating unit updates the basic face key point set and / or the supplementary face key point set according to at least part of the key points reflecting the global face motion predicted for the face image of the current frame.

15. The device according to claim 13, wherein the detection unit performs the face motion detection based on the predicted basic face key points and supplementary face key points of the face image of the previous frame and the face image of the current frame, specifically comprising: Selecting some key points from the basic face key points and the supplementary face key points predicted for the face image of the current frame respectively; Based on the selected key points, a global key point subset reflecting the entire face is determined for performing the global motion detection, and one or more local key point subsets reflecting different local face areas are determined for performing the local motion detection of the corresponding local face areas.

16. The apparatus according to claim 15, wherein the processing unit executes the face motion detection result, and if it is determined that face motion has occurred in the designated area, the face supplementary key point set is updated according to the face supplementary key points predicted for the face image of the current frame, specifically comprising: If the result of the local motion detection determines that face motion has occurred in the designated area, then: updating the face supplementary key point set according to the key points reflecting the designated area among the selected key points; If the key points reflecting the designated area in the selected key points also include basic face key points, the basic face key point set is also updated accordingly.

17. The device according to claim 10, after the prediction unit performs the step of inputting the face image of the current frame into the pre-trained face key point model, the device further comprises: A visibility prediction unit, which predicts whether the key points in the basic face key points and / or the supplementary face key points are visible or not by processing the face key point model; an adjacent relationship determining unit, which determines an area to be processed from among the plurality of preset areas, and determines an adjacent relationship between the key points to be processed in the area to be processed according to the basic key points of the face and / or the supplementary key points of the face; A key point division unit, which divides the locally adjacent invisible key points to be processed into the same set to be processed according to the adjacent relationship and the prediction result of whether the key points are visible or not; The occlusion determination unit determines that there is occlusion on the face in the face image if the number of the key points to be processed included in the set to be processed is greater than a set threshold, and uses the set to be processed as an occlusion set.

18. The device according to claim 17, wherein the image processing comprises performing face display effect processing on the face image; The device also includes: A connected domain determining unit, which determines a connected domain surrounded by the key points to be processed in the occlusion set; The effect processing unit excludes the communication domain when performing face display effect processing on the face image, so that the face display effect processing is not performed in the communication domain.

19. A facial key point processing device, comprising: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: Inputting the face image of the current frame into a pre-trained face key point model, predicting the basic face key points of multiple preset areas, and predicting the face supplementary key points of a designated area among the multiple preset areas; Determine a basic face key point set and a supplementary face key point set obtained by updating face images of frames before a current frame, wherein the basic face key point set and the supplementary face key point set are used for image processing; Performing face motion detection based on the face basic key points and face supplementary key points predicted for the face image of the previous frame and the face image of the current frame respectively; If it is determined that face motion has occurred in the designated area according to the face motion detection result, the face supplementary key point set is updated according to the face supplementary key points predicted for the face image of the current frame, so as to update the image processing accordingly; Before inputting the face image of the current frame into the pre-trained face key point model, the following is further performed: Acquire a first face training image set in which basic key points of faces are marked in the plurality of preset areas; According to the first face training image set, training a deep network model consisting of a backbone network and a basic prediction head to obtain a basic key point model; Acquire a second face training image set in which supplementary face key points are annotated in the designated area alone; The trained network parameters in the basic key point model are fixed, and a supplementary prediction head is added to the basic key point model. The supplementary prediction head is trained according to the second face training image set, and the basic key point model trained by the supplementary prediction head is used as the face key point model.

Images