A face image anti-shake method, device, equipment and readable storage medium

Abstract

This application provides a method, apparatus, device, and readable storage medium for stabilizing facial images. The method includes: acquiring a plurality of facial feature points in a current image frame; determining displacement vectors of the facial feature points based on the current image frame and at least two adjacent image frames; determining a clustering space corresponding to the facial feature points; clustering the facial feature points in the clustering space according to the displacement vectors, wherein the clustering space includes the facial feature points and facial feature points in a specified region surrounding the facial feature points; and performing image stabilization processing on the facial feature points based on the clustering results.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, device, and readable storage medium for stabilizing facial images. Background Technology

[0002] With the development of technology, deep learning technology, using models trained on large datasets, is applied to all aspects of life and industry. Face recognition applications, in particular, have been very successful due to the inherent structured and stable nature of human faces.

[0003] Whether using deep learning algorithms or traditional facial landmark algorithms, pixel jitter is inevitable in the output results. This jitter is usually caused by external interference factors such as random noise, image background, or changes in lighting when taking multiple images. Traditional face stabilization techniques typically use different filtering algorithms, such as Kalman filtering, median filtering, and Gaussian filtering. However, these algorithms generally do not consider whether the output reflects normal movement of facial landmarks or jitter; they treat all outputs as jitter for filtering. If the algorithm that removes jitter has a large weight, it will result in a long processing time and image delay; if the weight is small, unavoidable jitter will occur. Theoretically, when the face is still, the feature points output after filtering multiple images should have no obvious jitter in visual effect. When the face is moving, the stabilization effect needs to be maintained without significant delay, but current commonly used filtering algorithms cannot achieve this. Summary of the Invention

[0004] In view of this, this application provides a method, apparatus, device, and readable medium for stabilizing facial images.

[0005] Specifically, this application is implemented through the following technical solution:

[0006] In a first aspect, embodiments of this application provide a method for stabilizing facial images, the method comprising:

[0007] Obtain several facial feature points in the current image frame;

[0008] Based on the current image frame and at least two adjacent image frames of the current image frame, determine the displacement vector of the facial feature point;

[0009] Determine the clustering space corresponding to the facial feature points, wherein the clustering space includes the facial feature points and facial feature points in a specified area surrounding the facial feature points;

[0010] The facial feature points in the clustering space are clustered according to the displacement vector;

[0011] Based on the clustering results, the facial feature points are subjected to anti-shake processing.

[0012] Secondly, embodiments of this application provide a face image stabilization device, the device comprising:

[0013] The image detection module is used to obtain several facial feature points in the current image frame;

[0014] The processing module is used to determine the displacement vectors of the acquired facial feature points in the current image frame and at least two adjacent frames of the current image frame.

[0015] A clustering module is used to determine the clustering space corresponding to the facial feature points, and to cluster the facial feature points in the clustering space according to the displacement vector; the clustering space includes the facial feature points and facial feature points in a specified area around the facial feature points;

[0016] The image stabilization module is used to perform image stabilization processing on the facial feature points based on the clustering results.

[0017] Thirdly, embodiments of this application provide a face image stabilization device, the device comprising:

[0018] processor;

[0019] Memory used to store processor-executable instructions;

[0020] The processor is configured to perform the operations described in any of the methods described in the first aspect above.

[0021] Fourthly, this application provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, perform the operations described in any of the methods described in the first aspect above.

[0022] The beneficial effects achieved by this application are as follows: by capturing the displacement vector of facial feature points, clustering the facial feature points based on the displacement vector, determining whether the face is moving normally or shaking based on the displacement of the facial feature points in the cluster, and applying different filtering processes to the shaking and normal movement of the target face to achieve a good anti-shake effect without delay, and still achieving a good anti-shake effect after the target face movement state changes. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of a method according to an embodiment of this application;

[0025] Figure 2 This is a schematic diagram showing the theoretical and actual positions of facial feature points during movement in one embodiment of this application;

[0026] Figure 3 This is a schematic diagram of the displacement of facial feature points in the same cluster in one embodiment of this application;

[0027] Figure 4 This is a device frame diagram according to an embodiment of this application;

[0028] Figure 5 This is a diagram of a live streaming network architecture in one embodiment of this application;

[0029] Figure 6 This is a flowchart illustrating the application of a face stabilization device in a live streaming scenario according to one embodiment of this application.

[0030] Figure 7 This is a schematic diagram of a face stabilization device according to an embodiment of this application. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] With the development of technology, deep learning technology, which uses big data models to train models, is applied to all aspects of life and industry. Face recognition applications, in particular, have been very successful due to the structured and stable nature of faces.

[0033] During the facial recognition process, due to environmental influences and differences in the target face itself, it is often necessary to process the target face in some facial recognition algorithms. For example, when performing facial identity authentication in a dark environment, it is necessary to adjust the exposure of the target face image, or during live streaming, it is necessary to perform beautification processing on the target face image.

[0034] Whether it's deep learning algorithms or traditional facial landmark algorithms, they all require detecting and extracting facial landmarks on the target face image, followed by further processing of these landmarks. However, the facial landmarks output by these algorithms inevitably have some jitter. The reasons for this jitter are usually random noise and the influence of changes in image background / lighting on the results. Moreover, jitter in algorithm output is quite common and cannot be completely eliminated.

[0035] In traditional image stabilization technology, the algorithm processes the motion of each extracted facial feature point, ignoring the fact that these facial feature points may be moving normally as part of the face. As a result, when the image stabilization weight parameter is large, there is a large delay in the image, while when the image stabilization weight parameter is small, the image stabilization effect is not obvious.

[0036] Therefore, embodiments of this application provide a method for stabilizing facial images, referring to... Figure 1 , Figure 1 These are some of the steps in this embodiment.

[0037] In step S101, several target face feature points in the current image frame are obtained;

[0038] In step S102, the displacement vector of the facial feature point is determined based on the current image frame and at least two adjacent image frames of the current image frame;

[0039] In step S103, a clustering space corresponding to the facial feature points is determined, and the facial feature points are clustered according to the clustering space. The clustering space includes the facial feature points and facial feature points in a specified area around the facial feature points.

[0040] In step S104, the facial feature points are subjected to anti-shake processing based on the clustering results.

[0041] In this embodiment, the movement of facial feature points is reflected in their different positions within different image frames. Therefore, in step S102, when calculating the displacement vector of the several facial feature points in the current frame, it is necessary to use an image frame containing the same target face as a reference to calculate the position change of the facial feature points in the two image frames, and obtain the displacement vector of the facial feature points. Since facial feature points are not completely stationary in different image frames, even if the face does not move, the position of the same facial feature point in different image frames may not be completely consistent. This movement of facial feature points is called jitter. Therefore, in this application, the changes in the displacement vector of facial feature points in the current frame and at least two adjacent image frames containing the current frame are considered. It is worth noting that the number of adjacent image frames selected can be determined according to the actual needs of the designer.

[0042] Facial feature point jitter, such as Figure 2 As shown, A is a facial feature point extracted in the current frame, B is the position of point A in the adjacent frame without considering jitter, and B1 is the position of point B after jitter occurs in the actual situation. Theoretically, the displacement vector of point A after the face moves should be... However, during the algorithm processing, these facial feature points will exhibit disordered jitter, therefore the actual displacement vector extracted for point A is... Alternatively, the displacement vector can be calculated using a pseudo-optical flow method, which treats the extracted facial feature points as pixels and calculates the positional change of the pixel within the two frames.

[0043] In this embodiment, the face image stabilization method provided by this application can be used for both 3D and 2D face images. Taking 3D face images as an example: when processing 3D face images, only their planar features can be processed, because the shaking of facial feature points in a face image only appears as planar shaking in visual effect. For example, in the face recognition process for online account authentication, when stabilizing the facial features of the target person, the shooting device captures a 3D face image. After extracting the facial feature points from the 3D image, the 3D coordinates of these facial feature points are mapped to coordinates in a 2D coordinate system to obtain the mapping points of these 3D facial feature points in the 2D coordinate system. Then, these mapping points are processed. Processing 2D coordinates achieves the same effect as processing 3D coordinates, but reduces the amount of computation, making the processing efficiency higher.

[0044] Furthermore, after determining the motion of each extracted facial feature point, these facial feature points are purposefully combined, and the motion pattern of the combined Part is observed to determine whether the target face image is moving normally or jittering. Here, "Part" means "division," meaning that the facial feature points are combined to form several Parts, and these Parts combine to form the whole of the target face image. It should be noted that the difference between normal movement and jittering of facial feature points is that, in the process of normal face movement, the movement direction of all facial feature points should theoretically be ordered and consistent, while jittering generated by the algorithm is disordered. However, when the movement of several facial feature points is composed of a combination of ordered movement and disordered jittering, these facial feature points have a common range of movement directions, that is, their average movement direction should be consistent with the movement direction of the target face. Therefore, when observing these Parts composed of feature points, we mainly observe whether the average motion parameters of these Parts are ordered in the movement direction. The process of purposefully combining the above-mentioned facial feature points into Parts is called clustering.

[0045] In step S103, the first step is to divide the clustering space. The clustering space is formed by combining a certain extracted facial feature point with the extracted facial feature points around it. Facial feature points are usually clustered with facial feature points in the same clustering space. After confirming the clustering space, the extracted facial feature points in the clustering space are clustered.

[0046] In the specific implementation process, clustering can be performed based on the mathematical characteristics of the displacement vectors of the facial feature points. These mathematical characteristics include the vector length and direction angle. In this embodiment, the direction angle of the displacement vectors is considered. By presetting the range of variation of the displacement vector direction angle, the facial feature points are clustered. It should be noted that this range of displacement vector direction angle can be adjusted according to specific usage. For example, when the target face is moving with a large amplitude, the range of displacement vector direction angle can be adjusted to be smaller. This is because during large-amplitude movements, the change in the direction angle of the displacement vector caused by shaking is very small; therefore, reducing the range of direction angle helps improve the anti-shake effect. In this embodiment, the range of variation of the displacement vector direction angle is 20 degrees.

[0047] In step S103, the clustering process requires calculating the Part for each point. Each facial feature point can only exist in one clustering space, and facial feature points that are not in the same clustering space cannot be clustered even if the difference in the displacement vector direction between the two points is less than 20 degrees. The advantage of doing this is that some boundary points at the critical value will also be calculated, and the accuracy will not be affected by too many points being ignored after clustering. Especially when the target face image is a close-up static image, all the facial feature points may be jittering in disorder. Therefore, even with the constraint of the clustering space, the difference in the displacement vector direction angle between the facial feature points will be very large, which cannot meet the above direction angle range. Therefore, most points will not be clustered. So in this case, these facial feature points that have not been clustered cannot be ignored.

[0048] In step S104, the mathematical characteristics of the displacement vectors in the cluster are analyzed. Generally, a displacement vector is a vector. In this embodiment, the vector is analyzed, such as... Figure 3 As shown in the figure, A1 and A2 are face feature points within a certain cluster in the current frame, and B1 and B2 are points of A1 and A2 in adjacent frames, respectively. The calculations are performed on each point. The vectors are then subjected to statistical analysis to obtain a weight vector. For example, the average of the direction and length of the displacement vectors in the same cluster is calculated, and the two averages form a new vector, which is the weight vector. Using this weight vector as the displacement vector for the movement of the cluster, the displacement vector of each facial feature point in the cluster is then calculated (e.g., ...). The difference between the weight vector and the target facial feature point can be used to obtain the parameters that need to be stabilized. Stabilizing these facial feature points by applying stabilization parameters can achieve the stabilization effect. Stabilization methods can include median filtering and other techniques.

[0049] It should be noted that when a face is stationary at close range, no clustering may occur, resulting in the inability to obtain a weight vector. This is because, in a motion state where the face remains stationary, all facial feature points detected by the algorithm are in the aforementioned jittering state. Since jittering is completely random, it is very likely that there will be no facial feature points in the same clustering space that satisfy the range of changes in the displacement vector direction angle. To address this, this application determines the motion state of the face by judging whether the number of facial feature points in an unclustered state is greater than a specified threshold. The specified threshold can be a pre-set threshold or can be set before judgment according to the usage, such as detecting the motion state parameters of the target image, and automatically generating appropriate weight vectors based on these motion state parameters. The threshold values ​​are used to determine the number of facial feature points. After the determination, if the number of facial feature points not included in the cluster is greater than the threshold, it indicates that the face has not moved or has moved very little. This suggests that the movement of these facial feature points is mostly random shaking (such as the face being stationary or the face moving very little close to the camera). In this case, the smoothing algorithm can be strengthened for all the facial feature points to achieve anti-shake effect. Conversely, if the number is less than the threshold, it indicates that the face is moving. This suggests that the movement of these facial feature points is a combination of normal movement and shaking. In this case, the anti-shake processing can be applied only to the points in the cluster, and no processing is applied to the facial feature points not included in the cluster. Therefore, the above technical means are universal for anti-shake of dynamic and static facial images and do not require additional parameter or process changes.

[0050] In addition, since this application calculates each obtained facial feature point when clustering facial feature points, when the number of facial feature points not included in the cluster is greater than the threshold, each extracted facial feature point can be processed accordingly. The advantage of doing so is that the accuracy will not be affected by too many ignored thresholds. The reason why these facial feature points can be ignored when the number of facial feature points not included in the cluster is less than the threshold is that during the movement of the face, the movement of all facial feature points is mainly orderly. It is only necessary to perform anti-shake processing on the facial feature points in each cluster space to achieve the anti-shake effect. The shaking of individual critical points does not affect the overall visual perception.

[0051] In another embodiment, a face stabilization device is provided, such as... Figure 4 As shown,

[0052] Image detection module 201 is used to acquire several facial feature points in the current frame;

[0053] Processing module 202 is used to determine the displacement vectors of the acquired facial feature points in the current image frame and at least two adjacent frames of the current image frame;

[0054] The clustering module 203 is used to determine the clustering space corresponding to the facial feature points, and to cluster the facial feature points in the clustering space according to the displacement vector; the clustering space includes the facial feature points and facial feature points in a specified area around the facial feature points;

[0055] The image stabilization module 204 is used to perform image stabilization processing on the facial feature points based on the clustering results.

[0056] The image frame is a 3D image; the image detection module 201 is used to acquire several facial features in the current image frame, including:

[0057] Map the coordinates of facial feature points identified in a 3D image to coordinates in a 2D coordinate system.

[0058] The clustering module 203 performs the following process: clustering facial feature points in the clustering space based on facial feature point displacement vectors.

[0059] The facial feature points in the clustering space are clustered based on the direction and angle of the displacement vector, wherein the direction and angle are not greater than a preset range of 20 degrees.

[0060] The image stabilization module 204 performs image stabilization processing on facial feature points based on the clustering results, specifically including:

[0061] The facial feature points in the clustering results are compared with a specified threshold, and based on the comparison results, anti-shake processing is performed on all the acquired facial feature points or facial feature points in the clustering space.

[0062] One application scenario for the face image stabilization solution provided in this application is to perform face image stabilization processing for broadcasters in live streaming scenarios. The following is an application example using this scenario. Figure 5 As shown, the architecture of the live streaming network includes: a live streaming server 301, a broadcaster client 302, and a viewer client 303. The face image stabilization solution provided in this application can be deployed on different clients to achieve the stabilization effect, or it can be deployed on the server to achieve the same stabilization effect. In this embodiment, the face image stabilization solution provided in this application is deployed on the live streaming client 302, and its process is as follows: Figure 6 As shown:

[0063] In step S401, the camera installed on the device where the broadcaster client is located inputs the captured image;

[0064] In step S402, the anchor image frame is acquired through the input image;

[0065] In step S403, the target face image is detected from the image frame, and several face feature points on the target face image are extracted;

[0066] In step S404, the displacement vectors of these facial feature points are calculated based on the current image frame and at least two adjacent image frames of the current image frame;

[0067] In step S405, clustering is performed based on the displacement vectors of facial feature points;

[0068] In step S406, it is determined whether the number of facial feature points not included in the cluster exceeds a threshold. When the anchor's face is moving, the clustering result will contain most of the extracted facial feature points, and the number of remaining facial feature points not included in the cluster is less than the threshold. Furthermore, the clustering weight vector is basically consistent with the displacement vector of the anchor's face. At this point, step S407 is executed, where the facial feature points included in the clustering result are subjected to anti-shake processing based on the difference between the displacement vector of the facial feature points in the clustering result and the clustering weight vector. This does not affect the aforementioned... Normal movement of facial feature points; however, when the anchor's face is stationary, the clustering result will only contain a small number of extracted facial feature points. The number of remaining facial feature points that do not enter the cluster is greater than the threshold, and a cluster weight vector cannot be obtained or the weight vector is 0. At this time, step S408 is executed to perform anti-shake processing on all extracted facial feature points. Throughout the live broadcast, regardless of whether the anchor's face is moving or stationary, the image seen by the audience is an image that has been anti-shake processed, and there will be no delay due to the use of the anti-shake device.

[0069] In step S409, the processed image is output.

[0070] In this embodiment, the image frames captured by the shooting device are 3D images. The processing method for 3D face images only processes their planar features. This is because the shaking of facial feature points in a face image only manifests as planar shaking in visual effect. For example, when stabilizing the facial features of the anchor during a live broadcast, the shooting device captures a 3D face image. After extracting the facial feature points from the 3D image, the 3D coordinates of these facial feature points are mapped to the coordinates in the 2D coordinate system to obtain the mapping points of these 3D facial feature points in the 2D coordinate system. Then, these mapping points are processed. Processing 2D coordinates achieves the same effect as processing 3D coordinates while reducing computational load, resulting in higher processing efficiency. It should be noted that since all image frames are continuous in real-time scenarios, this module can adjust the sampling frequency according to specific circumstances when processing these continuous image frames. Here, the sampling frequency refers to the image detection module 201 detecting continuous image frames at certain intervals or frame counts. The original continuous frame is defined as L, and after detection, the new continuous frames arranged in the detection order are defined as M, where M is less than or equal to L. In this application scenario, L and M continuously extend as the live broadcast progresses, and during this extension, there may be back-and-forth movement of the target face. For example, if a broadcaster's face moves 10 units to the left and then 10 units to the right, the target face's displacement is 0 during this time. Therefore, during image detection, the sampling interval can be adjusted according to the actual situation to adapt to different application scenarios. In this embodiment, no special sampling interval is set; the sampling frequency is the shooting frequency of the camera.

[0071] After obtaining M from L, M is processed by calculating the positional changes of the facial feature points in the (N+1)th frame of M and the facial feature points in the Nth frame to obtain the displacement vector between the two points. In this embodiment, the displacement vector points from the facial feature points in the Nth frame to the facial feature points in the (N+1)th frame. Optionally, the displacement vector can be calculated by using the optical flow method to calculate the pixel displacement, or by other methods for calculating the displacement vector.

[0072] After calculation, the facial feature points are clustered based on their displacement vectors. In this embodiment, the direction angle of the displacement vector is chosen as the reference, and the preset range of direction angle variation is 20 degrees. Optionally, other mathematical features such as the length of the displacement vector can also be used as reference standards for clustering. When using direction angle as the reference, the range of direction angle variation can be determined according to the actual situation. It should be noted that in the same cluster, without considering jitter, when the face moves, the absolute position of the facial feature points changes between adjacent frames, but the relative position does not change. For example, if the distance between two points A and B before the movement is 2 units, the distance between them after the overall movement to the right is still 2 units. Therefore, during clustering, the facial feature points before the face movement can be clustered, that is, based on the facial feature points from the Nth frame... The facial feature points in frame N are clustered based on the direction and angle of the displacement vector from the position to the position in frame N+1 and the clustering space. Similarly, the facial feature points can also be clustered after the face has moved, that is, clustered based on the direction and angle of the displacement vector from frame N+1 to frame N and the clustering space. Although the displacement vector of the cluster obtained by this method is opposite in direction to the result obtained by the first method, the displacement vector of the facial feature points in the cluster also becomes from N+1 to N, which is also opposite to the displacement vector of the facial feature points in the first method. Therefore, the relative difference between the displacement vector of the facial feature points and the displacement vector of the cluster does not change, so the final result is the same. The appropriate method can be selected according to the specific application. In this embodiment, the displacement vector from frame N to frame N+1 is used for calculation.

[0073] In this embodiment, when processing the clustering results, the number of unclustered facial feature points in the clustering results is compared with a specified threshold. If the number of unclustered facial feature points exceeds the threshold, the result is determined that the current face is in a shaking state, and all acquired facial feature points are then processed. If the number of unclustered facial feature points does not exceed the threshold, the result is determined that the face is in a normal moving state, and only the facial feature points included in the cluster are processed. Unclustered facial feature points refer to: 1. Boundary points that are not classified into the clustering space; 2. Points in the clustering space where the random displacement caused by shaking is large or the target face's movement displacement is small, resulting in the displacement vector direction angle change range of this point and other facial feature points in the same clustering space not meeting the conditions. It should be noted that the threshold specified in this embodiment can be a threshold preset inside the device, or it can be an input quantity input in real time according to the actual situation. The size of the threshold can be different in different application scenarios. For example, in low motion scenarios, the threshold can be as small as possible to improve the processing accuracy of each point, while in high motion scenarios, the threshold can be set appropriately larger to ensure that normally moving points are not treated as jitter.

[0074] Correspondingly, such as Figure 7 As shown, this application also provides an image stabilization device 50, including a processor 51; a memory 52 for storing executable instructions, the memory 52 including a computer program; wherein, the processor 51 is configured to:

[0075] Obtain several facial feature points in the current image frame;

[0076] Based on the current image frame and at least two adjacent image frames of the current image frame, determine the displacement vector of the facial feature point;

[0077] Determine the clustering space corresponding to the facial feature points, wherein the clustering space includes the facial feature points and facial feature points in a specified area surrounding the facial feature points;

[0078] The facial feature points in the clustering space are clustered according to the displacement vector;

[0079] Based on the clustering results, the facial feature points are subjected to anti-shake processing.

[0080] The processor 51 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0081] The memory 52 may include at least one type of storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. Furthermore, the device can cooperate with network storage devices that perform storage functions via a network connection. The memory 52 may be an internal storage unit of the device 50, such as the hard disk or RAM of the device 50. The memory 52 may also be an external storage device of the device 50, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the device 50.

[0082] Furthermore, memory 52 may include both internal storage units of device 50 and external storage devices. Memory 52 is used to store computer programs and other programs and data required by the device. Memory 52 can also be used to temporarily store data that has been output or will be output.

[0083] Device 50 can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. The device may include, but is not limited to, a processor 51 and a memory 52. ​​Those skilled in the art will understand that... Figure 7 This is merely an example of device 50 and does not constitute a limitation on device 50. It may include more or fewer components than shown, or combine certain components, or different components. For example, the device may also include input / output devices, network access devices, buses, etc.

[0084] The specific implementation process of the functions and roles of each unit in the above-mentioned equipment can be found in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0085] Furthermore, in some embodiments, traditional image stabilization devices process the movement of each facial feature point separately, resulting in the phenomenon that normally moving facial feature points are treated as jitter, causing a large delay in the target image during motion. However, since the device provided in this application can correlate the extracted discrete facial feature points, it can determine whether the face is moving normally or jittering, and thus perform targeted image stabilization processing. It will not treat normally moving facial feature points as jitter, thus having the characteristic of zero delay. This is one of the reasons why the device provided in this application can be well-suited for real-time application scenarios such as live streaming.

[0086] In this document, the terms "certain embodiments," "one embodiment," "another embodiment," "some embodiments," or "other embodiments," etc., refer to specific features, structures, materials, or conditions described in connection with the embodiments, which are included in at least one embodiment of the present application. In this document, illustrative expressions of the above terms do not necessarily refer to the same embodiments. Furthermore, the specific features, structures, materials, or conditions described above may be combined in any suitable manner in one or more embodiments.

[0087] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0088] The apparatus and method provided in the embodiments of this application have been described in detail above. Specific examples have been used in this application to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. For those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. The content of this specification should not be construed as a limitation of this application.

Claims

1. A method for stabilizing facial images, characterized in that, The method includes: Obtain several facial feature points in the current image frame; Based on the current image frame and at least two adjacent image frames, determine the displacement vector of the facial feature point, wherein the current image frame and the at least two adjacent image frames contain the same target face; Determine the clustering space corresponding to the facial feature points, wherein the clustering space includes the facial feature points and facial feature points in a specified area surrounding the facial feature points; The facial feature points in the clustering space are clustered according to the displacement vector to obtain the clustering result; If the number of facial feature points in the clustering results that are not clustered is greater than a specified threshold, then the target face is determined to be in a stationary state, and anti-shake processing is performed on all the acquired facial feature points. If the number of facial feature points in the clustering result that are not clustered is less than or equal to the specified threshold, then the target face is determined to be in a normal moving state, and anti-shake processing is only performed on the facial feature points in the clustering space.

2. The method according to claim 1, characterized in that: The current image frame is a 3D image; the steps for obtaining several facial feature points in the current image frame include: The coordinates of the facial feature points determined in the 3D image are mapped to coordinates in a 2D coordinate system.

3. The method according to claim 1, characterized in that: The steps of clustering facial feature points in the clustering space based on the displacement vector include: The facial feature points in the clustering space are clustered according to the direction angle of the displacement vector, wherein the direction angle is not greater than a preset range of variation.

4. The method according to claim 3, characterized in that: The preset range of variation is 20 degrees.

5. A face image stabilization device, characterized in that, The device includes: The image detection module is used to obtain several facial feature points in the current image frame; The processing module is used to determine the displacement vector of the facial feature point based on the current image frame and at least two adjacent image frames of the current image frame, wherein the current image frame and the at least two adjacent image frames contain the same target face; The clustering module is used to determine the clustering space corresponding to the facial feature points, and to cluster the facial feature points in the clustering space according to the displacement vector to obtain the clustering result; the clustering space includes the facial feature points and facial feature points in a specified area around the facial feature points; The image stabilization module is used to determine that the target face is stationary if the number of facial feature points in the clustering result that are not clustered is greater than a specified threshold, and to perform image stabilization on all the acquired facial feature points; if the number of facial feature points in the clustering result that are not clustered is less than or equal to the specified threshold, the module determines that the target face is in a normal moving state, and to perform image stabilization only on the facial feature points in the clustering space.

6. The apparatus according to claim 5, characterized in that: The image frame is a 3D image; the image detection module is used to obtain several facial features in the current image frame, including: The coordinates of the facial feature points determined in the 3D image are mapped to coordinates in a 2D coordinate system.

7. The apparatus according to claim 5, wherein the clustering module performs clustering of facial feature points in the clustering space based on the displacement vector, specifically comprising: The facial feature points in the clustering space are clustered according to the direction angle of the displacement vector, wherein the direction angle is not greater than a preset range of variation.

8. The apparatus according to claim 7, characterized in that: The preset range of variation is 20 degrees.

9. A device for stabilizing facial images, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to perform the operation as described in any one of claims 1-4.

10. A computer-readable storage medium storing computer instructions thereon, characterized in that: When executed by the processor, this instruction performs the operation described in any one of claims 1-4.

Images