Facial tracking method, device and storage medium

Abstract

The present application provides a facial tracking method, device and storage medium, which includes: receiving multiple image frames, the multiple image frames including the face of a target person, obtaining the posture data of the target person's face in two adjacent image frames, obtaining the change in the posture data of the target person's face in the two adjacent image frames, and then adjusting the key point positions of the target person's face in the latter image frame of the two adjacent image frames based on the change in the posture data of the target person's face in the two adjacent image frames, so as to improve the detection and tracking effect of the facial key points.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to a face tracking method, device, and storage medium. Background Art

[0002] Facial detection and tracking involves detecting key facial points in a video stream or image frame sequence and then capturing their locations in subsequent frames. It is widely used in video and teleconferencing, live broadcasts, short video recording, airport security, and other fields. Currently, existing facial detection and tracking solutions consume excessive computing resources, have high equipment requirements, and perform poorly. Summary of the Invention

[0003] The embodiments of the present application provide a facial tracking method, device, and storage medium to improve the detection and tracking effect of facial key points.

[0004] A first aspect of an embodiment of the present application provides a face tracking method, comprising:

[0005] receiving a plurality of image frames, wherein the plurality of image frames include a face of a target person;

[0006] Obtaining a change in the pose data of the target person's face in a first image frame relative to a second image frame; the first image frame is any image frame other than the first frame among the multiple image frames, and the second image frame is an image frame previous to the first image frame;

[0007] According to the change in the posture data, the key points of the target person's face in the first image frame are adjusted.

[0008] In an optional embodiment of the first aspect of the present application, the change in the pose data of the target person's face includes a change in three-dimensional pose data; and obtaining the change in the pose data of the target person's face in the first image frame relative to the second image frame includes:

[0009] Acquire three-dimensional posture data of the target person's face in the first image frame and the second image frame;

[0010] The three-dimensional posture data change is determined based on the three-dimensional posture data of the target person's face in the first image frame and the second image frame.

[0011] In an optional embodiment of the first aspect of the present application, obtaining three-dimensional posture data of the target person's face in the first image frame and the second image frame includes:

[0012] Acquire first key point detection data of the target person's face in the first image frame, and determine three-dimensional posture data of the target person's face in the first image frame based on the first key point detection data;

[0013] Acquire second key point detection data of the target person's face in the second image frame, and determine three-dimensional pose data of the target person's face in the second image frame based on the second key point detection data.

[0014] In an optional embodiment of the first aspect of the present application, the method further includes:

[0015] Obtaining the area ratio of the target person's face in the first image frame;

[0016] Determining a first parameter according to an area ratio of the target person's face in the first image frame;

[0017] The first parameter is used to adjust the change of the posture data, and the first parameter is negatively correlated with the area ratio of the target person's face in the first image frame.

[0018] In an optional embodiment of the first aspect of the present application, obtaining the change in pose data of the target person's face in the first image frame relative to the second image frame includes:

[0019] Adjusting the change in the pose data of the target person's face in the first image frame relative to the second image frame by using the first parameter to obtain the adjusted pose data change;

[0020] Adjusting the key point positions of the target person's face in the first image frame according to the change in the posture data includes: adjusting the key point positions of the target person's face in the first image frame according to the change in the adjusted posture data.

[0021] In an optional embodiment of the first aspect of the present application, adjusting the key point positions of the target person's face in the first image frame according to the change in the posture data includes:

[0022] If the change in the posture data of the target person's face in at least two dimensions is less than a threshold, adjusting the key points of the target person's face in the first image frame using a first filter; or

[0023] If the change in the pose data of the target person's face in at least two dimensions is greater than or equal to the threshold, adjusting the key points of the target person's face in the first image frame using a second filter;

[0024] The first filter and the second filter have different adjustment ranges for the key point positions on the target person's face.

[0025] In an optional embodiment of the first aspect of the present application, the method further includes:

[0026] Obtaining the serial number of the first image frame;

[0027] If the number is greater than a preset number value, triggering facial detection frame recognition for the first image frame;

[0028] Key point detection data of the target person's face in the first image frame is determined based on the facial detection frame recognition result.

[0029] In an optional embodiment of the first aspect of the present application, the method further includes:

[0030] If the number is less than or equal to the preset number value, the first key point detection data of the target person's face in the first image frame is determined based on the facial detection result of the second image frame and the first image frame; the facial detection result of the second image frame is used to indicate the position of the target person's face in the second image frame.

[0031] A second aspect of the embodiments of the present application provides a face tracking method, comprising:

[0032] Receiving multiple image frames captured by a camera of a terminal device, wherein the multiple image frames include a face of a target person;

[0033] Adjusting the key point positions of the target person's face in a later image frame of the two adjacent image frames according to a change in the pose data of the target person's face in two adjacent image frames of the plurality of image frames;

[0034] Based on the adjusted key point positions of the target person's face, the object selected by the user is displayed in the multiple image frames.

[0035] Optionally, the object selected by the user includes an animated avatar or a beauty patch.

[0036] A third aspect of the embodiments of the present application provides an object tracking method, including:

[0037] receiving a plurality of image frames, wherein the plurality of image frames include a target object;

[0038] Acquire a change in the pose data of the target object in a first image frame relative to a second image frame; the first image frame is any image frame other than the first frame among the multiple image frames, and the second image frame is an image frame previous to the first image frame;

[0039] Adjust key point positions of the target object in the first image frame according to changes in the posture data.

[0040] A fourth aspect of the embodiments of the present application provides a facial tracking device, comprising:

[0041] A receiving module, configured to receive a plurality of image frames, wherein the plurality of image frames include a face of a target person;

[0042] an acquisition module, configured to acquire a change in the posture data of the target person's face in a first image frame relative to a second image frame; the first image frame being any image frame other than the first frame among the plurality of image frames, and the second image frame being an image frame preceding the first image frame;

[0043] A processing module is used to adjust the key points of the target person's face in the first image frame according to the change of the posture data.

[0044] A fifth aspect of the embodiments of the present application provides a facial tracking device, comprising:

[0045] A receiving module, configured to receive a plurality of image frames captured by a camera of a terminal device, wherein the plurality of image frames include a face of a target person;

[0046] a processing module, configured to adjust the key point positions of the target person's face in a later image frame of the two adjacent image frames according to a change in the pose data of the target person's face in two adjacent image frames of the plurality of image frames;

[0047] The display module is used to display the object selected by the user in the multiple image frames based on the adjusted key point positions of the target person's face.

[0048] A sixth aspect of the embodiments of the present application provides an object tracking device, including:

[0049] A receiving module, configured to receive a plurality of image frames, wherein the plurality of image frames include a target object;

[0050] an acquisition module, configured to acquire a change in the pose data of the target object in a first image frame relative to a second image frame; the first image frame being any image frame other than the first frame among the plurality of image frames, and the second image frame being an image frame preceding the first image frame;

[0051] A processing module is used to adjust the key point position of the target object in the first image frame according to the change of the posture data.

[0052] The seventh aspect of the embodiments of the present application provides an electronic device, comprising: a memory, a processor, and a computer program; the computer program is stored in the memory and is configured to be executed by the processor to implement the method described in any one of the first aspects of the present application, or the method described in the second aspect of the present application, or the method described in the third aspect of the present application.

[0053] An eighth aspect of the embodiments of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the method described in any one of the first aspects of the present application, or the method described in the second aspect of the present application, or the method described in the third aspect of the present application.

[0054] A ninth aspect of the embodiments of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method described in any one of the first aspects of the present application, or the method described in the second aspect of the present application, or the method described in the third aspect of the present application.

[0055] An embodiment of the present application provides a facial tracking method, device and storage medium, which includes: receiving multiple image frames, the multiple image frames including the face of a target person, obtaining the posture data of the target person's face in two adjacent image frames, obtaining the change in the posture data of the target person's face in the two adjacent image frames, and then adjusting the key point positions of the target person's face in the latter image frame of the two adjacent image frames based on the change in the posture data of the target person's face in the two adjacent image frames, so as to improve the detection and tracking effect of the facial key points. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A schematic diagram of an application scenario of the face tracking method provided in an embodiment of the present application;

[0057] Figure 2 Schematic diagram of the face tracking method provided in this embodiment Figure 1 ;

[0058] Figure 3 A schematic diagram of three-dimensional posture data of a person's face provided in an embodiment of the present application;

[0059] Figure 4 A schematic diagram of an image frame in an image frame sequence provided in an embodiment of the present application;

[0060] Figure 5 Schematic diagram of the face tracking method provided in this embodiment Figure 2 ;

[0061] Figure 6 Schematic diagram of the face tracking method provided in this embodiment Figure 3 ;

[0062] Figure 7 Schematic diagram of the face tracking method provided in this embodiment Figure 4 ;

[0063] Figure 8 Schematic diagram of the structure of the face tracking device provided in the embodiment of the present application Figure 1 ;

[0064] Figure 9 Schematic diagram of the structure of the face tracking device provided in the embodiment of the present application Figure 2 ;

[0065] Figure 10 This is a hardware structure diagram of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0067] The terms "first," "second," and the like in the description, claims, and accompanying drawings of the embodiments of the present application are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, such that the embodiments of the present application described herein can be practiced in an order other than that illustrated or described herein.

[0068] It should be understood that the terms "include" and "have" and any variations thereof as used herein are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, product or apparatus.

[0069] In the description of the embodiments of the present application, the term "corresponding" may indicate a direct or indirect correspondence between the two, or an association relationship between the two, or a relationship between indication and being indicated, configuration and being configured, etc.

[0070] There are numerous facial landmark tracking solutions in the industry, each tailored to the specific business scenario and algorithm capabilities. Currently, facial landmark tracking suffers from the following major issues: even when the face is essentially still, the output facial landmark positions can exhibit subtle, irregular jitter. Especially in the case of rapid facial movement, the output landmark positions can become misaligned with the face, resulting in poor detection and tracking results.

[0071] In response to the above problems, the embodiment of the present application proposes a facial tracking method to correspond to the jitter of facial key points and improve the detection and tracking effect of facial key points. This application takes into account the user groups and algorithm capabilities, and designs an algorithm system suitable for most devices (including some mid- and low-end devices, such as elderly mobile phones) based on the consumption of computing resources and actual anti-shake of edge devices (such as smartphones, tablets, etc.). In the process of facial key point detection and tracking, efficient numerical calculation methods are used, and complex operations involving convolutional neural networks are not introduced. It supports the use of various devices and supports running on more platforms (such as short video platforms, picture platforms, live broadcast platforms, etc.) with lower resource consumption.

[0072] The facial tracking method proposed in this application designs and introduces a pose checker and filters with different filtering rules. The pose checker analyzes and determines the speed of change in the three-dimensional pose of a person's face between adjacent image frames in a video stream. It then matches the corresponding filters, adjusts key facial points, accurately detects facial points, and stably tracks faces in continuous video streams. Based on this solution, applying beauty algorithms and various accessories or animations to a person's face can also be precisely linked to them, providing a better user experience.

[0073] The following first briefly introduces the application scenarios of the technical solutions provided in the embodiments of the present application.

[0074] Figure 1 Schematic diagram of the application scenario of the face tracking method provided in the embodiment of the present application. Figure 1 As shown, the application scenario of this embodiment includes a terminal device 101 and a server 102, and the terminal device 101 is in communication with the server 102. The terminal device 101 obtains a video stream or an image frame sequence through an image acquisition device, uploads the obtained video stream or image frame sequence to the server 102, and performs facial tracking of a person on the video stream or image frame sequence in the server 102. In this scenario, the server 102 includes a facial tracking device, and the facial tracking device can execute the facial tracking method provided in the embodiment of the present application. Optionally, in some embodiments, the server includes an object tracking device (not shown in the figure), and the object tracking device can execute the object tracking method provided in the embodiment of the present application.

[0075] Optionally, the server 102 may return the character face tracking result to the terminal device 101, where the character face tracking result includes facial key point detection data for each frame image in the video stream or image frame sequence, and the terminal device 101 visually displays the character face tracking result.

[0076] Optionally, based on the facial tracking result, the server 102 superimposes an animation, sticker, makeup, virtual avatar, etc. on each frame in the video stream or image frame sequence to obtain a facial processing result of the person. The server 102 can return the facial processing result of the person to the terminal device 101.

[0077] The terminal device 101 of this embodiment can be any electronic device with a shooting function, including but not limited to smartphones, laptops, tablets, smart car devices, smart wearable devices, smart screens, smart makeup mirrors, etc.

[0078] The server 102 of this embodiment can be a common server or a cloud server. A cloud server, also known as a cloud computing server or cloud host, is a host product in a cloud computing service system. The server 102 can also be a server in a distributed system or a server integrated with a blockchain.

[0079] It should be noted that the technical solution provided in this application can be applied to a server or a terminal device, and part of the processing can be performed by the terminal device and the other part can be performed by the server. For example, the part of the character face tracking algorithm that involves a large amount of calculations of the central processing unit CPU and the image processing unit GPU can be executed on the server side. This embodiment does not impose any restrictions on this.

[0080] The following are some specific examples to illustrate the application scenarios of the human face tracking solution.

[0081] In a possible scenario, a user records a short video through a terminal device. During the recording process, if the user turns on beauty mode, facial key points need to be detected and tracked to ensure that the beauty patches are accurately rendered to the corresponding positions on the user's face. For example, the lipstick patch is rendered to the user's mouth, and the eyelash patch is rendered to the upper and lower eyelashes of the user's eyes.

[0082] In one possible scenario, a user broadcasts live online through a terminal device. During the live broadcast, if the user turns on beauty mode, facial key points need to be detected and tracked to ensure that the beauty patch is accurately rendered to the corresponding position on the user's face.

[0083] In one possible scenario, a user is participating in a video conference via a terminal device. During the video conference, if the user does not want their appearance to be publicly displayed, they can use an animated avatar, such as replacing the user's entire head with an animal or cartoon avatar. This process requires the detection and tracking of facial key points to ensure that the animated avatar's facial expressions align with those of the user.

[0084] Other human face tracking scenarios similar to the above scenarios all fall within the protection scope of the technical solution of this application.

[0085] Based on the above application scenarios, the following detailed description of the human face tracking solution provided by the embodiment of the present application is provided through specific embodiments. It should be noted that the technical solution provided by the embodiment of the present application may include part or all of the following contents, and the following specific embodiments may be combined with each other. For the same or similar concepts or processes, some embodiments may not be repeated.

[0086] Figure 2 Schematic diagram of the face tracking method provided in this embodiment Figure 1 The face tracking method of this embodiment can be applied to Figure 1 The terminal device or server shown in the figure is not limited in this embodiment. Figure 2 As shown, the face tracking method includes:

[0087] Step 201: Receive multiple image frames, where the multiple image frames include the face of a target person.

[0088] Step 202: Obtain the change in the pose data of the target person's face in the first image frame relative to the second image frame.

[0089] The first image frame is any image frame except the first frame among the multiple image frames, and the second image frame is the previous image frame of the first image frame.

[0090] In one embodiment, a terminal device receives a sequence of image frames (i.e., multiple image frames) from an image acquisition device (e.g., a camera of the terminal device), and at least some of the image frames in the sequence include facial features of an unspecified person. For example, there are no people in the first 10 frames of the sequence, person A appears in frames 11 through 30, and newly added persons B and C appear in frames 31 through 50.

[0091] In one possible implementation, a terminal device receives a sequence of image frames and sends the sequence of image frames to a server, so that the server performs face detection and tracking on the sequence of image frames. In one possible implementation, a terminal device receives a sequence of image frames and performs face detection and tracking on the sequence of image frames.

[0092] For ease of understanding, the following solution is described using an image frame sequence including a specific target person as an example. For face tracking of multiple target persons in an image frame sequence, it is only necessary to perform face detection and tracking separately.

[0093] Assuming that each frame of the image frame sequence includes the face of the target person, the following steps are described using any two adjacent image frames in the image frame sequence.

[0094] In the process of facial tracking of an image frame sequence, three-dimensional posture data of the target person's face in the first image frame and three-dimensional posture data of the target person's face in the second image frame are obtained respectively; based on the three-dimensional posture data of the target person's face in the first image frame and the second image frame, the three-dimensional posture data change is determined.

[0095] Figure 3 This is a schematic diagram of the three-dimensional posture data of the human face provided in the embodiment of the present application. Figure 3 As shown, the 3D pose data of the target person's face in an image frame includes three-dimensional pose data: pitch, yaw, and roll. The 3D pose data change indicates the difference in the three-dimensional pose data of the target person's face in adjacent image frames, namely, the difference in the pitch, yaw, and roll angles of the target person's face in adjacent image frames.

[0096] Step 203: Adjust the key point positions of the target person's face in the first image frame according to the change in the posture data of the target person's face in the first image frame relative to the second image frame.

[0097] From the above steps, it can be seen that the change in the pose data of the target person's face in the first image frame relative to the second image frame includes three-dimensional pose data changes. Based on the three-dimensional pose data changes, the key point positions of the target person's face in the first image frame can be adjusted as follows:

[0098] In one possible implementation, if the change in the pose data of the target person's face in at least two dimensions is less than a threshold, the positions of the key points of the target person's face in the first image frame are adjusted using a first filter. That is, if the change in the pose data of the target person's face in any two of the three dimensions is less than a threshold, the positions of the key points of the target person's face in the first image frame are adjusted using the first filter. Alternatively, if the change in the pose data of the target person's face in all three dimensions is less than a threshold, the positions of the key points of the target person's face in the first image frame are adjusted using the first filter.

[0099] In one possible implementation, if the change in the target person's facial pose data in at least two dimensions is greater than or equal to a threshold, the key points of the target person's face in the first image frame are adjusted using a second filter. That is, if the change in the pose data in any two of the three dimensions is greater than or equal to a threshold, the key points of the target person's face in the first image frame are adjusted using a second filter. Alternatively, if the change in the pose data in all three dimensions is greater than or equal to a threshold, the key points of the target person's face in the first image frame are adjusted using a second filter.

[0100] In the above embodiment, the threshold is used to determine how fast the posture data of the target person's face changes in three dimensions, thereby determining which filter to use to adjust the key point positions of the target person's face in the first image frame.

[0101] In this embodiment, the functions of the first filter and the second filter are to combine the three-dimensional posture changes of the target person's face in the first image frame relative to the second image frame, predict the key point positions of the target person's face in the first image frame, and reduce the jitter of the key point positions of the target person's face between adjacent image frames.

[0102] It should be noted that the first filter and the second filter adjust the key point positions of the target person's face to different degrees. The first filter, also known as the low-speed filter, is used to predict the key point positions of the target person's face whose posture data between adjacent image frames has little change. The second filter, also known as the high-speed filter, is used to predict the key point positions of the target person's face whose posture data between adjacent image frames has significant change.

[0103] The facial tracking method shown in the embodiment of the present application, in the process of facial tracking of an image frame sequence, first obtains the posture data of the target person's face in two adjacent image frames, obtains the change in the posture data of the target person's face in the two adjacent image frames, and then adjusts the position of the key points of the target person's face in the first image frame according to the change in the posture data of the target person's face in the two adjacent image frames, so as to reduce the irregular jitter of the facial key points and improve the detection and tracking effect of the facial key points.

[0104] To facilitate understanding, the following formula illustrates how to adjust facial key points in the first image frame based on changes in pose data. Assuming the threshold is θ, we first use the solvePNP function to obtain the 3D pose data of the target person's face in the image frame. This 3D pose data can be expressed as: pose = {pitch, yaw, roll}.

[0105] Among them, pitch represents the pitch angle, which rotates around the x-axis; yaw represents the yaw angle, which rotates around the y-axis; roll represents the roll angle, which rotates around the z-axis. For details, please refer to Figure 3 .

[0106] The three-dimensional pose data of the target person's face in the first image frame can be recorded as: pose now ={pitch now ,yaw now ,roll now}.

[0107] The three-dimensional pose data of the target person's face in the second image frame can be recorded as: pose pre ={pitch pre,yaw pre ,roll pre}.

[0108] The change in the three-dimensional pose data of the target person's face in the first image frame relative to the second image frame can be recorded as:

[0109] pose diff ={p diff, y diff ,r diff}, where p diff =│pitch now -pitch pre │,y diff =│yaw now -yaw pre │,r diff＝ │roll now -roll pre │}.

[0110] One possible case is that if {p diff, y diff ,r diff If the values ​​of any two of} are less than θ, it is determined that the target person's face is in "slow motion" and the key point positions of the target person's face in the first image frame can be adjusted by a first filter, i.e., a low-speed filter.

[0111] One possible case is that if {p diff, y diff ,r diff If the three values ​​in} are all less than θ, it is determined that the target person's face is in "slow motion" and the key point positions of the target person's face in the first image frame can be adjusted through the first filter, that is, the low-speed filter.

[0112] One possible case is that if {p diff, y diff ,r diff If the values ​​of any two of} are greater than or equal to θ, the target person's face is determined to be "fast moving", and the key point positions of the target person's face in the first image frame can be adjusted by a second filter, i.e., a high-speed filter.

[0113] One possible case is that if {p diff, y diff ,r diff If the three values ​​in} are all greater than or equal to θ, it is determined that the target person's face is "moving quickly" and the second filter, i.e., the high-speed filter, can be used to adjust the key point positions of the target person's face in the first image frame.

[0114] Based on the above embodiment, considering that for facial shaking of the same amplitude, such as turning or shaking the head, the target person's face occupies different sizes of the entire screen, the device has different sensitivities in detecting facial key point shaking. Figure 4 This is a schematic diagram of a certain image frame in the image frame sequence provided in an embodiment of the present application. Figure 4 As shown, if adjacent image frames in an image frame sequence include two persons, person A and person B, person A is located at the front of the image, and person B is located behind person A. By obtaining the face detection frame of person A and the face detection frame of person B, the proportion of person A's face in the entire image and the proportion of person B's face in the entire image are determined respectively. Figure 4 The face of person A occupies a larger proportion of the entire screen than the face of person B.

[0115] based on Figure 4 How to optimize the face tracking solution to balance the face detection and tracking effects of different screen proportions is an urgent problem to be solved. Figure 4 The face tracking optimization scheme is described in detail.

[0116] Figure 5 Schematic diagram of the face tracking method provided in this embodiment Figure 2 The face tracking method of this embodiment can be applied to Figure 1 The terminal device or server shown is not limited in this embodiment.

[0117] For ease of understanding, this embodiment uses an image frame including a specific target person as an example to illustrate the solution. Figure 5 As shown, the face tracking method includes:

[0118] Step 501: Acquire three-dimensional posture data of the target person's face in a first image frame and a second image frame.

[0119] In an alternative embodiment of this embodiment, first key point detection data of the target person's face in a first image frame is obtained, and three-dimensional pose data of the target person's face in the first image frame is determined based on the first key point detection data. Similarly, second key point detection data of the target person's face in a second image frame is obtained, and three-dimensional pose data of the target person's face in the second image frame is determined based on the second key point detection data.

[0120] The key point detection data for the target person's face in the image frame includes multiple key points of facial features such as eyebrows, eyes, nose, mouth, and chin. Each facial feature includes multiple key points. For example, eye key points include multiple key points of the outer corners of the eyes, the beginnings of the eyes, the upper eyelids, and the lower eyelids. Mouth key points include multiple key points of the upper and lower lips. Chin key points include multiple key points that outline the chin contour.

[0121] In an alternative embodiment of this embodiment, the first image frame is input into a face detector to obtain a face detection frame of the target person's face in the first image frame; then, the image block corresponding to the face detection frame of the target person's face in the first image frame is input into a facial key point detector to obtain key point detection data of the target person's face in the first image frame, i.e., the first key point detection data described above. The facial key point detection process for other image frames can refer to this embodiment.

[0122] In an optional embodiment of this embodiment, the facial detection result of the target person's face in the second image frame and the first image frame are input into the facial key point detector to obtain the key point detection data of the target person's face in the first image frame. The facial detection result of the target person's face in the second image frame is used to indicate the position of the target person's face in the second image frame. In this embodiment, the facial detection result of the target person's face in the second image frame is used to assist the facial key point detector in preliminarily locating the target person's face in the first image frame. On the one hand, it can improve the computational efficiency of the tracking device and increase the overall processing speed; on the other hand, since the detection result of the previous frame is reused, the facial key point detection results between two adjacent image frames will not have a large difference, thereby reducing the irregular jitter of the person's facial points.

[0123] In an alternative embodiment of this embodiment, key point detection data of the target person's face in the second image frame, i.e., the aforementioned second key point detection data, is obtained from the facial detection result of the target person's face in the second image frame. It can be seen that the facial detection result of the target person's face in the second image frame not only indicates the position of the target person's face in the second image frame, but also may indicate the key point detection data of the target person's face in the second image frame.

[0124] It should be noted that the aforementioned face detector is used to detect whether an image frame contains a human face and the approximate location of the human face. The aforementioned facial key point detector is used to detect the locations of key points on the human face in the image frame. This embodiment does not impose any restrictions on the detection algorithms of the face detector and the facial key point detector.

[0125] In an optional embodiment of this embodiment, after obtaining the three-dimensional pose data of the target person's face in the first image frame and the second image frame, the three-dimensional pose estimation solvePNP function can be used to obtain the three-dimensional pose data of the target person's face in the first image frame and the second image frame. Wherein, the input of the solvePNP function includes the key point detection data of the target person's face in the first image frame or the second image frame, and the output of the solvePNP function includes the three-dimensional pose data of the target person's face in the first image frame or the second image frame. That is, the coordinate data of multiple key points of the target person's face in the image frame, that is, the key point positions of the target person's face in the image coordinate system, are used as the input of the solvePNP function, and the solvePNP function calculates the three-dimensional pose data of the target person's face in the world coordinate system through coordinate transformation.

[0126] Step 502: Obtain the area ratio of the target person's face in the first image frame.

[0127] In this embodiment, the first image frame can be input into a face detector to obtain a face detection frame of the target person's face in the first image frame. Based on the face detection frame of the target person's face in the first image frame, the area ratio of the target person's face in the first image frame can be obtained. The area ratio of the target person's face in the image frame is the ratio of the pixel size of the target person's face detection frame to the overall pixel size of the image frame.

[0128] Step 503: Determine a first parameter based on the area ratio of the target person's face in the first image frame.

[0129] The first parameter is negatively correlated with the target person's facial area within the image frame; that is, the larger the facial area, the smaller the first parameter. The first parameter is used to adjust the positional data of the target person's face in the first image frame relative to the second image frame, specifically adjusting the difference in the three-dimensional positional data of the target person's face between two adjacent image frames.

[0130] In this embodiment, the first parameter is also called a hyperparameter, and its value range is (0,1). Its function is to adjust the different sensitivities of the facial three-dimensional posture data obtained due to the different sizes of the person's face occupying the entire screen. For example, if the person's face occupies a large proportion of the entire screen, that is, the distance between the person's face and the camera is very close, then a slight turn of the head will be quickly fed back in the three-dimensional posture data, the change is very fast, and the sensitivity is very high. However, if the person's face occupies a small proportion of the entire screen, that is, the distance between the person's face and the camera is far, then a slight turn of the head will not be obviously fed back in the three-dimensional posture data, and the sensitivity is low. Low sensitivity (or insensitivity) means that when the proportion of the person's face in this screen is too small, the three-dimensional posture data of the face will become "sluggish" during movement. For this reason, the three-dimensional posture data needs to be adjusted to make it sensitive.

[0131] In order to simultaneously utilize the three-dimensional posture data in images of both distant and near people, this embodiment designs hyperparameters to reduce the problem of inconsistent changes in the three-dimensional posture data caused by the different proportions of the person's face occupying the entire image. The changes in the three-dimensional posture data include the angle values ​​of three dimensions, namely pitch angle, yaw angle, and roll angle.

[0132] It should be noted that there are many ways to obtain hyperparameters. A value can be obtained based on the proportion of the character's face to the entire screen, or hyperparameters can be obtained according to certain rules or by adding them to network training. This embodiment does not impose any restrictions on this. Hyperparameters can be obtained in the following ways:

[0133] In an optional embodiment, the first parameter corresponding to the area ratio of the target person's face in the first image frame is determined by the correspondence between the preset hyperparameter and the person's face area ratio. The correspondence can be stored in a table format.

[0134] In an optional embodiment, a first parameter corresponding to the area ratio of the target person's face in the first image frame is determined by a preset function. The input of the preset function is the area ratio of the target person's face in the image frame, and the output of the preset function is the corresponding hyperparameter.

[0135] In one optional embodiment, a first parameter corresponding to the area ratio of the target person's face in the first image frame is determined using a pretrained network. The input of the pretrained network is the area ratio of the target person's face in the image frame, and the output of the pretrained network is the corresponding hyperparameter. This embodiment does not specifically limit the structure of the network.

[0136] Step 504: Adjust the change in the pose data of the target person's face in the first image frame relative to the second image frame using the first parameter to obtain the adjusted pose data change.

[0137] Step 505: Adjust the key points of the target person's face in the first image frame according to the adjusted posture data. This step is similar to the implementation principle of step 203 in the above embodiment, and can be referred to the above embodiment.

[0138] For ease of understanding, the following describes how to adjust the pose data changes and how to adjust the facial key points of the first image frame based on the adjusted pose data changes in combination with formulas.

[0139] Assuming the threshold is θ, firstly, based on the solvePNP function, the three-dimensional pose data of the target person's face in the first image frame and the second image frame are obtained, which are respectively recorded as pose now ={pitch now ,yawnow ,roll now} and pose pre ={pitch pre ,yaw pre ,roll pre Secondly, obtain the three-dimensional pose data change of the target person's face in the first image frame relative to the second image frame, which is recorded as pose diff ={p diff, y diff ,r diff}, where p diff =│pitch now -pitch pre │,y diff =│yaw now -yaw pre │,r diff＝ │roll now -roll pre │}.

[0140] and Figure 2 The difference between the embodiments is that this embodiment determines a hyperparameter based on the area ratio of the target person's face in the first image frame. Assuming that the hyperparameter is recorded as a, the three-dimensional pose data change of the target person's face in the first image frame relative to the second image frame is adjusted by the hyperparameter, and the adjusted three-dimensional pose data change can be obtained, which is recorded as:

[0141] pose diff `={a×p diff, a×y diff ,a×r diff}.

[0142] One possible case is that if {a×p diff, a×y diff ,a×r diff If the values ​​of any two of} are less than θ, it is determined that the target person's face is in "slow motion" and the key point positions of the target person's face in the first image frame can be adjusted by a first filter, i.e., a low-speed filter.

[0143] One possible case is that if {a×p diff, a×y diff ,a×r diff If the three values ​​in} are all less than θ, it is determined that the target person's face is in "slow motion" and the key point positions of the target person's face in the first image frame can be adjusted through the first filter, that is, the low-speed filter.

[0144] One possible case is that if {a×p diff, a×y diff ,a×r diffIf the values ​​of any two of} are greater than or equal to θ, the target person's face is determined to be "fast moving", and the key point positions of the target person's face in the first image frame can be adjusted by a second filter, i.e., a high-speed filter.

[0145] One possible case is that if {a×p diff, a×y diff ,a×r diff If the three values ​​in} are all greater than or equal to θ, it is determined that the target person's face is "moving quickly" and the second filter, i.e., the high-speed filter, can be used to adjust the key point positions of the target person's face in the first image frame.

[0146] The face tracking method shown in the embodiment of the present application, in the process of face tracking of an image frame sequence, first obtains the posture data of the target person's face in two adjacent image frames, obtains the posture data change of the target person's face in the two adjacent image frames, then determines the hyperparameter according to the area ratio of the target person's face in the first image frame, adjusts the posture data change of the target person's face in the two adjacent image frames based on the hyperparameter, and finally adjusts the key point position of the target person's face in the first image frame according to the adjusted posture data change. On the one hand, the hyperparameter is used to reduce the problem of inconsistent posture data change amplitude caused by different facial area ratios; on the other hand, different adjustment methods, such as different filters, are used based on different posture data changes to obtain more stable facial key point positions, thereby reducing the irregular jitter of the person's face in adjacent image frames and improving the detection and tracking effect of facial key points.

[0147] Figure 6 Schematic diagram of the face tracking method provided in this embodiment Figure 3 The face tracking method of this embodiment can be applied to Figure 1 The terminal device or server shown in the figure is not limited in this embodiment. Figure 6 As shown, the face tracking method includes:

[0148] Step 601: Get each frame of the video stream.

[0149] Step 602: Input the current picture of the video stream into a face detector to determine whether a human face is detected in the current picture.

[0150] If it is determined that a human face is detected in the current image, step 603a is executed; if it is determined that no human face is detected in the current image, step 603b is executed.

[0151] The face detector is used to detect whether there is a human face in the current image and the approximate location of the human face. If the face detector detects a human face, it will output a face detection frame. Figure 6As shown, the faces of persons A, B, and C are detected in the current picture, and the face detector outputs the detection frames corresponding to the faces of persons A, B, and C.

[0152] In one possible case, if the current picture is the first frame of the video stream, it directly enters the face detector.

[0153] In one possible scenario, if the current image is a frame after the first frame, a preset triggering strategy may be used to determine whether to enter the face detector. The triggering strategy sets a period for triggering the face detector, i.e., triggering the face detector every N frames, where N is a positive integer.

[0154] For example, assuming that the face detector is triggered every 30 frames, the entry logic is that the first frame triggers the face detector, the 31st frame triggers the face detector again, and so on, the 1+30N frame triggers the face detector once.

[0155] For example, if the current image is the second frame of the video stream, the facial key point detector can be directly entered to perform key point detection on the face of the person in the second frame. It should be noted that when performing key point detection on the face of the person in the second frame, the facial detection result (including the facial detection frame) of the previous frame (i.e., the first frame) and the second frame can be simultaneously input into the facial key point detector to perform facial key point detection. The facial detection result of the previous frame is used to assist the facial key point detector in performing preliminary positioning of the face of the person in the current image.

[0156] It should be understood that, under normal circumstances, 1s may correspond to, for example, 30 frames of pictures, and these 30 frames of pictures are likely to have the same number of people. Therefore, for 30 consecutive frames of pictures, there is no need to enter the face detector for face detection frame recognition every time, thereby reducing the device's computing power and improving the efficiency of face detection and tracking.

[0157] Step 603a: Input the current image containing the face detection frame into the facial key point detector to obtain key point detection data of the face of the person in the current image.

[0158] The facial key point detector is used to detect the position coordinates of facial key points in the current image.

[0159] Step 603b: Read the next frame of image from the video stream.

[0160] Step 604: perform facial identification matching on the current image.

[0161] In one possible scenario, if the current image is the first frame of the video stream, there are no existing facial identifiers. For example, if the faces of persons A, B, and C are detected in the current image, the faces can be numbered starting from 1, such that person A's face is numbered 1, person B's face is numbered 2, and person C's face is numbered 3. The numbers represent the facial identifiers of the persons.

[0162] In one possible scenario, if the current image is a frame after the first frame of the video stream and a certain number of facial identifiers exist, facial identifier matching can be used to determine whether the face in the current image is new or existing. For example, suppose the faces of persons A and B are existing, and the face of person C is newly added.

[0163] It should be noted that the correspondence between the current image and the face of the person in the previous frame can be determined through the human face matching algorithm. By comparing the facial detection frames in the two frames of the picture, facial matching is performed to determine the identification of the face of the person in the current image, thereby completing the human face identification matching process.

[0164] Based on the above situation, it can be seen that if the existing facial identification of the person is matched, step 605 is executed; if the existing facial identification of the person is not matched, step 607 is executed.

[0165] Step 605: Determine the speed of the facial movement of the person in the current image through a posture checker.

[0166] Step 606: Determine whether to use a fast filter or a slow filter based on the speed of the facial movement of the person in the current image, and obtain facial key point detection data processed by the filter.

[0167] For example, if the current picture is a frame picture after the first frame picture of the video stream, the face of person C in the current picture is a newly added face of a person, and the faces of persons A and B are existing faces of persons. For the face of person C, the facial key point detection data of person C is directly retained without any further processing, waiting to be merged with the facial detection results of persons A and B. For the faces of persons A and B, the facial key point detection data of persons A and B in the current picture are input into the posture checker, and the changes in the facial posture data of persons A and B in the current picture relative to the previous frame picture can be determined, that is, the speed of the facial movements of persons A and B can be determined. The determination of the speed can refer to the above embodiment and will not be expanded here. Figure 6As shown in the figure, to determine "fast motion" of person A's face, a fast filter can be used to predict the facial key point detection data of person A in the current image; to determine "slow motion" of person B's face, a slow filter can be used to predict the facial key point detection data of person B in the current image. The fast filter and the slow filter use different filtering parameters to output more stable facial key point detection data.

[0168] Step 607: Merge the face detection results of the current image and update the face identification of the person. The face detection results of this step include the face detection frame and the face key point detection data.

[0169] like Figure 6 As shown, the faces of characters A, B, and C are finally detected, among which the face of character C is a newly added character. The existing character facial identifiers are recorded and updated so as to perform corresponding matching on the new target in the next frame.

[0170] The face tracking method shown in this embodiment, in the process of face tracking a sequence of image frames containing a person's face, first determines whether the first image frame detects a person's face through a face detector. If a person's face is detected, it determines whether the person's face is a newly added person's face or an existing person's face. For a newly added person's face, the person's face identification is updated, and the key point detection data of the newly added person's face is recorded; for an existing person's face, the posture checker compares the changes in the three-dimensional posture data of the person's face in the first image frame relative to the second image frame (the previous image frame of the first image frame) to determine the speed of the person's face movement. If the person's face moves quickly, it enters a fast filter, and if the person's face moves slowly, it enters a slow filter. After being processed by the filter, more stable facial key point detection data is obtained. Finally, the processing results of each branch are merged and the processing of the next frame of image is entered. Based on the above scheme, the irregular jitter of the facial key points can be reduced, and the detection and tracking effect of the facial key points can be improved.

[0171] based on Figure 6 Example, in Figure 2 or Figure 5 Based on the illustrated embodiment, the face tracking method further includes:

[0172] In some embodiments, the number of the first image frame is obtained. If the number of the first image frame is greater than a preset number value (i.e., the cycle N that triggers entering the face detector, for example, N is 30), facial detection frame recognition of the first image frame is triggered.

[0173] In an optional embodiment, triggering facial detection frame recognition on the first image frame includes: inputting the first image frame into a facial detector for facial detection frame recognition, the facial detector being configured to output an approximate position of a person's face in the first image frame.

[0174] In some embodiments, the serial number of the first image frame is obtained. If the serial number of the first image frame is less than or equal to a preset serial number, first key point detection data of the target person's face in the first image frame is determined based on the facial detection result of the second image frame and the first image frame. The facial detection result of the second image frame is used to indicate the position of the target person's face in the second image frame, that is, to indicate the detection frame of the target person's face in the previous image frame.

[0175] In one optional embodiment, based on the facial detection results of the second image frame and the first image frame, determining the first key point detection data of the target person's face in the first image frame includes: inputting the facial detection results of the second image frame and the first image frame into a facial key point detector to determine the first key point detection data of the target person's face in the first image frame. In this embodiment, the first image frame is no longer input into the facial detector. Instead, the facial detection results of the second image frame are combined and analyzed by the facial key point detector to output the facial key point detection data of the first image frame, thereby avoiding unnecessary detection processing (such as detecting a face frame), thereby improving the efficiency of facial key point detection.

[0176] In some embodiments, key point detection data of the target person's face (the face of at least one person) in the first image frame is determined based on the facial detection frame recognition result.

[0177] In an optional embodiment, key point detection data of the target person's face in the first image frame is determined based on the facial detection frame recognition result, including: inputting the facial detection frame recognition result output by the facial detector and the first image frame into the facial key point detector to obtain the key point detection data of the person's face in the first image frame.

[0178] Based on the above embodiments, the facial tracking solution can be promoted as a software product to third-party platforms such as live streaming platforms, online office platforms, and beauty camera platforms. Third-party platforms can use this facial tracking solution to enhance the functionality of their own products, such as enhancing beauty effects in live video broadcasts and optimizing animated avatar effects in video conferencing.

[0179] In this regard, an embodiment of the present application also provides a facial tracking method that can be applied to terminal devices with display functions, such as smart phones, tablet computers, smart screens, etc.

[0180] For example, Figure 7 Schematic diagram of the face tracking method provided in this embodiment Figure 4 .like Figure 7 As shown, the face tracking method includes:

[0181] Step 701: Receive multiple image frames captured by a camera of a terminal device, where the multiple image frames include a face of a target person.

[0182] Step 702: Adjust the key point positions of the target person's face in the latter image frame of the two adjacent image frames according to the change in the posture data of the target person's face in two adjacent image frames of the plurality of image frames.

[0183] The implementation principle of this step can refer to any of the above method embodiments and will not be repeated here.

[0184] Step 703: Based on the adjusted key point positions of the target person's face, display the user-selected object in multiple image frames. The user-selected object includes an animated avatar or a beauty patch.

[0185] In an optional embodiment, the user opens the beauty camera application in the terminal device and retrieves multiple image frames including the user's face captured by the terminal device camera. The user can choose to add beauty patches in the shooting interface of the beauty camera application. For example, the user chooses to add eye patches, which may include eye shadow, eyelashes, eyeliner, etc.; for another example, the user chooses to add lip patches (i.e., lipstick patches); for another example, the user chooses one-click beauty, which means adding multiple beauty patches (including eye patches, lip patches, etc.) at the same time. Based on step 702, the terminal device adjusts the key point positions of the user's face in each frame of the image captured by the terminal device, and based on the adjusted facial key point positions, the beauty patches selected by the user are superimposed and displayed at the corresponding positions of the user's face in each frame of the image.

[0186] In an optional embodiment, a user opens a video conferencing application in a terminal device and retrieves multiple image frames including the user's face captured by the terminal device's camera. The user can select an animated avatar, such as an animal avatar or an anime avatar, in the shooting interface of the video conferencing application. The terminal device replaces the user's face in the image frame with the animated avatar selected by the user. The changes in the positions of the key points of the face in the animated avatar are determined based on the changes in the positions of the key points of the face of the user in the original image frame. Based on step 702, the terminal device adjusts the positions of the key points of the user's face in each frame of the image captured by the terminal device, determines the positions of the key points of the face in the animated avatar based on the adjusted positions of the key points of the face, and finally displays the animated avatar selected by the user at the position of the user's face in each frame of the image, and the changes in the facial expressions in the animated avatar are consistent with the changes in the facial expressions of the user.

[0187] The above embodiment shows a product application solution based on facial key point detection and tracking, which improves the beauty effect or animated avatar effect of the characters in the video screen by detecting and tracking the facial key points of the characters in the video screen.

[0188] The present invention also provides an object tracking method, which includes:

[0189] Step a: receiving a plurality of image frames, wherein the plurality of image frames include a target object;

[0190] Step b: obtaining a change in the pose data of the target object in the first image frame relative to the second image frame; the first image frame is any image frame other than the first frame among the multiple image frames, and the second image frame is an image frame previous to the first image frame;

[0191] Step c: adjusting the key point positions of the target object in the first image frame according to the changes in the posture data.

[0192] In this embodiment, the target object is not limited to a person, but can also be an animal, a virtual animated character, etc. The key points of the target object can be the key points of any part of the target object. Taking a person as an example, the detection part can be the face, head, legs, upper limbs, or the whole body of the person.

[0193] The above describes the face tracking method provided by the embodiment of the present application. The following describes the face tracking device provided by the embodiment of the present application.

[0194] In the embodiment of the present application, the facial tracking device can be divided into functional modules according to the above-mentioned method embodiment. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above-mentioned integrated modules can be implemented in the form of hardware or software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical functional division. In actual implementation, there may be other division methods. The following is an example of dividing each functional module according to each function.

[0195] Figure 8 Schematic diagram of the structure of the face tracking device provided in the embodiment of the present application Figure 1 .like Figure 8 As shown, the face tracking device 800 of this embodiment includes: a receiving module 801 , an acquiring module 802 and a processing module 803 .

[0196] The receiving module 801 is configured to receive a plurality of image frames, wherein the plurality of image frames include a face of a target person;

[0197] An acquisition module 802 is configured to acquire a change in the pose data of the target person's face in a first image frame relative to a second image frame; the first image frame is any image frame other than the first frame among the multiple image frames, and the second image frame is an image frame preceding the first image frame;

[0198] The processing module 803 is used to adjust the key points of the target person's face in the first image frame according to the change of the posture data.

[0199] In an optional embodiment of this embodiment, the change in the pose data of the target person's face includes a change in three-dimensional pose data; the acquisition module 802 is configured to acquire the three-dimensional pose data of the target person's face in the first image frame and the second image frame;

[0200] The processing module 803 is configured to determine a change in the three-dimensional posture data based on the three-dimensional posture data of the target person's face in the first image frame and the second image frame.

[0201] In an optional embodiment of this embodiment, the acquisition module 802 is used to obtain first key point detection data of the target person's face in the first image frame, and the processing module 803 is used to determine the three-dimensional posture data of the target person's face in the first image frame based on the first key point detection data;

[0202] The acquisition module 802 is used to obtain the second key point detection data of the target person's face in the second image frame, and the processing module 803 is used to determine the three-dimensional posture data of the target person's face in the second image frame based on the second key point detection data.

[0203] In an optional embodiment of this embodiment, the acquisition module 802 is configured to acquire an area ratio of the target person's face in the first image frame;

[0204] Processing module 803 is used to determine a first parameter based on the area ratio of the target person's face in the first image frame; the first parameter is used to adjust the change of the posture data, and the first parameter is negatively correlated with the area ratio of the target person's face in the first image frame.

[0205] In an optional embodiment of this embodiment, the acquisition module 802 is configured to adjust the pose data change of the target person's face in the first image frame relative to the second image frame using the first parameter to obtain the adjusted pose data change;

[0206] The processing module 803 is used to adjust the key points of the target person's face in the first image frame according to the adjusted posture data changes.

[0207] In an optional embodiment of this embodiment, the processing module 803 is configured to:

[0208] If the change in the posture data of the target person's face in at least two dimensions is less than a threshold, adjusting the key points of the target person's face in the first image frame using a first filter; or

[0209] If the change in the pose data of the target person's face in at least two dimensions is greater than or equal to the threshold, adjusting the key points of the target person's face in the first image frame using a second filter;

[0210] The first filter and the second filter have different adjustment ranges for the key point positions on the target person's face.

[0211] In an optional embodiment of this embodiment, the acquisition module 802 is configured to acquire the serial number of the first image frame;

[0212] If the number is greater than a preset number value, the processing module 803 is used to trigger facial detection frame recognition for the first image frame; and determine key point detection data of the target person's face in the first image frame based on the facial detection frame recognition result.

[0213] In an optional embodiment of this embodiment, if the number is less than or equal to the preset number value, the processing module 803 is used to determine the first key point detection data of the target person's face in the first image frame based on the facial detection result of the second image frame and the first image frame; the facial detection result of the second image frame is used to indicate the position of the target person's face in the second image frame.

[0214] The face tracking device provided in this embodiment can perform the above Figure 2 、 Figure 5 or Figure 6 The technical solution of the method embodiment shown has similar implementation principles and technical effects, which will not be repeated here.

[0215] Figure 9 Schematic diagram of the structure of the face tracking device provided in the embodiment of the present application Figure 2 .like Figure 9 As shown, the face tracking device 900 of this embodiment includes a receiving module 901 , a processing module 902 and a display module 903 .

[0216] A receiving module 901 is configured to receive a plurality of image frames captured by a camera of a terminal device, wherein the plurality of image frames include a face of a target person;

[0217] A processing module 902 is configured to adjust key point positions of the target person's face in a later image frame of the two adjacent image frames according to a change in the pose data of the target person's face in two adjacent image frames of the plurality of image frames;

[0218] The display module 903 is configured to display the object selected by the user in the plurality of image frames based on the adjusted key point positions of the target person's face.

[0219] Optionally, the object selected by the user includes an animated avatar or a beauty patch.

[0220] The face tracking device provided in this embodiment can perform the above Figure 7 The technical solution of the method embodiment shown has similar implementation principles and technical effects, which will not be repeated here.

[0221] The embodiment of the present application further provides an object tracking device, comprising: a receiving module, a processing module, and a processing module.

[0222] A receiving module, configured to receive a plurality of image frames, wherein the plurality of image frames include a target object;

[0223] an acquisition module, configured to acquire a change in the pose data of the target object in a first image frame relative to a second image frame; the first image frame being any image frame other than the first frame among the plurality of image frames, and the second image frame being an image frame preceding the first image frame;

[0224] A processing module is used to adjust the key point position of the target object in the first image frame according to the change of the posture data.

[0225] Figure 10 This is a hardware structure diagram of the electronic device provided in the embodiment of this application. Figure 10 As shown, the electronic device 1000 provided in this embodiment includes:

[0226] Memory 1001, processor 1002 and computer program; wherein, the computer program is stored in memory 1001 and is configured to be executed by processor 1002 to implement the technical solution of any of the aforementioned method embodiments, and its implementation principles and technical effects are similar and will not be repeated here.

[0227] Optionally, the memory 1001 may be independent or integrated with the processor 1002. When the memory 1001 is a device independent of the processor 1002, the electronic device 1000 further includes a bus 1003 for connecting the memory 1001 and the processor 1002.

[0228] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. The computer program is executed by the processor 1002 to implement the technical solution of any of the aforementioned method embodiments.

[0229] An embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the technical solution of any of the aforementioned method embodiments.

[0230] An embodiment of the present application provides a chip, including: a processing module and a communication interface, wherein the processing module can execute the technical solution of any of the aforementioned method embodiments.

[0231] Optionally, the chip also includes a storage module (such as a memory), the storage module is used to store instructions, the processing module is used to execute the instructions stored in the storage module, and the execution of the instructions stored in the storage module enables the processing module to execute the technical solution of any of the aforementioned method embodiments.

[0232] It should be understood that the processor described above may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), or application-specific integrated circuits (ASICs). A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0233] The memory may include a high-speed RAM memory, and may also include non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk.

[0234] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be classified into address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0235] The storage medium may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0236] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium can be located in an application-specific integrated circuit (ASIC). Of course, the processor and storage medium can also exist as discrete components in an electronic device.

[0237] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A facial tracking method, characterized in that: include: receiving a plurality of image frames, wherein the plurality of image frames include a face of a target person; Obtaining a change in the pose data of the target person's face in the first image frame relative to the second image frame; The first image frame is any image frame except the first frame among the multiple image frames, and the second image frame is an image frame previous to the first image frame; If the change in the posture data of the target person's face in at least two dimensions is less than a threshold, adjusting the key points of the target person's face in the first image frame using a first filter; or If the change in the posture data of the target person's face in at least two dimensions is greater than or equal to the threshold, the key point position of the target person's face in the first image frame is adjusted by the second filter; the first filter and the second filter have different adjustment amplitudes for the key point position of the target person's face; the first filter is used to predict the key point position of the target person's face whose posture data of adjacent image frames do not change much, and the second filter is used to predict the key point position of the target person's face whose posture data of adjacent image frames change greatly.

2. The method according to claim 1, characterized in that The change in the pose data of the target person's face includes a change in three-dimensional pose data; obtaining the change in the pose data of the target person's face in the first image frame relative to the second image frame, comprising: Acquire three-dimensional posture data of the target person's face in the first image frame and the second image frame; The three-dimensional posture data change is determined based on the three-dimensional posture data of the target person's face in the first image frame and the second image frame.

3. The method according to claim 2, characterized in that The acquiring of three-dimensional posture data of the target person's face in the first image frame and the second image frame includes: Acquire first key point detection data of the target person's face in the first image frame, and determine three-dimensional posture data of the target person's face in the first image frame based on the first key point detection data; Acquire second key point detection data of the target person's face in the second image frame, and determine three-dimensional pose data of the target person's face in the second image frame based on the second key point detection data.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Obtaining the area ratio of the target person's face in the first image frame; Determining a first parameter according to an area ratio of the target person's face in the first image frame; The first parameter is used to adjust the change of the posture data, and the first parameter is negatively correlated with the area ratio of the target person's face in the first image frame.

5. The method according to claim 4, characterized in that The acquiring of the change in the pose data of the target person's face in the first image frame relative to the second image frame includes: Adjusting the change in the pose data of the target person's face in the first image frame relative to the second image frame by using the first parameter to obtain the adjusted pose data change; Adjusting the key point positions of the target person's face in the first image frame according to the change in the posture data includes: adjusting the key point positions of the target person's face in the first image frame according to the change in the adjusted posture data.

6. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Obtaining the serial number of the first image frame; If the number is greater than a preset number value, triggering facial detection frame recognition for the first image frame; Key point detection data of the target person's face in the first image frame is determined based on the facial detection frame recognition result.

7. The method according to claim 6, characterized in that The method further comprises: If the number is less than or equal to the preset number value, the first key point detection data of the target person's face in the first image frame is determined based on the facial detection result of the second image frame and the first image frame; the facial detection result of the second image frame is used to indicate the position of the target person's face in the second image frame.

8. A facial tracking method, characterized in that: include: Receiving multiple image frames captured by a camera of a terminal device, wherein the multiple image frames include a face of a target person; Adjusting the key point positions of the target person's face in a later image frame of the two adjacent image frames according to a change in the pose data of the target person's face in two adjacent image frames of the plurality of image frames; Based on the adjusted key point positions of the target person's face, displaying the object selected by the user in the multiple image frames; The adjusting the key point positions of the target person's face in the latter image frame of the two adjacent image frames according to the change in the pose data of the target person's face in two adjacent image frames of the plurality of image frames comprises: If the change in the posture data of the target person's face in at least two dimensions is less than a threshold, adjusting the key point positions of the target person's face in the subsequent image frame using a first filter; or If the change in the posture data of the target person's face in at least two dimensions is greater than or equal to the threshold, the key point position of the target person's face in the subsequent image frame is adjusted by the second filter; the first filter and the second filter have different adjustment amplitudes for the key point position of the target person's face; the first filter is used to predict the key point position of the target person's face whose posture data of adjacent image frames does not change much, and the second filter is used to predict the key point position of the target person's face whose posture data of adjacent image frames change greatly.

9. The method according to claim 8, characterized in that The object selected by the user includes an animated avatar or a beauty patch.

10. An object tracking method, characterized in that: include: receiving a plurality of image frames, wherein the plurality of image frames include a target object; Acquire a change in the pose data of the target object in the first image frame relative to the second image frame; The first image frame is any image frame except the first frame among the multiple image frames, and the second image frame is an image frame previous to the first image frame; If a change in the posture data of the target object in at least two dimensions is less than a threshold, adjusting the key point position of the target object in the first image frame using a first filter; or If the change in the posture data of the target object in at least two dimensions is greater than or equal to the threshold, the key point position of the target object in the first image frame is adjusted by the second filter; the first filter and the second filter have different adjustment amplitudes for the key point position of the target object; the first filter is used to predict the key point position of the target object whose posture data of adjacent image frames does not change much, and the second filter is used to predict the key point position of the target object whose posture data of adjacent image frames changes greatly.

11. An electronic device, characterized in that: include: A memory, a processor, and a computer program; the computer program is stored in the memory and configured to be executed by the processor to implement the method according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 10.

13. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 10 when being executed by a processor.

Images