Face tracking method, device, equipment and storage medium

By extracting and adjusting the feature information and posture information of facial images and building a facial matrix, the problem of insufficient facial tracking accuracy and real-time performance in the prior art is solved, and a more accurate and fast facial tracking effect is achieved.

CN112488058BActive Publication Date: 2024-11-26SOPHGO TECH LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011503489.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-12-17
Publication Date
2024-11-26
Estimated Expiration
2040-12-17

AI Technical Summary

Technical Problem

Existing facial tracking technologies are difficult to accurately track when facial movements are performed, resulting in insufficient tracking accuracy and real-time performance.

Method used

By obtaining the facial images in two adjacent frames of images, extracting image feature information and facial posture information, adjusting image feature information to correct offsets, building a facial matrix to determine the matching relationship, and realizing facial tracking.

Benefits of technology

Improve the accuracy and speed of facial tracking, and simple and fast facial matching and tracking are improved, improving real-time.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112488058B_ABST
    Figure CN112488058B_ABST
Patent Text Reader

Abstract

The present application provides a facial tracking method, device, equipment and storage medium, wherein the method includes: determining the facial image of each frame image in two adjacent frames; determining the image feature information of each facial image in the latter frame image in the two adjacent frames according to the facial image of each frame image in the two adjacent frames, determining the facial posture information of each facial image in the latter frame image in the two adjacent frames, wherein the facial posture information characterizes the degree of displacement of the face in the facial image; adjusting the image feature information of each facial image in the latter frame image according to the facial posture information of each facial image in the latter frame image to obtain a facial matrix; determining the matching relationship between each facial image in the previous frame image and each facial image in the latter frame image according to the facial matrix. Accurately tracking the facial image based on the adjusted image feature information improves the accuracy of facial tracking.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of video processing, and in particular to a face tracking method, device, equipment and storage medium. Background Art

[0002] With the development of image and video technology, face detection has been applied to various fields. Face detection technology can be used to obtain facial images and track faces in real time.

[0003] However, in the existing method, the method of extracting facial features to perform face tracking is affected by facial movement and cannot track the face more accurately. Summary of the invention

[0004] The present application provides a face tracking method, apparatus, device and storage medium to solve the problem of how to accurately perform face tracking.

[0005] A first aspect of the present application provides a face tracking method, comprising:

[0006] Acquire two adjacent frames of images in the video stream to be analyzed, and determine the facial image of each of the two adjacent frames of images;

[0007] Determining image feature information of each facial image in a later frame of the two adjacent frames according to the facial image in each frame of the two adjacent frames, and determining facial posture information of each facial image in the later frame of the two adjacent frames, wherein the facial posture information represents the degree of deviation of the face in the facial image;

[0008] According to the facial posture information of each facial image of the next frame image, the image feature information of each facial image of the next frame image is adjusted to obtain a facial matrix;

[0009] According to the facial matrix, a matching relationship between each facial image in a previous frame image and each facial image in a subsequent frame image in the two adjacent frames of images is determined.

[0010] A second aspect of the present application provides a facial tracking device, comprising:

[0011] An acquisition unit, used to acquire two adjacent frames of images in the video stream to be analyzed, and determine a facial image of each of the two adjacent frames of images;

[0012] A first determining unit, configured to determine image feature information of each facial image of a latter frame of two adjacent frames of images according to the facial image of each frame of two adjacent frames of images;

[0013] A second determining unit is used to determine facial posture information of each facial image of the latter frame of the two adjacent frames of images, wherein the facial posture information represents the degree of deviation of the face in the facial image;

[0014] an adjustment unit, configured to adjust the image feature information of each facial image of the subsequent frame image according to the facial posture information of each facial image of the subsequent frame image to obtain a facial matrix;

[0015] A matching unit is used to determine, according to the facial matrix, a matching relationship between each facial image in a previous frame image and each facial image in a subsequent frame image in the two adjacent frames of images.

[0016] A third aspect of the present application provides a computer, comprising a facial tracking device provided by any implementation of the second aspect.

[0017] The fourth aspect of the present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor executes the facial tracking method provided by any implementation of the first aspect.

[0018] The fifth aspect of the present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor executes the facial tracking method provided in any implementation of the second aspect.

[0019] A sixth aspect of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the facial tracking method provided by any implementation of the first aspect.

[0020] The seventh aspect of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the facial tracking method provided by any implementation of the first aspect.

[0021] The present application determines the image feature information and facial posture information of each facial image of the latter frame of two adjacent frames, adjusts the image feature information according to the degree of facial deviation represented by the facial posture information, and then corrects the image feature information; then, based on the facial matrix formed by the image feature information of each facial image of the latter frame, obtains the matching relationship between the facial images of the two adjacent frames, and then tracks the face. Since the image feature information is adjusted according to the degree of facial deviation represented by the facial posture information, and then corrects the image feature information; thus, the facial image can be accurately tracked based on the adjusted image feature information, thereby improving the accuracy of facial tracking. Moreover, in this implementation, matching between facial images is performed based on the method of extracting image feature information and facial posture information, and the matching and tracking algorithms are simpler and faster, thereby improving the speed and real-time performance of facial tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] One or more embodiments are exemplarily described by corresponding drawings, which do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements, and the drawings do not constitute a scale limitation, and wherein:

[0023] Figure 1 Application scenario provided for this application Figure 1 ;

[0024] Figure 2 Application scenario provided for this application Figure 2 ;

[0025] Figure 3 A flowchart of a face tracking method provided in an embodiment of the present application;

[0026] Figure 4 A schematic diagram of facial tracking provided by an embodiment of the present application;

[0027] Figure 5 A flowchart of another face tracking method provided in an embodiment of the present application;

[0028] Figure 6 Schematic diagram of facial organ positioning provided for this application Figure 1 ;

[0029] Figure 7 Schematic diagram of facial organ positioning provided for this application Figure 2 ;

[0030] Figure 8 Schematic diagram of facial organ positioning provided for this application Figure 3 ;

[0031] Fig. 9Schematic diagram of facial organ positioning provided for this application Figure 4 ;

[0032] Fig.10 Schematic diagram of facial organ positioning provided for this application Figure 5 ;

[0033] Fig.11 A flowchart of another face tracking method provided in an embodiment of the present application;

[0034] Fig.12 A flowchart of another face tracking method provided in an embodiment of the present application;

[0035] Fig.13 A schematic diagram of the structure of a facial tracking device provided in an embodiment of the present application;

[0036] Fig.14 A schematic diagram of the structure of another facial tracking device provided in an embodiment of the present application;

[0037] Fig.15 A structural schematic diagram of another facial tracking device provided in an embodiment of the present application;

[0038] Fig.16 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to be able to understand the features and technical contents of the embodiments of the present application in more detail, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0040] With the development of image and video technology, face detection has been applied to various fields. Face detection technology can be used to obtain facial images and track faces in real time.

[0041] In one example, the method of extracting facial features for face tracking is affected by facial movement and cannot track the face more accurately.

[0042] In one example, the model can be used to track the face directly. However, the speed of tracking the face through the model is slow and the real-time performance is poor; especially for scenes with high real-time performance, the speed of tracking the face through the model cannot meet the requirements of scenes with high real-time performance.

[0043] Figure 1 Application scenario provided for this application Figure 1 ,like Figure 1 As shown, a person's face can be tracked.

[0044] Figure 2 Application scenario provided for this application Figure 2 ,like Figure 2 As shown, the face of the animal can be tracked.

[0045] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0046] Figure 3 A flow chart of a facial tracking method provided in an embodiment of the present application. Figure 3 As shown, the method includes:

[0047] 101. Obtain two adjacent frames of images in a video stream to be analyzed, and determine a facial image in each of the two adjacent frames of images.

[0048] Exemplarily, the execution subject of this embodiment is a processor, or a controller, or a facial tracking method device, or other devices or equipment that can execute the method of this embodiment. This embodiment is described with the execution subject being a processor.

[0049] The processor is connected to the camera, and the camera can collect video streams, and then the processor obtains the video streams collected by the camera. The processor can analyze the video streams collected by the camera in real time to track the face; or the processor can analyze the video streams after obtaining the video streams, and then track the face.

[0050] The processor analyzes every two adjacent frames of images in the video stream, wherein there may be N frames of images between the two adjacent frames of images, where N is an integer greater than or equal to 0.

[0051] The processor may extract the facial image in each frame of two adjacent frames of images. In one example, the processor may use a facial extraction algorithm to extract the facial image in the image.

[0052] For example, for two adjacent frames of images, namely image 1 and image 2; each facial image in image 1 can be extracted, namely facial image a1, facial image b1, and facial image c1; each facial image in image 2 can be extracted, namely facial image a2, facial image b2, and facial image c2.

[0053] 102. Determine image feature information of each facial image in the latter frame of the two adjacent frames based on the facial image in each frame of the two adjacent frames, and determine facial posture information of each facial image in the latter frame of the two adjacent frames, wherein the facial posture information represents the degree of deviation of the face in the facial image.

[0054] Exemplarily, the image feature of the facial image of each frame of two adjacent frames is used to determine the image feature information of each facial image of the latter frame of the two adjacent frames. The image feature information may be spatial distance information and / or feature distance information.

[0055] Since the tracking object is moving, the position of the facial image in two adjacent frames can move and change. After extracting the position information of each facial image in the previous frame and the position information of each facial image in the next frame, the processor calculates the position movement relationship between the two facial images, that is, calculates the position movement relationship between each facial image in the previous frame and each facial image in the next frame; thereby, the spatial distance information of each facial image in the next frame is obtained.

[0056] The processor extracts the image information of each facial image on the previous frame image and the image information of each facial image on the next frame image, and then calculates the similarity between the facial images, that is, calculates the similarity between each facial image in the previous frame image and each facial image in the next frame image; thereby, the feature distance information of each facial image in the next frame image is obtained.

[0057] Furthermore, since the face of each facial image has an angle, that is, the degree of facial deviation, the processor can extract the facial posture information of each facial image of the latter frame of two adjacent frames; the facial posture information is the degree of facial deviation. The facial posture information can represent the degree of facial deviation.

[0058] For example, Figure 4 A schematic diagram of face tracking provided in an embodiment of the present application. Figure 4As shown, for two adjacent frames of images, namely image 1 and image 2, each facial image in image 1 can be extracted, namely facial image a1, facial image b1, and facial image c1; each facial image in image 2 can be extracted, namely facial image a2, facial image b2, and facial image c2. The spatial distance information between facial image a2 and facial image a1, facial image b1, and facial image c1 can be calculated, the spatial distance information between facial image b2 and facial image a1, facial image b1, and facial image c1 can be calculated, and the spatial distance information between facial image c2 and facial image a1, facial image b1, and facial image c1 can be calculated; and the feature distance information between facial image a2 and facial image a1, facial image b1, and facial image c1 can be calculated, the feature distance information between facial image b2 and facial image a1, facial image b1, and facial image c1 can be calculated, and the feature distance information between facial image c2 and facial image a1, facial image b1, and facial image c1 can be calculated. The degree of deviation of the facial image a2 can also be calculated to obtain the facial posture information of the facial image a2. Similarly, the facial posture information of the facial image b2 and the facial posture information of the facial image c2 can be calculated.

[0059] The facial image may be a human face image or a facial image of other animals.

[0060] 103. According to the facial posture information of each facial image of the next frame image, the image feature information of each facial image of the next frame image is adjusted to obtain a facial matrix.

[0061] Exemplarily, the facial images of two adjacent frames can be matched directly based on the image feature information of each facial image in the next frame, thereby completing facial tracking; however, in this way, since the tracking object is moving, the position of the facial image in the two adjacent frames can move and change, and the offset of the facial image also affects the facial tracking, and the offset degree of the face needs to be taken into account.

[0062] Then in this step, for each facial image of the next frame image, the image feature information of the facial image is adjusted according to the facial posture information of the facial image, and the facial matrix composed of each facial image of the next frame image is obtained. In one example, the facial posture information and the image feature information can be fused to obtain the facial matrix composed of each facial image of the next frame image. In another example, if the facial posture information of the facial image indicates that the facial offset is more serious, the corresponding image feature information can be removed; the adjusted image feature information of each facial image of the next frame image is used as the facial matrix. In another example, if the facial posture information of the facial image indicates that the facial offset is more serious, the weight corresponding to the spatial distance information of the facial image is adjusted, and the weight corresponding to the feature distance information is adjusted, so as to obtain the spatial distance information with adjusted weights and the feature distance information with adjusted weights; the adjusted image feature information of each facial image of the next frame image is used as the facial matrix.

[0063] 104. Determine, according to the facial matrix, a matching relationship between each facial image in a previous frame image and each facial image in a subsequent frame image in two adjacent frame images.

[0064] Exemplarily, the processor analyzes the facial matrix obtained in step 103, for example, using a model, or using template matching, or using a matching algorithm, to obtain a matching relationship between each facial image in the previous frame image and each facial image in the next frame image. Thus, each facial image in the next frame image corresponds to which facial image in the previous frame image. For example, the Hungarian matching algorithm can be used to calculate the matching relationship between each facial image in the previous frame image and each facial image in the next frame image.

[0065] Since the matching relationship between the facial images in two adjacent frames is obtained, the facial image is tracked. Steps 101-104 can be continuously performed to track the facial image in the video stream.

[0066] For example, for two adjacent frames of images, namely image 1 and image 2, each facial image in image 1 can be extracted, namely facial image a1, facial image b1, and facial image c1; each facial image in image 2 can be extracted, namely facial image a2, facial image b2, and facial image c2. Using the method of this embodiment, it can be determined that facial image a1 matches facial image b2, facial image b1 matches facial image c2, and facial image c1 has no matching image. Thus, the tracking of each facial image is completed.

[0067] In this embodiment, two adjacent frames of images in the video stream to be analyzed are obtained, and the facial image of each frame of the two adjacent frames is determined; according to the facial image of each frame of the two adjacent frames, the image feature information of each facial image of the latter frame of the two adjacent frames is determined, and the facial posture information of each facial image of the latter frame of the two adjacent frames is determined, wherein the facial posture information represents the degree of displacement of the face in the facial image; according to the facial posture information of each facial image of the latter frame, the image feature information of each facial image of the latter frame is adjusted to obtain a facial matrix; according to the facial matrix, the matching relationship between each facial image in the previous frame and each facial image in the latter frame is determined. By determining the image feature information and facial posture information of each facial image of the latter frame of the two adjacent frames, the image feature information is adjusted according to the degree of displacement of the face represented by the facial posture information, and then the image feature information is corrected; then, according to the facial matrix composed of the image feature information of each facial image of the latter frame, the matching relationship between the facial images of the two adjacent frames is obtained, and then the face is tracked. Since the image feature information is adjusted according to the degree of facial deviation represented by the facial posture information, and then the image feature information is corrected, the facial image can be accurately tracked according to the adjusted image feature information, thereby improving the accuracy of facial tracking. In addition, in this embodiment, facial images are matched according to the method of extracting image feature information and facial posture information, and the matching and tracking algorithms are simpler and faster, thereby improving the speed and real-time performance of facial tracking.

[0068] Figure 5 FIG. 1 is a flow chart of another face tracking method provided in an embodiment of the present application. Figure 5 As shown, the method includes:

[0069] 201. Obtain two adjacent frames of images in a video stream to be analyzed, and determine a facial image in each of the two adjacent frames of images.

[0070] Exemplarily, the execution subject of this embodiment is a processor, or a controller, or a facial tracking method device, or other devices or equipment that can execute the method of this embodiment. This embodiment is described with the execution subject being a processor.

[0071] This step can be seen in Figure 3 The step 101 shown is not repeated any more.

[0072] 202. Determine image feature information of each facial image of the latter frame of the two adjacent frames of images according to the facial image of each frame of the two adjacent frames of images.

[0073] In one example, the image feature information includes: spatial distance information and feature distance information; wherein the spatial distance information represents the position movement relationship between the facial image in two adjacent frames of images, and the feature distance information represents the similarity between the facial image in two adjacent frames of images.

[0074] For example, this step can be referred to Figure 3 Step 102 is shown and will not be described in detail.

[0075] 203. Extract facial organ position information of each facial image of the next frame of image, wherein the facial organ position information represents position information of at least one facial organ in the facial image on the facial image.

[0076] In one example, the facial organ position information includes one or more of the following: nose position, left eye position, right eye position, left corner of mouth position, and right corner of mouth position.

[0077] Exemplarily, when extracting facial posture information of a facial image, it is first necessary to extract the position information of each facial organ in the facial image. For example, the position of the nose on the facial image, the position of the left eye on the facial image, the position of the right eye on the facial image, the position of the left corner of the mouth on the facial image, the position of the right corner of the mouth on the facial image, the position of the left ear on the facial image, and the position of the right ear on the facial image are extracted. Any one or more of the above positions constitute the facial organ position information of the facial image.

[0078] For two adjacent frames of images, it is only necessary to extract the facial organ position information of each facial image of the latter frame of image.

[0079] 204. Determine facial posture information of each facial image in the next frame of image according to the facial organ position information of each facial image in the next frame of image, wherein the facial posture information represents the degree of displacement of the face in the facial image.

[0080] In one example, step 204 specifically includes:

[0081] The first step is to determine the facial positioning information of each facial image in the next frame of image according to the facial organ position information of each facial image in the next frame of image, wherein the facial positioning information represents the positioning relationship between the facial organs of the facial images.

[0082] The second step is to determine the facial posture information of each facial image in the next frame of image according to the facial positioning information of each facial image in the next frame of image.

[0083] Exemplarily, for each facial image, the position information of each facial organ in the facial image has been obtained. Based on the position information of each facial organ, the relative position relationship between the facial organs can be clearly determined, and then the degree of facial deviation can be obtained, that is, the facial posture information of the facial image can be obtained.

[0084] In one example, in this step, for each facial image of the next frame image, the positioning relationship between the facial organs (i.e., the relative position relationship between the facial organs, or the relative position relationship between the facial organs and the facial contour) can be determined based on the position of each facial organ in the facial organ position information, and then a facial positioning information pose_score can be obtained.

[0085] Then, the facial positioning information pose_score is converted into a numerical value to obtain the facial posture information alpha. In one example, there is a positive correlation between the numerical value represented by the facial positioning information and the numerical value represented by the facial posture information, that is, the facial positioning information pose_score represents the approximate deviation of the face, the smaller the value of the facial posture information alpha is, and the facial positioning information pose_score represents the approximate straightness of the face, the larger the value of the facial posture information alpha is. Alternatively, in another example, the facial positioning information of each facial image of the next frame image is linearly correlated to obtain the facial posture information of each facial image of the next frame image; wherein, the linear correlation processing can use a linear algorithm, so that the facial positioning information pose_score represents the approximate deviation of the face, the smaller the value of the facial posture information alpha is.

[0086] In the embodiment, this step includes the following implementation methods.

[0087] The first implementation method: according to the facial organ position information of each facial image of the next frame image, determine the first positioning relationship between the nose position and the left face, the second positioning relationship between the nose position and the right face; and determine the third positioning relationship between the nose position and the upper half of the face, and the fourth positioning relationship between the nose position and the lower half of the face; determine the maximum value or minimum value between the first ratio value between the first positioning relationship and the second positioning relationship, and the second ratio value between the third positioning relationship and the fourth positioning relationship, as the value represented by the facial positioning information. According to the facial positioning information of each facial image of the next frame image, determine the facial posture information of each facial image of the next frame image.

[0088] In one example, the first positioning relationship is a first mean, which is the average of the distance between the nose and the left eye in the horizontal direction, and the distance between the nose and the left corner of the mouth in the horizontal direction. The second positioning relationship is a second mean, which is the average of the distance between the nose and the right eye in the horizontal direction, and the distance between the nose and the right corner of the mouth in the horizontal direction. The third positioning relationship is a third mean, which is the average of the distance between the nose and the left eye in the vertical direction, and the distance between the nose and the right eye in the vertical direction. The fourth positioning relationship is a fourth mean, which is the average of the distance between the nose and the left corner of the mouth in the vertical direction, and the distance between the nose and the right corner of the mouth in the vertical direction.

[0089] In the first implementation method, for each facial image of the next frame image, the obtained facial organ position information includes nose position, left eye position, right eye position, left mouth corner position, right mouth corner position, left facial position information, right facial position information, lower half facial position information, and upper half facial position information. A first positioning relationship between the nose position and the left face can be determined based on the nose position and the left face position information, or a first positioning relationship between the nose position and the left face can be determined based on the positional relationships between the nose position and the left eye position and the left corner of the mouth; a second positioning relationship between the nose position and the right face can be determined based on the nose position and the right face position information, or a second positioning relationship between the nose position and the right face can be determined based on the positional relationships between the nose position and the right face position and the right corner of the mouth; a third positioning relationship between the nose position and the upper face can be determined based on the nose position and the upper face position information, or a third positioning relationship between the nose position and the upper face can be determined based on the positional relationships between the nose position and the left eye position and the right eye position; a fourth positioning relationship between the nose position and the lower face can be determined based on the nose position and the upper face position information, or a fourth positioning relationship between the nose position and the lower face can be determined based on the positional relationships between the nose position and the left corner of the mouth and the right corner of the mouth.

[0090] Then, the ratio between the numerical value represented by the first positioning relationship and the numerical value represented by the second positioning relationship is calculated as the first ratio value (which can be the numerical value represented by the first positioning relationship divided by the numerical value represented by the second positioning relationship, or the numerical value represented by the second positioning relationship divided by the numerical value represented by the first positioning relationship), and the ratio between the numerical value represented by the third positioning relationship and the numerical value represented by the fourth positioning relationship is calculated as the second ratio value (which can be the numerical value represented by the third positioning relationship divided by the numerical value represented by the fourth positioning relationship, or the numerical value represented by the fourth positioning relationship divided by the numerical value represented by the third positioning relationship); the maximum value or the minimum value between the first ratio value and the second ratio value is used as the numerical value represented by the facial positioning information.

[0091] In one example, Figure 6 Schematic diagram of facial organ positioning provided for this application Figure 1 ,like Figure 6 As shown, the horizontal distance nose-to-left eye-1 between the nose position and the left eye position is calculated, the horizontal distance nose-to-left mouth-1 between the nose position and the left corner of the mouth is calculated, and the average of the above distance nose-to-left eye-1 and nose-to-left mouth-1 is calculated to obtain a first mean d_l, and the first mean d_l is used as the numerical value represented by the first positioning relationship.

[0092] Calculate the horizontal distance between the nose and the right eye, nose-to-right eye-1, calculate the horizontal distance between the nose and the right corner of the mouth, nose-to-right mouth-1, and calculate the average of the above distances, nose-to-right eye-1 and nose-to-right mouth-1 to obtain a second mean d_r, and use the second mean d_r as the value represented by the second positioning relationship.

[0093] The vertical distance between the nose position and the left eye position, nose-to-left eye-2, and the vertical distance between the nose position and the right eye position, nose-to-right eye-2, are calculated to obtain a third mean d_t, and the third mean d_t is used as the value represented by the third positioning relationship.

[0094] The vertical distance between the nose position and the left corner of the mouth, nose-to-left mouth-2, and the vertical distance between the nose position and the right corner of the mouth, nose-to-right mouth-2, are calculated to obtain a fourth mean d_b, and the fourth mean d_b is used as the value represented by the fourth positioning relationship.

[0095] Then, the value d_l / d_r obtained by dividing the first mean d_l by the second mean d_r, and the value d_r / d_l obtained by dividing the second mean d_r by the first mean d_l, the minimum of which is the first proportion value score_h=min(d_l / d_r, d_r / d_l). The value d_t / d_b obtained by dividing the third mean d_t by the fourth mean d_b, and the value d_b / d_t*factor obtained when the fourth mean d_b is at the preset value d_t*factor, the minimum of which is the second proportion value score_v=min(d_t / d_b, d_b / d_t*factor). Where factor is a preset proportion value, and factor is an empirical value. Since the eyes and mouth are not symmetrical in the vertical direction, when the face is facing forward, the average distance between the nose and the two eyes and the average distance between the nose and the two corners of the mouth are different; therefore, when calculating the second ratio value score_v, it is necessary to assign it a preset ratio value factor for adjustment.

[0096] Furthermore, the minimum value of the first ratio value score_h and the second ratio value score_v is used as the numerical value pose_score=min(score_h, score_v) represented by the face positioning information.

[0097] Finally, there is a positive correlation between the value represented by the facial positioning information and the value represented by the facial posture information. The facial posture information alpha can be obtained according to the facial positioning information pose_score, where: Figure 7 Schematic diagram of facial organ positioning provided for this application Figure 2 ,like Figure 7 As shown, the facial positioning information pose_score represents that the face is slightly tilted, then the smaller the value of the facial posture information alpha is; the facial positioning information pose_score represents that the face is slightly straight, then the larger the value of the facial posture information alpha is.

[0098] In another example, Figure 8 Schematic diagram of facial organ positioning provided for this application Figure 3 ,like Figure 8As shown, the position of the center point of the left face is calculated, and the distance between the position of the nose and the position of the center point of the left face is used as the numerical value A1 represented by the first positioning relationship; the position of the center point of the right face is calculated, and the distance between the position of the nose and the position of the center point of the right face is used as the numerical value A2 represented by the second positioning relationship; the position of the center point of the upper half of the face is calculated, and the distance between the position of the nose and the position of the center point of the upper half of the face is used as the numerical value A3 represented by the third positioning relationship; the position of the center point of the lower half of the face is calculated, and the distance between the position of the nose and the position of the center point of the lower half of the face is used as the numerical value A4 represented by the fourth positioning relationship.

[0099] Then, divide A1 by A2, divide A2 by A1, and take the minimum value as the first ratio score_h=min(A1 / A2, A2 / A1). Divide A3 by A4, divide A4 by A3*factor, and take the minimum value as the second ratio score_v=min(A3 / A4, A4 / A3*factor).

[0100] Furthermore, the minimum value of the first ratio score_h and the second ratio score_v is used as the numerical value represented by the facial positioning information pose_score=min(score_h, score_v). Finally, there is a positive correlation between the numerical value represented by the facial positioning information and the numerical value represented by the facial posture information, and the facial posture information alpha can be obtained according to the facial positioning information pose_score.

[0101] The facial posture information alpha in the above method may have a value range of 0 to 1. The closer the pose_score is to 1, the more positive the facial posture is; the closer the pose_score is to 0, the more biased the facial posture is.

[0102] The second implementation method is to determine the image center point of each facial image of the next frame image; determine the distance between the nose position of each facial image of the next frame image and the image center point, which is a value represented by the facial positioning information of each facial image of the next frame image. According to the facial positioning information of each facial image of the next frame image, determine the facial posture information of each facial image of the next frame image.

[0103] In the second implementation, Fig. 9 Schematic diagram of facial organ positioning provided for this application Figure 4 ,like Fig. 9As shown, for each facial image of the next frame image, the image center point is calculated based on each pixel point of the facial image; then the distance between the nose position and the image center point is calculated; the distance between the nose position and the image center point is used as the value represented by the facial positioning information pose_score; there is a positive correlation between the value represented by the facial positioning information and the value represented by the facial posture information, and the facial posture information alpha can be obtained according to the facial positioning information pose_score.

[0104] The third implementation method: according to the number of facial organs represented by the facial organ position information of each facial image of the next frame image, determine the value represented by the facial positioning information of each facial image of the next frame image. According to the facial positioning information of each facial image of the next frame image, determine the facial posture information of each facial image of the next frame image.

[0105] In the third implementation, since the positions of the facial organs in each facial image have been obtained, the number of facial organs in the facial image can be determined; according to the number of facial organs, the value represented by the facial positioning information is determined, wherein the number of facial organs and the value represented by the facial positioning information are positively correlated. There is a positive correlation between the value represented by the facial positioning information and the value represented by the facial posture information, and the facial posture information alpha can be obtained according to the facial positioning information pose_score.

[0106] Fourth implementation method: for each facial image of the next frame image, determine a first line between the left eye position and the right eye position, and determine a second line between the left mouth corner position and the right mouth corner position; for each facial image of the next frame image, determine the value represented by the facial positioning information according to the relative position relationship between the first line and the second line. According to the facial positioning information of each facial image of the next frame image, determine the facial posture information of each facial image of the next frame image.

[0107] In the fourth implementation, Fig.10 Schematic diagram of facial organ positioning provided for this application Figure 5 ,like Fig.10 As shown, for each facial image of the next frame image, the position of the facial organs in each facial image has been obtained, and the left eye position and the right eye position are connected to obtain a first line; the left corner of the mouth position and the right corner of the mouth position are connected to obtain a second line.

[0108] Then, according to the relative position relationship between the first line and the second line, the value represented by the facial positioning information is determined; for example, the angle between the first line and the second line is determined, and the value represented by the facial positioning information is determined according to the angle, and there is a negative correlation between the angle and the value represented by the facial positioning information, that is, the larger the angle between the first line and the second line, the more biased the face is. Then, there is a positive correlation between the value represented by the facial positioning information and the value represented by the facial posture information, and the facial posture information alpha can be obtained according to the facial positioning information pose_score.

[0109] 205. Determine, according to the facial posture information of each facial image of the next frame image, a first weight corresponding to the feature distance information and a second weight corresponding to the spatial distance information of each facial image of the next frame image.

[0110] In one example, there is a positive correlation between the first weight and the numerical value represented by the facial posture information, and there is a negative correlation between the second weight and the numerical value represented by the facial posture information.

[0111] Exemplarily, after the facial posture information alpha of each facial image of the next frame image is obtained, the first weight corresponding to the feature distance information and the second weight corresponding to the spatial distance information are adjusted according to the facial posture information alpha.

[0112] When the facial posture information represents the biased direction of the face in the facial image (i.e., the face is approximately biased), the first weight corresponding to the feature distance information is reduced, and the second weight corresponding to the spatial distance information is increased; when the facial posture information represents the positive direction of the face in the facial image (i.e., the face is approximately positive), the first weight corresponding to the feature distance information is increased, and the second weight corresponding to the spatial distance information is reduced. That is, the first weight is positively correlated with the value represented by the facial posture information, and the second weight is negatively correlated with the value represented by the facial posture information.

[0113] 206. For each facial image of the next frame of image, obtain a feature value of each facial image of the next frame of image according to the first weight, the feature distance information, the second weight, and the spatial distance information.

[0114] Exemplarily, for each facial image of the next frame of image, a feature value of the facial image is obtained by weighted calculation of the first weight, the feature distance information, the second weight, and the spatial distance information.

[0115] In one example, the feature value of the facial image is cost_matrix=(1-alpha)*spatial distance information+alpha*feature distance information.

[0116] It can be seen that the facial posture information represents that the face is slightly biased (ie, not straight), so it is necessary to reduce the proportion of the feature distance information (ie, the similarity of the facial image in two adjacent frames).

[0117] 207. Obtain a facial matrix according to the eigenvalues ​​of each facial image of the next frame of image.

[0118] Exemplarily, a feature value is obtained for each facial image of the next frame of image, and the feature values ​​of each facial image can be used to form a facial matrix.

[0119] For example, there are two adjacent frames, the first frame has 5 facial images, and the second frame has 6 facial images. For each facial image in the second frame, 5 eigenvalues ​​can be obtained, and a total of 5*6 eigenvalues ​​can be obtained; finally, a 5*6 facial matrix is ​​obtained.

[0120] 208. Determine, according to the facial matrix, a matching relationship between each facial image in a previous frame image and each facial image in a subsequent frame image in two adjacent frames of images.

[0121] This step can be seen in Figure 3 Step 104 is shown and will not be described in detail.

[0122] In this embodiment, on the basis of the above embodiment, facial posture information is obtained according to the position of facial organs, wherein the facial posture information represents the degree of deviation of the face in the facial image; when the facial posture information represents the approximate deviation of the face, it is determined that the feature distance information (i.e., the similarity of the facial images in two adjacent frames) is less important, and it is necessary to reduce the proportion of the feature distance information (i.e., the similarity of the facial images in two adjacent frames) and increase the proportion of the spatial distance information (i.e., the position movement relationship of the facial images in two adjacent frames), and then correct and adjust the image feature information to obtain a facial matrix, so as to facilitate more accurate determination of the matching image and facilitate facial tracking; thereby improving the accuracy of facial tracking.

[0123] Fig.11 A flowchart of another face tracking method provided in an embodiment of the present application is shown below. Fig.11 As shown, the method includes:

[0124] 301. Obtain two adjacent frames of images in a video stream to be analyzed, and determine a facial image of each of the two adjacent frames of images.

[0125] 302. Determine image feature information of each facial image in the latter frame of the two adjacent frames based on the facial image in each frame of the two adjacent frames, and determine facial posture information of each facial image in the latter frame of the two adjacent frames, wherein the facial posture information represents the degree of deviation of the face in the facial image.

[0126] 303. According to the facial posture information of each facial image of the next frame image, adjust the image feature information of each facial image of the next frame image to obtain a facial matrix.

[0127] 304. Determine, according to the facial matrix, a matching relationship between each facial image in a previous frame image and each facial image in a subsequent frame image in two adjacent frame images.

[0128] Exemplarily, steps 301-304 can refer to the introduction of the above embodiment and will not be described in detail.

[0129] 305. Extract the global semantic features and local detail features of the image to be retrieved.

[0130] Exemplarily, the execution subject of this embodiment is a processor, or a controller, or a facial tracking method device, or other devices or equipment that can execute the method of this embodiment. This embodiment is described with the execution subject being a processor.

[0131] Image retrieval can also be performed on the image to be retrieved. For example, when searching for a user, the image to be retrieved in the video stream can be analyzed in real time to determine whether the image to be retrieved in the video stream matches the image in the preset image library.

[0132] The image to be retrieved may be each frame of an image in a video stream, or the image to be retrieved may be a facial image of each frame of an image in a video stream, or the image to be retrieved may be other images to be analyzed.

[0133] First, it is necessary to extract the global semantic features and local detail features of the image to be retrieved.

[0134] In one example, a deep learning model can be used to extract features from the image to be retrieved to obtain deep learning features, wherein the deep learning features represent the global semantic features of the image. For example, a convolutional neural network (CNN) model can be used to extract features from the image to be retrieved to obtain CNN features, wherein the CNN features represent the global semantic features of the image.

[0135] In one example, a traditional model can be used to extract features from the image to be retrieved to obtain local detail features of the image. For example, a traditional model is used to extract features from the image to be retrieved to obtain scale invariant feature transform (SIFT) features; wherein SIFT features are a type of local detail features of the image. For another example, a traditional model is used to extract features from the image to be retrieved to obtain SIFT features; after aggregating the SIFT features, vector of locally aggregated descriptors (VLAD) features are obtained; wherein VLAD features are a type of local detail features of the image.

[0136] 306. Determine an image set in the preset image library that matches the image to be retrieved based on the image global semantic features of the image to be retrieved and the image global semantic features of each image in the preset image library, wherein the image set includes at least one image.

[0137] For example, if the image is retrieved and analyzed based on the global semantic features of the image, the simple global semantic features of the image cannot provide good resolution, which affects the retrieval results and makes the retrieval effect unsatisfactory. If the image is retrieved and analyzed based on the global semantic features of the image, the image matching process based on the global semantic features of the image is relatively time-consuming, and the image retrieval efficiency is low.

[0138] In this embodiment, the image global semantic features and the image local detail features may be combined to perform image analysis and complete image retrieval.

[0139] First, for each image in the preset image library, the image global semantic features and image local detail features of each image have been extracted. Feature matching can be performed based on the image global semantic features of the image to be retrieved and the image global semantic features of each image in the preset image library, and then multiple images matching the image to be retrieved are obtained; the obtained multiple images constitute an image combination.

[0140] 307. Determine the best image in the image set that matches the image to be retrieved based on the image local detail features of the image to be retrieved and the image local detail features of each image in the image set.

[0141] Exemplarily, after step 307, preliminary image matching has been performed on the images based on the local detail features of the images to obtain an image set, and then an image that best matches the image to be retrieved is obtained from the image set based on the local detail features of the images.

[0142] Therefore, feature matching can be performed based on the local detail features of the image to be retrieved and the local detail features of each image in the image set, thereby obtaining an optimal image that matches the image to be retrieved.

[0143] In this embodiment, based on the above-mentioned embodiments, image analysis is performed in combination with the global semantic features of the image and the local detail features of the image, and the image to be retrieved is matched and analyzed with the images in the preset image library to obtain a most matching image. Since the image analysis and retrieval are performed in combination with the global semantic features of the image and the local detail features of the image, the accuracy of image retrieval can be improved. In addition, a large number of images in the image library are preliminarily screened based on the global semantic features of the image to obtain a small image set. Since image retrieval based on the global semantic features of the image is faster, the retrieval speed is faster; then the images in the image set are further screened based on the local detail features of the image. Since the number of images in the image set is small, the screening based on the local detail features of the image is faster; thus, the image retrieval speed provided by this embodiment is faster.

[0144] Fig.12 A flowchart of another facial tracking method provided in an embodiment of the present application is shown below. Fig.12 As shown, the method includes:

[0145] 401. Obtain two adjacent frames of images in a video stream to be analyzed, and determine a facial image in each of the two adjacent frames of images.

[0146] 402. Determine image feature information of each facial image in the latter frame of the two adjacent frames based on the facial image in each frame of the two adjacent frames, and determine facial posture information of each facial image in the latter frame of the two adjacent frames, wherein the facial posture information represents the degree of deviation of the face in the facial image.

[0147] 403. According to the facial posture information of each facial image of the next frame image, adjust the image feature information of each facial image of the next frame image to obtain a facial matrix.

[0148] 404. Determine, according to the facial matrix, a matching relationship between each facial image in a previous frame image and each facial image in a subsequent frame image in two adjacent frame images.

[0149] Exemplarily, steps 301-304 can refer to the introduction of the above embodiment and will not be described in detail.

[0150] 405. Based on the deep learning model, extract the first feature map of the image to be retrieved; perform pooling processing on the first feature map to obtain a feature vector, and perform regularization processing on the feature vector to obtain a global semantic feature of the image.

[0151] Exemplarily, the execution subject of this embodiment is a processor, or a controller, or a facial tracking method device, or other devices or equipment that can execute the method of this embodiment. This embodiment is described with the execution subject being a processor.

[0152] A deep learning model has been trained, wherein the deep learning model is, for example, a VGG (Visual Geometry Group Network) model or a Residual Network (ResNet) model. Then, based on the deep learning model, a first feature map of the image to be retrieved is extracted; for example, a feature map output by the last convolutional layer of the deep learning model is used as the first feature map.

[0153] The image to be retrieved may be each frame of an image in a video stream, or the image to be retrieved may be a facial image of each frame of an image in a video stream, or the image to be retrieved may be other images to be analyzed.

[0154] Then, the first feature map is pooled to obtain a feature vector, where the dimension of the feature vector is d. For example, the first feature map is subjected to a max pooling process; wherein the max pooling process refers to selecting the maximum value of the image region as the value after pooling of the region. Alternatively, the first feature map is subjected to an average pooling process; wherein the average pooling process refers to calculating the average value of the image region as the value after pooling of the region.

[0155] Finally, the feature vector is regularized to obtain the global semantic features of the image. For example, the feature vector is L2 regularized.

[0156] 406. Extract SIFT features of the image to be retrieved; aggregate the SIFT features to obtain a local feature matrix, and perform regularization and dimensionality reduction on the local feature matrix to obtain local detail features of the image.

[0157] Exemplarily, SIFT features of the image to be retrieved can be extracted based on a preset model, where the SIFT features are N m-dimensional vectors. For example, SIFT features are N 128-dimensional vectors. Since the amount of SIFT features is large, in order to reduce the subsequent calculation process, SIFT features can be processed by local aggregation features (Vector Of Locally Aggregated Descriptors, abbreviated as VLAD) to obtain a local feature matrix, where the dimension of the local feature matrix is ​​(K, m); K is the number of cluster centers specified when VLAD processing is performed on N m-dimensional vectors.

[0158] Next, the (K, m)-dimensional local feature matrix is ​​subjected to power norm processing by column, and then concatenated by column into a K*m-dimensional local feature vector; wherein the exponent of the power norm processing can be taken as 0.5.

[0159] The above K*m-dimensional local feature vector is subjected to regularization processing, such as L2 regularization processing. Then, the regularized local feature vector is subjected to dimensionality reduction processing to obtain a VLAD feature of p dimensions; the VLAD feature is a local detail feature of an image. For example, principal component analysis (PCA) dimensionality reduction processing can be used to obtain a 512-dimensional VLAD feature.

[0160] 407. Determine a first similarity feature matrix between the image global semantic features of the image to be retrieved and the image global semantic features of each image in the preset image library; determine an image set according to the numerical values ​​represented by the first similarity feature matrix of each image in the preset image library.

[0161] Exemplarily, the global semantic features and local detail features of each image in the preset image library may be obtained by adopting the method of steps 405-406.

[0162] First, based on the global semantic features of the image, a preliminary search is performed on the images to be searched. Since the global semantic features of each image in the preset image library are extracted, the global semantic features of each image can be stored as a (Q, d)-dimensional matrix A, where Q is the total number of images in the preset image library and d is the dimension of the global semantic features of the image.

[0163] For an image to be retrieved, the global semantic feature of the image to be retrieved is a matrix B of (1, d) dimension.

[0164] Calculate the similarity between the matrix A of (Q, d) dimension and the matrix B of (1, d) dimension to obtain a first similarity feature matrix S = dot(G, BT), where dot represents matrix multiplication, BT represents the transpose of the matrix B, and the dimension of the first similarity feature matrix S1 is (Q, 1).

[0165] The values ​​in the first similarity feature matrix S1 are sorted from high to low to obtain the largest first M indexes, and then M candidate images in the preset image library are obtained. These M candidate images are the images most similar to the image to be retrieved. For example, M=20.

[0166] 408. Determine a second similarity feature matrix between the image local detail features of the image to be retrieved and the image local detail features of each image in the image set; determine the optimal image according to the numerical values ​​represented by the second similarity feature matrix of each image in the image set.

[0167] Exemplarily, after step 407, preliminary image matching has been performed on the images based on the local detail features of the images to obtain an image set, and then an image that best matches the image to be retrieved is obtained from the image set based on the local detail features of the images.

[0168] Since the local detail features of each image in the image set are extracted, the local detail features of each image in the image set can be stored as a (M, p)-dimensional matrix V, where p is the dimension of the VLAD feature (i.e., the dimension of the local detail features of the image); and M is the total number of images in the image set.

[0169] For an image to be retrieved, the local detail features of the image to be retrieved are a matrix C of (1, p) dimension.

[0170] Calculate the similarity between the matrix V of (M, p) dimension and the matrix C of (1, p) dimension to obtain a second similarity feature matrix S2 = dot(V, CT), where dot represents matrix multiplication, CT represents the transpose of the matrix C, and the dimension of the second similarity feature matrix S2 is (M, 1).

[0171] The values ​​in the second similarity feature matrix S2 are sorted from high to low to obtain the largest index, and then an optimal image that best matches the image to be retrieved is obtained.

[0172] This embodiment, based on the above-mentioned embodiments, combines the global semantic features of the image and the local detail features of the image for image analysis and retrieval, thereby improving the accuracy of image retrieval. In addition, a large number of images in the image library are preliminarily screened based on the global semantic features of the image to obtain a small image set. Since the image retrieval based on the global semantic features of the image is faster, the retrieval speed is faster; then the images in the image set are further screened based on the local detail features of the image. Since the number of images in the image set is small, the screening based on the local detail features of the image is faster; thus, the image retrieval speed provided by this embodiment is faster. In addition, the local detail features of the image obtained are VLAD features. Since the data volume of the VLAD features is small, the image retrieval based on the local detail features of the image is accelerated, further accelerating the entire image retrieval process.

[0173] Fig.13A schematic diagram of the structure of a facial tracking device provided in an embodiment of the present application is shown in FIG. Fig.13 As shown, the face tracking device provided in this embodiment includes:

[0174] The acquisition unit 31 is used to acquire two adjacent frames of images in the video stream to be analyzed, and determine the facial image of each of the two adjacent frames of images.

[0175] The first determining unit 32 is used to determine the image feature information of each facial image in the latter frame of the two adjacent frames of images according to the facial image in each frame of the two adjacent frames of images.

[0176] The second determining unit 33 is used to determine the facial posture information of each facial image of the latter frame image of two adjacent frames of images, wherein the facial posture information represents the degree of deviation of the face in the facial image.

[0177] The adjustment unit 34 is used to adjust the image feature information of each facial image in the next frame image according to the facial posture information of each facial image in the next frame image to obtain a facial matrix.

[0178] The matching unit 35 is used to determine the matching relationship between each facial image in the previous frame image and each facial image in the next frame image in two adjacent frame images according to the facial matrix.

[0179] The face tracking device provided in this embodiment is the same as the technical solution in the face tracking method provided in any of the aforementioned embodiments and implementation modes, and its implementation principle and technical effects are similar, which will not be described in detail.

[0180] Fig.14 A schematic diagram of the structure of another facial tracking device provided in an embodiment of the present application, Fig.13 Based on the embodiment shown, Fig.14 As shown, in the face tracking device provided by this embodiment, the image feature information includes: spatial distance information and feature distance information; wherein the spatial distance information represents the position movement relationship between the facial image in two adjacent frames of images, and the feature distance information represents the similarity between the facial image in two adjacent frames of images.

[0181] In one example, the adjustment unit 34 includes:

[0182] The first determination module 341 is used to determine the first weight corresponding to the feature distance information and the second weight corresponding to the spatial distance information of each facial image in the next frame image according to the facial posture information of each facial image in the next frame image.

[0183] The second determination module 342 is used to obtain the feature value of each facial image in the next frame of image according to the first weight, feature distance information, second weight, and spatial distance information.

[0184] The third determination module 343 is used to obtain a facial matrix according to the feature values ​​of each facial image of the next frame image.

[0185] In one example, there is a positive correlation between the first weight and the numerical value represented by the facial posture information, and there is a negative correlation between the second weight and the numerical value represented by the facial posture information.

[0186] In one example, the second determining unit 33 includes:

[0187] The extraction module 331 is used to extract facial organ position information of each facial image of the next frame of image, wherein the facial organ position information represents the position information of at least one facial organ in the facial image on the facial image.

[0188] The fourth determination module 332 is used to determine the facial posture information of each facial image in the next frame of image according to the facial organ position information of each facial image in the next frame of image.

[0189] In one example, the fourth determining module 332 includes:

[0190] The first determination submodule 3321 is used to determine the facial positioning information of each facial image in the next frame image according to the facial organ position information of each facial image in the next frame image, wherein the facial positioning information represents the positioning relationship between the facial organs of the facial image.

[0191] The second determining submodule 3322 is used to determine the facial posture information of each facial image in the next frame of image according to the facial positioning information of each facial image in the next frame of image.

[0192] In one example, the facial organ position information includes one or more of the following: nose position, left eye position, right eye position, left corner of mouth position, and right corner of mouth position.

[0193] In one example, the first determining submodule 3321 is specifically configured to:

[0194] Based on the facial organ position information of each facial image of the next frame image, determine the first positioning relationship between the nose position and the left face, and the second positioning relationship between the nose position and the right face; determine the third positioning relationship between the nose position and the upper half of the face, and the fourth positioning relationship between the nose position and the lower half of the face; determine the maximum value or minimum value between the first ratio value between the first positioning relationship and the second positioning relationship, and the second ratio value between the third positioning relationship and the fourth positioning relationship, as the numerical value represented by the facial positioning information.

[0195] In one example, the first positioning relationship is a first mean, which is the average of the horizontal distance between the nose and the left eye, and the horizontal distance between the nose and the left corner of the mouth; the second positioning relationship is a second mean, which is the average of the horizontal distance between the nose and the right eye, and the horizontal distance between the nose and the right corner of the mouth; the third positioning relationship is a third mean, which is the average of the vertical distance between the nose and the left eye, and the vertical distance between the nose and the right eye; the fourth positioning relationship is a fourth mean, which is the average of the vertical distance between the nose and the left corner of the mouth, and the vertical distance between the nose and the right corner of the mouth.

[0196] In one example, the first ratio value is the minimum value of the value obtained by dividing the first mean by the second mean and the value obtained by dividing the second mean by the first mean; the second ratio value is the minimum value of the value obtained by dividing the third mean by the fourth mean and the value obtained when the fourth mean is at a preset value, and the preset value is the product of the third mean and the preset ratio value.

[0197] In one example, the first determination submodule is specifically used to:

[0198] Determine the image center point of each facial image of the next frame image; determine the distance between the nose position of each facial image of the next frame image and the image center point, which is a value represented by the facial positioning information of each facial image of the next frame image.

[0199] In one example, the first determining submodule 3321 is specifically configured to:

[0200] According to the number of facial organs represented by the facial organ position information of each facial image of the next frame image, the value represented by the facial positioning information of each facial image of the next frame image is determined.

[0201] In one example, the first determining submodule 3321 is specifically configured to:

[0202] For each facial image in the next frame image, determine a first line between the left eye position and the right eye position, and determine a second line between the left corner of the mouth position and the right corner of the mouth; for each facial image in the next frame image, determine the value represented by the facial positioning information based on the relative position relationship between the first line and the second line.

[0203] In one example, there is a positive correlation between the value represented by the facial positioning information and the value represented by the facial posture information.

[0204] Alternatively, the second determining submodule 3322 is specifically configured to perform linear correlation processing on the facial positioning information of each facial image of the next frame image to obtain the facial posture information of each facial image of the next frame image.

[0205] The face tracking device provided in this embodiment is the same as the technical solution in the face tracking method provided in any of the aforementioned embodiments and implementation modes, and its implementation principle and technical effects are similar, which will not be described in detail.

[0206] Fig.15 A structural diagram of another facial tracking device provided in an embodiment of the present application, Fig.13 or Fig.14 Based on the embodiment shown, Fig.15 As shown, the face tracking device provided in this embodiment further includes:

[0207] The extraction unit 41 is used to extract the global semantic features and local detail features of the image to be retrieved.

[0208] The third determining unit 42 is used to determine an image set in the preset image library that matches the image to be retrieved based on the image global semantic features of the image to be retrieved and the image global semantic features of each image in the preset image library, and the image set includes at least one image.

[0209] The fourth determining unit 43 is used to determine the best image in the image set that matches the image to be retrieved according to the image local detail features of the image to be retrieved and the image local detail features of each image in the image set.

[0210] In one example, the extraction unit 41, when extracting the global semantic features of the image to be retrieved, is specifically used to: extract the first feature map of the image to be retrieved based on a deep learning model; perform pooling processing on the first feature map to obtain a feature vector, and perform regularization processing on the feature vector to obtain the global semantic features of the image.

[0211] In one example, the extraction unit 41, when extracting local detail features of an image to be retrieved, is specifically used to: extract SIFT features of the image to be retrieved; aggregate the SIFT features to obtain a local feature matrix, and perform regularization and dimensionality reduction on the local feature matrix to obtain local detail features of the image.

[0212] In one example, the third determining unit 42 is specifically configured to:

[0213] Determine a first similarity feature matrix between the image global semantic features of the image to be retrieved and the image global semantic features of each image in the preset image library; determine an image set according to the numerical values ​​represented by the first similarity feature matrix of each image in the preset image library.

[0214] In one example, the fourth determining unit 43 is specifically configured to:

[0215] Determine a second similarity feature matrix between the image local detail feature of the image to be retrieved and the image local detail feature of each image in the image set;

[0216] The optimal image is determined according to the value represented by the second similarity feature matrix of each image in the image set.

[0217] The face tracking device provided in this embodiment is the same as the technical solution in the face tracking method provided in any of the aforementioned embodiments and implementation modes, and its implementation principle and technical effects are similar, which will not be described in detail.

[0218] The embodiment of the present application also provides a computer (or a device such as a mobile phone) comprising the above-mentioned facial tracking device.

[0219] An embodiment of the present application also provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the above-mentioned face tracking method.

[0220] An embodiment of the present application also provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the above-mentioned facial tracking method.

[0221] The computer-readable storage medium mentioned above may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0222] The present application also provides an electronic device, the structure of which is as follows: Fig.16 As shown, the electronic device includes:

[0223] at least one processor 120, Fig.16 In the embodiment, a processor 120 is taken as an example; and a memory 121 may also include a communication interface 122 and a bus 123. The processor 120, the communication interface 122, and the memory 121 may communicate with each other through the bus 123. The communication interface 122 may be used for information transmission. The processor 120 may call the logic instructions in the memory 121 to execute the face tracking method of the above embodiment.

[0224] In addition, the logic instructions in the above-mentioned memory 121 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0225] The memory 121 is a computer-readable storage medium that can be used to store software programs and computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the present application. The processor 120 executes functional applications and data processing by running the software programs, instructions and modules stored in the memory 121, that is, implementing the face tracking method in the above method embodiment.

[0226] The memory 121 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 121 may include a high-speed random access memory and may also include a non-volatile memory.

[0227] The technical solution of the embodiment of the present application can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method of the embodiment of the present application. The aforementioned storage medium may be a non-transient storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, and other media that can store program codes, or a transient storage medium.

[0228] When used in this application, although the terms "first", "second", etc. may be used in this application to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, without changing the meaning of the description, a first element can be called a second element, and similarly, a second element can be called a first element, as long as all occurrences of "first element" are renamed consistently and all occurrences of "second element" are renamed consistently. The first element and the second element are both elements, but they may not be the same element.

[0229] The words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include the plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of one or more associated listings. In addition, when used in this application, the term "comprise" and its variants "comprises" and / or including (comprising) refer to the existence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the existence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups of these.

[0230] The various aspects, implementations, implementations or features of the described embodiments can be used alone or in any combination. The various aspects of the described embodiments can be implemented by software, hardware or a combination of software and hardware. The described embodiments can also be embodied by a computer-readable medium storing computer-readable code, which includes instructions that can be executed by at least one computing device. The computer-readable medium can be associated with any data storage device capable of storing data that can be read by a computer system. Computer-readable media for example can include read-only memory, random access memory, CD-ROM, HDD, DVD, magnetic tape, and optical data storage devices. The computer-readable medium can also be distributed in computer systems connected by a network so that the computer-readable code can be stored and executed in a distributed manner.

[0231] The above technical description may refer to the accompanying drawings, which form a part of the present application, and the implementation methods according to the described embodiments are shown in the accompanying drawings by description. Although these embodiments are described in sufficient detail to enable those skilled in the art to implement these embodiments, these embodiments are non-restrictive; other embodiments can be used in this way, and changes can be made without departing from the scope of the described embodiments. For example, the order of operations described in the flowchart is non-restrictive, so the order of two or more operations illustrated in the flowchart and described according to the flowchart can be changed according to several embodiments. As another example, in several embodiments, one or more operations illustrated in the flowchart and described according to the flowchart are optional or deletable. In addition, certain steps or functions can be added to the disclosed embodiments, or the order of two or more steps can be replaced. All these changes are considered to be included in the disclosed embodiments and claims.

[0232] In addition, terms are used in the above technical description to provide a thorough understanding of the described embodiments. However, overly detailed details are not required to implement the described embodiments. Therefore, the above description of the embodiments is presented for the purpose of explanation and description. The embodiments presented in the above description and the examples disclosed according to these embodiments are provided separately to add context and help understand the described embodiments. The above description is not intended to be exhaustive or to limit the described embodiments to the precise form of the present application. According to the above teachings, several modifications, selective applications and changes are feasible. In some cases, well-known processing steps are not described in detail to avoid unnecessarily affecting the described embodiments.

Claims

1. A facial tracking method, It is characterized in that include: Acquire two adjacent frames of images in the video stream to be analyzed, and determine the facial image of each of the two adjacent frames of images; Determining image feature information of each facial image in a later frame of the two adjacent frames according to the facial image in each frame of the two adjacent frames, and determining facial posture information of each facial image in the later frame of the two adjacent frames, wherein the facial posture information represents the degree of deviation of the face in the facial image; According to the facial posture information of each facial image of the next frame image, the image feature information of each facial image of the next frame image is adjusted to obtain a facial matrix; the image feature information includes: spatial distance information and feature distance information; Determining, according to the facial matrix, a matching relationship between each facial image in a previous frame image and each facial image in a subsequent frame image in the two adjacent frames of images; The step of adjusting the image feature information of each facial image of the subsequent frame image according to the facial posture information of each facial image of the subsequent frame image to obtain a facial matrix includes: According to the facial posture information of each facial image of the next frame image, adjusting the first weight corresponding to the feature distance information and the second weight corresponding to the spatial distance information of each facial image of the next frame image; the first weight is positively correlated with the numerical value represented by the facial posture information, and the second weight is negatively correlated with the numerical value represented by the facial posture information; For each facial image of the next frame of image, performing weighted calculation according to the first weight, the feature distance information, the second weight, and the spatial distance information to obtain a feature value of each facial image of the next frame of image; Obtaining the facial matrix according to the eigenvalues ​​of each facial image of the next frame image; The determining of the facial posture information of each facial image of the latter frame of the two adjacent frames of images comprises: Extracting facial organ position information of each facial image of the next frame of image, wherein the facial organ position information represents position information of at least one facial organ in the facial image on the facial image; The facial posture information of each facial image of the subsequent frame image is determined according to the facial organ position information of each facial image of the subsequent frame image.

2. The method according to claim 1, It is characterized in that The spatial distance information represents the positional movement relationship of the facial image between the two adjacent frames of images, and the feature distance information represents the similarity of the facial image between the two adjacent frames of images.

3. The method according to claim 1, It is characterized in that Determining facial posture information of each facial image of the next frame of image according to facial organ position information of each facial image of the next frame of image includes: Determining facial positioning information of each facial image of the next frame of image according to facial organ position information of each facial image of the next frame of image, wherein the facial positioning information represents the positioning relationship between facial organs of the facial images; The facial posture information of each facial image in the next frame of image is determined according to the facial positioning information of each facial image in the next frame of image.

4. The method according to claim 3, It is characterized in that The facial organ position information includes one or more of the following: nose position, left eye position, right eye position, left mouth corner position, and right mouth corner position.

5. The method according to claim 4, It is characterized in that Determining facial positioning information of each facial image of the next frame of image according to facial organ position information of each facial image of the next frame of image comprises: Determine, based on the facial organ position information of each facial image of the next frame of image, a first positioning relationship between the nose position and the left face, a second positioning relationship between the nose position and the right face; and determine a third positioning relationship between the nose position and the upper half of the face, and a fourth positioning relationship between the nose position and the lower half of the face; Determine a maximum value or a minimum value between a first ratio value between the first positioning relationship and the second positioning relationship and a second ratio value between the third positioning relationship and the fourth positioning relationship as the value represented by the facial positioning information.

6. The method according to claim 5, It is characterized in that The first positioning relationship is a first mean value, which is the mean of the distance between the nose position and the left eye position in the horizontal direction and the distance between the nose position and the left corner of the mouth in the horizontal direction; The second positioning relationship is a second mean, which is the mean of the distance between the nose position and the right eye position in the horizontal direction and the distance between the nose position and the right corner of the mouth in the horizontal direction; The third positioning relationship is a third mean, and the third mean is the mean of the distance between the nose position and the left eye position in the vertical direction and the distance between the nose position and the right eye position in the vertical direction; The fourth positioning relationship is a fourth mean, and the fourth mean is the mean of the distance between the nose position and the left corner of the mouth in the vertical direction and the distance between the nose position and the right corner of the mouth in the vertical direction.

7. The method according to claim 6, It is characterized in that The first ratio value is a minimum value of a value obtained by dividing the first mean value by the second mean value and a value obtained by dividing the second mean value by the first mean value; The second proportional value is the minimum value of a value obtained by dividing the third mean by the fourth mean and a value obtained when the fourth mean is in a preset value, and the preset value is the product of the third mean and a preset proportional value.

8. The method according to claim 4, It is characterized in that Determining facial positioning information of each facial image of the next frame of image according to facial organ position information of each facial image of the next frame of image comprises: Determine the image center point of each facial image of the subsequent frame image; The distance between the nose position and the image center point of each facial image of the subsequent frame of image is determined as a value represented by the facial positioning information of each facial image of the subsequent frame of image.

9. The method according to claim 4, It is characterized in that Determining facial positioning information of each facial image of the next frame of image according to facial organ position information of each facial image of the next frame of image comprises: According to the number of facial organs represented by the facial organ position information of each facial image of the next frame image, the value represented by the facial positioning information of each facial image of the next frame image is determined.

10. The method according to claim 4, It is characterized in that Determining facial positioning information of each facial image of the next frame of image according to facial organ position information of each facial image of the next frame of image comprises: For each facial image of the next frame of image, determining a first connecting line between a left eye position and a right eye position, and determining a second connecting line between a left corner of the mouth position and a right corner of the mouth position; For each facial image of the subsequent frame of image, a value represented by the facial positioning information is determined according to a relative position relationship between the first connecting line and the second connecting line.

11. The method according to claim 3, It is characterized in that There is a positive correlation between the value represented by the facial positioning information and the value represented by the facial posture information; Alternatively, determining the facial posture information of each facial image of the subsequent frame image according to the facial positioning information of each facial image of the subsequent frame image comprises: Linear correlation processing is performed on the facial positioning information of each facial image of the subsequent frame image to obtain facial posture information of each facial image of the subsequent frame image.

12. The method according to any one of claims 1 to 2, It is characterized in that The method further comprises: Extract the global semantic features and local detail features of the image to be retrieved; Determine an image set in the preset image library that matches the image to be retrieved, according to the image global semantic features of the image to be retrieved and the image global semantic features of each image in the preset image library, wherein the image set includes at least one image; According to the image local detail features of the image to be retrieved and the image local detail features of each image in the image set, an optimal image in the image set that matches the image to be retrieved is determined.

13. The method according to claim 12, It is characterized in that Extracting the global semantic features of the image to be retrieved includes: Extracting a first feature map of the image to be retrieved based on a deep learning model; The first feature map is pooled to obtain a feature vector, and the feature vector is regularized to obtain the global semantic feature of the image.

14. The method according to claim 12, It is characterized in that Extracting local detail features of the image to be retrieved includes: Extracting SIFT features of the image to be retrieved; The SIFT features are aggregated to obtain a local feature matrix, and the local feature matrix is ​​regularized and dimensionally reduced to obtain the local detail features of the image.

15. The method according to claim 12, It is characterized in that Determining an image set in the preset image library that matches the image to be retrieved according to the image global semantic features of the image to be retrieved and the image global semantic features of each image in the preset image library, comprising: Determine a first similarity feature matrix between the image global semantic feature of the image to be retrieved and the image global semantic feature of each image in the preset image library; The image set is determined according to the numerical value represented by the first similarity feature matrix of each image in the preset image library.

16. The method according to claim 12, It is characterized in that Determining the best image in the image set that matches the image to be retrieved according to the image local detail features of the image to be retrieved and the image local detail features of each image in the image set includes: Determine a second similarity feature matrix between the image local detail feature of the image to be retrieved and the image local detail feature of each image in the image set; The optimal image is determined according to the value represented by the second similarity feature matrix of each image in the image set.

17. A facial tracking device, It is characterized in that include: An acquisition unit, used to acquire two adjacent frames of images in the video stream to be analyzed, and determine a facial image of each of the two adjacent frames of images; A first determining unit, configured to determine image feature information of each facial image of a latter frame of two adjacent frames of images according to the facial image of each frame of two adjacent frames of images; A second determining unit is used to determine facial posture information of each facial image of the latter frame of the two adjacent frames of images, wherein the facial posture information represents the degree of deviation of the face in the facial image; An adjustment unit, configured to adjust the image feature information of each facial image of the subsequent frame image according to the facial posture information of each facial image of the subsequent frame image to obtain a facial matrix; the image feature information includes: spatial distance information and feature distance information; A matching unit, configured to determine, according to the facial matrix, a matching relationship between each facial image in a previous frame of the two adjacent frames of images and each facial image in a subsequent frame of the image; The adjustment unit is specifically used to adjust the first weight corresponding to the feature distance information and the second weight corresponding to the spatial distance information of each facial image of the subsequent frame image according to the facial posture information of each facial image of the subsequent frame image; the first weight is positively correlated with the numerical value represented by the facial posture information, and the second weight is negatively correlated with the numerical value represented by the facial posture information; for each facial image of the subsequent frame image, a weighted calculation is performed according to the first weight, the feature distance information, the second weight, and the spatial distance information to obtain the feature value of each facial image of the subsequent frame image; and the facial matrix is ​​obtained according to the feature values ​​of each facial image of the subsequent frame image; The second determination unit is specifically used to extract facial organ position information of each facial image in the subsequent frame image, wherein the facial organ position information represents position information of at least one facial organ in the facial image on the facial image; and determine facial posture information of each facial image in the subsequent frame image based on the facial organ position information of each facial image in the subsequent frame image.

18. An electronic device, It is characterized in that include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor executes the method according to any one of claims 1 to 16.

19. A computer-readable storage medium, It is characterized in that Computer executable instructions are stored, and the computer executable instructions are configured to execute the method according to any one of claims 1 to 16.

20. A computer program product, It is characterized in that include: A computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 16 is implemented.