Face tracking processing method and electronic equipment

Abstract

The invention provides a face tracking processing method and electronic equipment, and the method comprises the steps: carrying out the face detection of a to-be-processed target frame image, determining the face position information and face orientation information of each to-be-processed face image contained in the target frame image, and carrying out the recognition of the to-be-processed face image based on the face position information and the face orientation information. Face matching is carried out on the corresponding to-be-processed face image and the tracked face image of the tracked object, the to-be-processed face image belonging to the untracked object in the target frame image is determined as the untracked face image, then face recognition is carried out on the untracked face image, and the face recognition result of the corresponding untracked object is determined.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to a face tracking processing method and electronic device. Background Art

[0002] With the development of artificial intelligence and edge computing, face tracking and detection technology has been widely used in multiple fields, such as intelligent transportation and driving, human-computer interaction and smart devices, medical care and health, education and office, as well as other special scenario applications such as sports analysis and robotics, to meet actual application needs.

[0003] Among them, in order to meet the high performance requirements in practical applications, a neural network-based prediction model can be used to correct the position information of the face area in the current frame image using offset parameters to predict the position information of the face area of ​​the same object in the next frame image, so as to realize face tracking detection, ensure the synchronization of image shooting and tracking processing speed, and avoid image loss.

[0004] However, this neural network-based face tracking processing method has high requirements for operator resources, and in scenarios such as multi-person video conferencing, it is easy to reduce the accuracy and reliability of face tracking due to dynamic changes in the position / posture of participants. Summary of the Invention

[0005] In view of the above problems, this application provides the following technical solutions:

[0006] The first aspect of the present application provides a face tracking processing method, the method comprising:

[0007] Performing face detection on the target frame image to be processed, and determining face position information and face orientation information of each face image to be processed contained in the target frame image;

[0008] performing face matching on the corresponding face image to be processed and the tracked face image of the tracked object based on the face position information and the face orientation information, and determining that the face image to be processed belonging to the untracked object in the target frame image is an untracked face image;

[0009] Performing face recognition on the untracked face image to determine a face recognition result corresponding to the untracked object.

[0010] In a possible implementation, performing face matching on the corresponding face image to be processed and the tracked face image based on the face position information and the face orientation information, and determining that the face image to be processed belonging to an untracked object in the target frame image is an untracked face image, includes:

[0011] Based on the face matching model, using the face position information and the face orientation information, face matching is performed on the corresponding face image to be processed with each tracking face image contained in the historical frame image to obtain a face matching result for the corresponding face image to be processed; the tracking face image is a face image of the tracked object;

[0012] In response to the face matching result being a face matching failure, the corresponding face image to be processed is determined as an untracked face image belonging to an untracked object.

[0013] In one possible implementation, based on the face matching model, using the face position information and the face orientation information, performing face matching on the corresponding face image to be processed with each tracked face image contained in the historical frame image, to obtain a face matching result for the corresponding face image to be processed, includes:

[0014] Based on the face matching model, using the face position information and the face orientation information, face matching is performed on the corresponding face image to be processed with each tracked face image contained in the same historical frame image, to obtain a pending matching result of the corresponding face image to be processed in the historical frame image;

[0015] The face matching result corresponding to the face image to be processed is obtained by utilizing the pending matching results of the same face image to be processed in a plurality of consecutive historical frame images.

[0016] In one possible implementation, performing face matching on the corresponding face image to be processed and the tracked face image of the tracked object based on the face position information and the face orientation information, and determining that the face image to be processed belonging to the untracked object in the target frame image is the untracked face image, includes:

[0017] performing normalized weighted processing on the face position information and the face orientation information corresponding to the same face image to be processed, respectively, to obtain face features to be matched corresponding to the face image to be processed;

[0018] Calculating the matching degree of each of the facial features to be matched and the tracked facial features of each tracked object to obtain a corresponding facial matching degree;

[0019] Based on the face matching degrees corresponding to the same face feature to be matched, the face image to be processed belonging to the untracked object is determined to be an untracked face image.

[0020] In one possible implementation, based on the face matching model, using the face position information and the face orientation information, performing face matching on the corresponding face image to be processed with each tracked face image contained in the historical frame image, to obtain a face matching result for the corresponding face image to be processed, includes:

[0021] Based on the Mahalanobis distance module included in the face matching model, the face position information, the face orientation information and the weight of each information are used to obtain the face matching degree between the corresponding face image to be processed and each tracked face image included in the historical frame image;

[0022] Based on the graph matching module included in the face matching model, and using the face matching degrees corresponding to the same face image to be processed, global nearest neighbor matching is performed on each face image to be processed and each tracked face image to determine the face matching results of each face image to be processed.

[0023] In a possible implementation, using the face matching degrees corresponding to the same face image to be processed, performing global nearest neighbor matching on each face image to be processed and each tracking face image, and determining a face matching result for each face image to be processed, includes:

[0024] In response to any of the face matching degrees being less than a matching threshold, determining that the face matching of the corresponding face image pair fails; the face image pair includes one of the face images to be processed and one of the tracked face images;

[0025] Eliminating the facial image pairs corresponding to the facial matching failures from all facial image pairs formed by the facial images to be processed and the facial images to be tracked, to obtain facial image pairs to be matched;

[0026] Using the face matching degree corresponding to each pair of face images to be matched, global nearest neighbor matching is performed on the pair of face images to be matched corresponding to each face image to be processed, to determine a face matching result for each face image to be processed.

[0027] In one possible implementation, the method further includes:

[0028] In response to the first number of frames of images containing facial images of the same untracked object, updating the untracked object to a tracked object;

[0029] In response to the second number of image frames not including the facial image of the same tracked object, the tracked object is updated to an untracked object.

[0030] In a possible implementation, performing face detection on the target frame image to be processed and determining face position information and face orientation information of each face image to be processed contained in the target frame image includes:

[0031] Performing face detection on the target frame image to be processed based on the face detection model, and determining face position information of each face image to be processed contained in the target frame image;

[0032] Performing face orientation detection on each of the face images to be processed based on the face orientation detection model, and determining the roll angle, pitch angle, and yaw angle of the face orientation image acquisition device in the corresponding face image to be processed as the face orientation information of the face image to be processed;

[0033] The method further comprises:

[0034] comparing the roll angle, the pitch angle, and the yaw angle corresponding to the untracked face image with their corresponding angle thresholds;

[0035] In response to the roll angle, the pitch angle, and the yaw angle corresponding to the untracked facial image being less than their respective corresponding angle thresholds, performing step 15 of performing facial recognition on the untracked facial image to determine a facial recognition result corresponding to the untracked object;

[0036] In response to at least one of the roll angle, the pitch angle, and the yaw angle corresponding to the untracked facial image being greater than or equal to a corresponding angle threshold, processing of the untracked facial image is terminated.

[0037] In a possible implementation, performing face recognition on the untracked facial image to determine a face recognition result corresponding to the untracked object includes:

[0038] Inputting the untracked face image into a face recognition model, performing feature extraction on the untracked face image through a feature extraction module included in the face recognition model to obtain a corresponding face feature vector;

[0039] Performing a distance calculation between the facial feature vector and a pre-stored reference facial feature vector by a similarity calculation module included in the facial recognition model to obtain a corresponding distance, and performing a similarity calculation between the facial feature vector and the pre-stored reference facial feature vector to obtain a corresponding similarity;

[0040] The decision module included in the face recognition model determines, based on the distance and the similarity between the face feature vector and the same reference face feature vector, that the face feature vector and the reference face feature vector belong to the same untracked object, and determines the pre-stored identity information of the untracked object as the face recognition result.

[0041] A second aspect of the present application further provides an electronic device, comprising: at least one communication element, at least one memory, and at least one processor, wherein:

[0042] The communication element is used to receive the target frame image to be processed;

[0043] The memory is used to store a plurality of computer instructions;

[0044] The processor is configured to execute the computer instructions to implement the following steps:

[0045] Performing face detection on the target frame image to determine face position information and face orientation information of each face image to be processed contained in the target frame image;

[0046] performing face matching on the corresponding face image to be processed and the tracked face image of the tracked object based on the face position information and the face orientation information, and determining that the face image to be processed belonging to the untracked object in the target frame image is an untracked face image;

[0047] Performing face recognition on the untracked face image to determine a face recognition result corresponding to the untracked object. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

[0049] Figure 1 A schematic diagram of the hardware structure of an optional embodiment of an electronic device applicable to the face tracking processing method proposed in this application;

[0050] Figure 2 This is a schematic diagram of an optional hardware structure when the electronic device applicable to the face tracking processing method proposed in this application is a terminal device;

[0051] Figure 3 A flowchart of a face tracking method provided in Example 1 of the present application;

[0052] Figure 4A flowchart of a face tracking method provided in Example 2 of the present application;

[0053] Figure 5 A flowchart of a face tracking method provided in Example 3 of the present application;

[0054] Figure 6 A flowchart of a face tracking method provided in Example 4 of the present application;

[0055] Figure 7 A flowchart of a face tracking method provided in Example 5 of the present application;

[0056] Figure 8 A flowchart of an optional scenario of a face tracking method provided in an embodiment of the present application;

[0057] Figure 9 A schematic diagram of performing global nearest neighbor matching using Mahalanobis distance on different facial images in a face tracking method provided in an embodiment of the present application;

[0058] Figure 10 A flowchart of a face tracking method provided in Example 6 of the present application;

[0059] Figure 11 A schematic diagram of the structure of a face tracking and processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation section of this application are only used to explain the specific embodiments of the present application and are not intended to limit this application. The embodiments of the present application are described below in conjunction with the drawings. It is known to those skilled in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0061] The terms "first", "second" etc. in the context of the present application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0062] It is understandable that before using the technical solutions disclosed in the embodiments of this application, the type of personal information involved in this application (such as facial images, etc.), scope of use, and usage scenarios should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0063] This application can be applied to many fields such as security and public safety, consumer electronics and smart devices, automobiles and smart driving, entertainment and social media, medical and health, retail and advertising marketing, online education and remote work, as well as industry and manufacturing. It can combine computer vision, artificial intelligence and sensor technology to realize intelligent applications in corresponding fields. It can also be combined with multimodal fusion, such as combining voice and gestures to enhance the interactive experience, and with the advancement of artificial intelligence algorithms and the improvement of hardware computing power, the application scenarios applicable to this application will be expanded. The following will take the video conferencing scenario as an example to introduce multiple application scenarios that have been implemented in products. It should be noted that this application can be applied to, but is not limited to, applications with image processing functions in terminal devices, or cloud services provided by cloud-side servers, or implemented by servers in conjunction with terminal devices, etc., depending on the situation.

[0064] Reference Figure 1 , is a hardware structure diagram of an optional embodiment of an electronic device applicable to the face tracking processing method proposed in this application. The electronic device can be a terminal device or a server. The electronic device is an example of a server. Figure 1 As shown, the electronic device may include but is not limited to: at least one communication element 110, at least one memory 120 and at least one processor 130, wherein:

[0065] The at least one communication element 110, the at least one memory 120, and the at least one processor 130 can communicate with each other via a bus. The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 1 The bus is represented by only one bidirectional line, but this does not mean that there is only one bus or one type of bus.

[0066] The communication element 110 can be used to receive a target frame image to be processed, such as receiving an original image captured in real time by an image acquisition device, or an original image of any frame to be processed that has been recorded as a video file, or reading a frame of original image from a database, etc. The present application does not limit the source of the target frame image. It can also be used to realize data or instruction transmission between internal components of an electronic device. Therefore, the communication element 110 may include a wireless communication element that supports one or more wireless communication methods, and an interface that supports one or more wired communication methods. The present application does not limit the composition structure of the communication element 110 to realize the corresponding communication function.

[0067] The memory 120 can be used to store multiple computer instructions for implementing the face tracking processing method proposed in the embodiment of the present application; the processor 130 can load and execute the multiple computer instructions stored in the memory 120 to implement the various steps of the face tracking processing method proposed in the embodiment of the present application. The implementation process can refer to the description of the corresponding part of the method embodiment below.

[0068] In the embodiment of the present application, the memory 120 may include a storage medium such as a floppy disk, a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. The processor 130 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a field-programmable gate array (FPGA).

[0069] It should be understood that Figure 1 The structure of the electronic device shown does not constitute a limitation on the electronic device in the embodiment of the present application. In actual applications, the electronic device may include Figure 1 In particular, when the electronic device is one or more terminal devices such as a mobile phone, a tablet computer, a wearable device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer or a robot, the electronic device may be more or less than the components shown, or some components may be combined. Figure 2As shown, the electronic device may further include: an image collector, an audio collector, an audio player, a display screen, a sensor module, an antenna, a power module, a radio frequency component, an external port, and other input / output components, etc. One or more components, Figure 2 It is only an example of an optional hardware structure of the terminal device and does not constitute a limitation on the hardware structure of the terminal device. It can be configured or adjusted according to actual functional processing requirements. This application does not give detailed examples one by one.

[0070] In order to solve the above problems, an embodiment of the present application provides a face tracking processing method. The face tracking processing method of the embodiment of the present application will be described in detail below in conjunction with the electronic device described above and the accompanying drawings.

[0071] Reference Figure 3 , is a flow chart of a face tracking method provided in the first embodiment of the present application. This embodiment can be applied to the above-mentioned electronic devices, such as Figure 3 As shown, the face tracking processing method provided in this embodiment may include but is not limited to the following steps:

[0072] Step S301, performing face detection on the target frame image to be processed, and determining the face position information and face orientation information of each face image to be processed contained in the target frame image;

[0073] In the embodiments of the present application, the target frame image can be an original image obtained by capturing an object of corresponding identity / category (e.g., pedestrians, drivers, athletes, consumers, students, children, patients, or production workers, or other categories of objects such as animals; this application uses the human object category as an example) in the fields listed above. In this case, the original image can be captured in real time by an image capture device deployed in the space where the object is located, or by an image capture device that moves with the object. The image capture device can be an independent camera / recorder, or a terminal device equipped with a camera as listed above. This application does not limit the product form of the image capture device, which can be determined based on the application requirements of the corresponding field.

[0074] In an optional implementation, for the image acquisition devices listed above, it is possible to enter an image shooting mode or a video shooting mode, and according to the method described above, the original image captured in real time can be directly determined as the target frame image. The captured video file (video stream) or image sequence can also be stored locally or uploaded to a server (which is performed when the electronic device is a server), and then the target frame image to be processed is read from the video file / image sequence.

[0075] For example, in a video conferencing scenario, an image acquisition device deployed in a conference room can capture images toward the interior of the conference room to capture participants entering the conference room. At this time, the original image captured in real time or periodically, or the image after pre-processing the original image, can be determined as the target frame image. It should be understood that as the participants change at will, the participants or their positions, postures, etc. contained in the original images of different frames captured by the image acquisition device may change. The present application can achieve face tracking and detection of participants through the method described in the following steps to achieve participant management (such as counting meeting attendance time, positional relationships, etc.), and can also automatically focus on the speaker to ensure that his or her facial image is presented on the conference display device.

[0076] Optionally, in the case of limited computing resources on the local terminal device in a video conferencing scenario, the image acquisition device can also upload the collected original image frames (image sequence) or video streams to the server, so that the server can execute the face tracking processing method proposed in this application. Similarly, in scenarios such as athlete tracking detection in a stadium, student tracking detection in a classroom, and employee tracking detection in a workshop, in order to meet the requirements of tasks such as attendance management, the target frame images in the corresponding scenarios can also be obtained according to the method described above. This application does not provide detailed examples of the implementation process one by one.

[0077] In another possible implementation, the above-mentioned target frame image can also be any frame of original image in an image sequence / video file from other data sources, such as film and television drama video files, variety show video files, etc. from the video sources corresponding to various video playback software, so as to realize the positioning / statistical analysis of information such as the appearance time, frame number or scene of different characters through the face tracking processing method provided by this application. In this case, the target frame image can be each frame of original image in the corresponding video file, etc. This application does not limit the source of the target frame image to be processed, and it can be determined according to the situation.

[0078] In the application scenarios listed above, this application will implement tracking detection of each object in the corresponding application scenario through face tracking detection to meet the actual task requirements. After obtaining the target frame image to be processed according to but not limited to the method described above, face detection can be performed on the target frame image to determine the face position information of each face image to be processed (which can be an image of the area where the face rectangular frame (also called the detection frame) is located) contained in the target frame image, such as the coordinates of the center point of the corresponding face rectangular frame (which can be represented by the pixel coordinates (x, y) in the image coordinate system) and the size (i.e., the face size / size, which can be represented by the width w and height h) and other spatial geometric attributes of the face.

[0079] Afterwards, face orientation detection can be performed on the face image to be processed within each face rectangular frame in the target frame image to obtain one or more posture geometric attributes of the corresponding face, such as deflection angle information such as yaw, pitch and roll of the face toward the image acquisition device (such as a camera, etc.), to form the face orientation information of the corresponding face image to be processed, which is used to reflect the spatial orientation of the corresponding face relative to the image acquisition device.

[0080] It can be seen that the present application uses the posture geometric attributes and spatial geometric attributes of the face in each face image to be processed to form a more complete geometric expression of the face, improve the accuracy of face tracking detection, and better meet the needs of subsequent tasks, such as expression driving in AR / VR scenarios, eye tracking or micro-expression control in human-computer interaction scenarios, etc. It should be noted that the process of obtaining the face orientation information of each of the above-mentioned face images to be processed is not limited to the execution order described above, and can also be obtained simultaneously with the face position information. In addition, the present application does not limit the respective content of the face position information and the face orientation information and their detection implementation methods, such as using an existing detection model or a pre-trained detection model for implementation, or based on a suitable detection algorithm, etc. This embodiment will not be described in detail here.

[0081] Step S302: performing face matching between the corresponding face image to be processed and the tracked face image of the tracked object based on the face position information and the face orientation information, and determining that the face image to be processed in the target frame image that belongs to the untracked object is an untracked face image;

[0082] In an embodiment of the present application, objects in any of the fields and scenarios listed above can be distinguished and represented by their facial images, and the state of the objects can be divided into a tracked state and an untracked state, so as to classify the objects into tracked objects and untracked objects. Among them, the tracked object can be the object corresponding to the facial image detected in the historical frame image (i.e., the target frame image for which face tracking detection has been completed). In order to improve the reliability and accuracy of face tracking detection, it can be the object whose facial image is detected in multiple consecutive historical frame images starting from the current frame. The untracked object can be the object corresponding to the facial image detected for the first time (i.e., a newly appeared facial image, that is, a facial image that has just entered the image acquisition area of ​​the image acquisition device) or the facial image that has been lost in tracking (i.e., a facial image that has re-entered the image acquisition area of ​​the image acquisition device, such as a face that appears again after the object has left, or a face that has been blocked by the object that has been restored).

[0083] It can be seen that as the objects and their positions in the actual scene change, the same object can dynamically switch between the tracked state and the untracked state, so that the tracked object and the untracked object change dynamically. That is to say, the face detection result of the untracked object meets the preset tracking conditions (such as detecting its face image or detecting its face in multiple consecutive frames of images) and can be updated to the tracked object. The face detection result of the tracked object meets the preset untracking conditions (such as the face tracking of the tracked object is lost due to factors such as occlusions in the image or changes in the position of the tracked object) can also be updated to the untracked object.

[0084] In an embodiment of the present application, the facial image of the tracked object can be recorded as a tracked facial image, and the facial image of the untracked object can be recorded as an untracked facial image. When the shooting scene / environment shown in the target frame image contains multiple objects, such as a multi-person video conferencing scene, a classroom teaching scene, etc., after executing the face tracking processing method proposed in this application for a period of time, usually, multiple tracked facial images corresponding to each tracked object will be obtained. Each tracked object and its tracked facial image can be stored in a list, array, queue or other data structure. This application does not limit the storage implementation method.

[0085] It should be understood that, as analyzed above, the tracked objects and their facial images stored by the electronic device, as well as the previously determined untracked objects, may dynamically update their status (tracked status and untracked status) as the actual detected facial image changes. When the originally stored tracked object is updated to an untracked object, the tracked object can be deleted from the storage space; after the untracked object is updated to a tracked object (that is, the corresponding untracked facial image is updated to a tracked facial image after tracking detection), it is added to the storage space, etc., to achieve dynamic update of the tracked object, thereby improving the reliability and accuracy of face tracking processing.

[0086] Among them, in order to facilitate the analysis of the face image to be processed detected from the subsequent frame image, the face position information and face orientation information of each continuously detected tracking face image of the tracked object in the corresponding frame image (these two types of information can be determined during the detection process of the corresponding tracking face image) can be associated and stored. In this way, during the implementation of step S302, the face position information and face orientation information of the tracking face image from the adjacent historical frame image can be directly read from the storage space to achieve facial geometric attribute matching between the face position information and face orientation information of the face image to be processed.

[0087] In one possible implementation, based on actual needs, this application can also assign a unique tracking identifier (such as a Track ID) to each identified tracked object, and can even configure a validity period or other validity conditions for the tracking identifier so that after the tracking identifier expires (such as when the above-mentioned untracking conditions are met), the corresponding tracked object can be promptly updated to an untracked object. It can be seen that the untracked objects in this application can be objects that have not been configured with a tracking identifier or whose originally configured tracking identifier has expired. The configuration or deletion of the tracking identifier can be dynamically updated based on the actual face detection results of each frame of the image. This application does not describe the implementation process in detail.

[0088] Combined with the above analysis, in the process of implementing face tracking and detection on the same object, since tracking and detection based on the face recognition algorithm often requires a lot of computing resources, which will affect the processing speed, after determining that the target frame image contains at least one face image to be processed, instead of spending a lot of time and computing resources to directly execute the face recognition algorithm on all face images to be processed, so as to implement face tracking and detection of the same object based on the determined object identity, this application proposes to first select the face images that are not currently tracked (that is, the face images belonging to the untracked objects, recorded as untracked face images) from all the face images to be processed contained in the target frame image, and then execute the face recognition process (such as the process of running the above-mentioned face recognition algorithm), and no longer execute the face recognition process for the face images that have been tracked (that is, the face images belonging to the tracked objects, recorded as tracked face images). It can be seen that the present application selectively executes the face recognition process for all face images to be processed. Even when all face images to be processed in the target frame image are tracking face images, there is no need to spend a lot of time and computing power resources to execute the face recognition process for the face images to be processed, which greatly reduces the computing burden of the electronic device and improves the processing speed.

[0089] Moreover, in the implementation process of selecting an untracked face image that requires face recognition, the embodiment of the present application combines the spatial geometric properties and posture geometric properties of the face to consider not only the position change of the face image, but also the face orientation change, that is, based on the face position information and face orientation information, each face image to be processed is matched with each currently stored tracking face image, that is, each face image to be processed is matched with each currently stored tracking face image for face position and face orientation, and according to the corresponding face position matching results and face orientation matching results, it is determined whether the face image to be processed belongs to an untracked face image or a tracked face image.

[0090] The face position matching result between each to-be-processed face image and each tracked face image can represent the spatial position correspondence of the same face in consecutive frame images or images captured from different camera perspectives. This can be obtained through methods such as overlap (e.g., intersection-over-union) of the face region (i.e., the region containing the face rectangle) or object motion prediction. It can also be expressed based on deep learning methods (e.g., similarity algorithms or distance matching algorithms) to measure the size similarity of the face images or the distance between the regions. In multi-camera scenarios, the face position matching result can be determined through 3D position estimation. The face orientation matching result between each to-be-processed face image and each tracked face image, combined with the above description of face orientation information, can be obtained by calculating the face offset angle difference between the two face images, or by converting the face orientation information into a feature vector and then obtaining the corresponding face orientation matching result through vector similarity / distance calculation methods. This application does not limit the implementation methods of face orientation matching and face position matching between different face images, which can be determined as appropriate.

[0091] In real-world scenarios, especially those involving non-high-speed motion capture, the position and orientation changes (i.e., changes in the same geometric attributes) of the face of the same subject in adjacent frames from the same camera perspective are often small. Thus, during the geometric attribute matching process between the tracked face image contained in the historical frame image and the currently determined face image to be processed, the differences in face position and face orientation between the face images obtained in adjacent frames can be calculated to determine whether the two face images are of the same subject. For example, if the face position difference and face orientation difference of two face images are both greater than a corresponding threshold, they do not belong to the face images of the same subject. The face image to be processed in the two face images can be considered a tracked face image, and no further face recognition is required for the face image to be processed. Conversely, if both are less than a corresponding threshold, the two face images can be considered not to belong to the face images of the same subject, and the face image to be processed can be determined as an untracked face image, and the subsequent face recognition process can be performed on it.

[0092] Optionally, the present application can also combine the actual shooting scene type to perform weighted matching on different types of geometric attributes, so as to use different weight values ​​to represent the degree of influence of the differences in different geometric attributes in the corresponding scene on the accuracy of judging the face of the same object. For example, the higher the degree of influence, the larger the weight value of the corresponding geometric attribute can be, so as to accurately obtain the matching result (such as similarity or difference value) between the corresponding two face images. If the difference value is greater than the difference threshold (or the similarity is less than the similarity threshold), it can be considered that the two face images do not belong to the face images of the same object, and then the subsequent face recognition process is performed on the face image to be processed. On the contrary, the face recognition process is no longer performed on the face image to be processed, which saves computing power resources and improves the processing speed of all face images to be processed (usually multiple here) contained in the target frame image. The present application does not limit the implementation method of how to determine whether two face images are the face of the same object.

[0093] In some other embodiments proposed in the present application, during the face matching implementation process of step S302, in addition to performing face matching based on face position information and face orientation information, matching can also be performed based on the relative positions and proportions of the detected facial features (such as eye distance, nose width and length, distance between mouth and chin, etc.), or one or more other matching methods such as matching based on the detected face shape and contour lines, thereby improving the detection accuracy of whether the processed face image belongs to a tracked object or an untracked object. The present application does not limit the implementation method of step S302.

[0094] Step S303 , performing face recognition on the untracked face image to determine a face recognition result of the corresponding untracked object.

[0095] Following the above analysis, for each face image to be processed detected from the target frame image, this application does not directly perform face recognition on each face image to be processed. Instead, it refers to the method described above to screen out the face images to be processed that belong to untracked objects and record them as untracked face images. Afterwards, face recognition is only performed on each untracked face image, such as matching facial features (such as geometric features such as face shape, proportion and key point position, texture features based on pixel distribution, depth features extracted based on deep learning, three-dimensional features, dynamic features such as micro-expressions and muscle movements, etc.) between the untracked face image and pre-stored reference face images of objects with different identities (such as one or more similarity algorithms) to identify the identity of the corresponding untracked object.

[0096] As for the method described above, the face images to be processed that belong to the tracked objects are determined from all the face images to be processed. Since their object identities have been obtained through face recognition during the processing of historical frame images, the corresponding identified object identities (such as the face recognition results obtained previously) can be read as needed. During the processing of the target frame images, there is no need to perform face recognition on such face images to be processed, which saves computing resources and enables the present application to be applicable to deployment on more hardware platforms, especially embedded platforms with lower computing power.

[0097] Reference Figure 4 , is a flow chart of a face tracking method provided in the second embodiment of the present application. This embodiment proposes that in the above face tracking method, a multi-model collaborative architecture is used to achieve efficient and stable face recognition and tracking. Based on this, Figure 4 As shown, the face tracking processing method implemented by multi-model collaboration in this embodiment may include but is not limited to the following steps:

[0098] Step S401, performing face detection on the target frame image to be processed based on the face detection model, and determining the face position information of each face image to be processed contained in the target frame image;

[0099] In an embodiment of the present application, in order to reduce the computational burden of electronic devices and increase processing speed, a lightweight and efficient face detection model can be used to perform preliminary face detection on the target frame image to be processed in the input image sequence / video stream (here it can be the current frame image, and the target frame image is updated frame by frame or after a specified frame interval), and determine the position information of each face rectangle box (detection box) in the target frame image, such as the center point coordinates, width and height of the face rectangle box, etc. The image of the area where each face rectangle box is located is a face image to be processed, and the position information of the face rectangle box is recorded as the face position information. It should be noted that the present application can also obtain the target frame image to be processed according to the method described above, and then input the target frame image into the face detection model for face detection to obtain the face position information of each face image to be processed contained in the target frame image.

[0100] In some embodiments, the face detection model may be, but is not limited to, the FaceDet model to better meet real-time face detection tasks. Alternatively, a face detection model optimized for face detection may be used, such as MTCNN (Multi-Task Cascaded Convolutional Networks), SCRFD (Sample and Computation Redistribution for Efficient Face Detection), a highly efficient face detection model and lightweight single-stage detector that meets the needs of face detection in images captured by real-time cameras, RetinaFace (Single-stage Dense Face Localization in the Wild), a single-stage multi-task convolutional neural network face detection model based on RetinaNet, or FaceBoxes (a deep learning-based face detection model with a single-stage detection design that can directly predict the location of faces in an image, meeting low computational resource requirements).

[0101] Of course, the above-mentioned face detection model can also be a general target detection model such as the YOLO series or SSD (Single Shot MultiBoxDetector) or EfficientDet (Scalable and Efficient ObjectDetection, scalable and efficient target detection model); or a dedicated face detection model that is more suitable for crowded scenes, such as AdaptiveScale Face Detector. In addition, this application can also use other open source tools and frameworks to form a face detection model to achieve face detection in the target frame image. This application does not limit the type and source of the face detection model. The face detection model can be flexibly selected according to the actual scene requirements (such as real-time or accuracy, etc.) and hardware resources (such as CPU or image processor GPU, etc.). The selected face detection model can even be fine-tuned according to actual needs to achieve more accurate and efficient image face detection. The implementation process is not described in detail in this application.

[0102] Step S402: performing face orientation detection on each face image to be processed based on a face orientation detection model to determine face orientation information of the corresponding face image to be processed;

[0103] In this embodiment of the present application, each facial image to be processed detected from the target frame image, i.e., the image region containing the facial rectangle (which can be determined based on the facial position information), can be input into a face orientation detection model. The facial deflection angle (such as the pitch angle, yaw angle, and roll angle described above) of each facial image to be processed is evaluated to obtain corresponding facial orientation information. Optionally, steps S401 and S402 can also be performed in parallel. In this case, the target frame image can be directly input into the face orientation model to detect the deflection angle of each face and obtain corresponding facial orientation information.

[0104] During the face detection model's face orientation detection process for the processed face image, different face detection models of different types / structures implement different face orientation detection methods. One possible implementation involves traditional methods based on geometric features, such as detecting facial key points (e.g., six or 68 key points such as the corners of the eyes, the tip of the nose, and the corners of the mouth), then using a PnP (Perspective-n-Point) algorithm to solve a rotation matrix and convert it into facial deflection angles, which provide facial orientation information. For cases where the face is barely deflected, the yaw angle can also be estimated by analyzing the symmetry between the left and right sides of the processed face image.

[0105] To improve the accuracy of face orientation detection, deep learning methods can also be used. In this case, the face detection model can be a trained deep convolutional neural network model that directly outputs the three deflection angles of each face. Alternatively, it can be a multi-task learning model that simultaneously predicts facial landmarks and orientation angles. Alternatively, models such as PRNet (Position Map Regression Network, a deep learning model used to predict UV position maps and convert them into 3D poses) or 3DDFA (3D Dense Face Alignment) can be used to solve for face pose and obtain face orientation information based on 3D face modeling. Alternatively, to balance the accuracy and speed of face orientation detection, a hybrid approach based on geometric features and deep learning, as described above, can be used to construct a face orientation detection model to detect the face orientation of the processed face image. Of course, this application can also dynamically select a suitable face detection model to implement step S402 based on the processing requirements of the actual application scenario, such as one or more of lightweight and real-time detection requirements, large-angle posture detection requirements, occlusion, lighting changes, etc. This application does not limit the type of face detection model.

[0106] In some embodiments, after obtaining the target frame image to be processed, in an optional implementation of steps S401 and S402, the target frame image can be input into a detection model capable of detecting both face position and face orientation to perform face detection, thereby obtaining face position information and face orientation information for each face image to be processed contained in the target frame image. For example, the face detection model and face orientation detection model described above can be integrated to form a detection model, or a new detection model supporting both face position and face orientation detection can be trained.

[0107] Step S403: Based on the face matching model, using the face position information and the face orientation information, the corresponding face image to be processed is matched with each tracked face image contained in the historical frame image to obtain a face matching result for the corresponding face image to be processed;

[0108] Step S404: in response to the face matching result being a face matching failure, determining the corresponding face image to be processed as an untracked face image belonging to an untracked object;

[0109] In the embodiment of the present application, in combination with the above description of the face matching process of step S302, the face matching model supports face position matching and face orientation matching capabilities, and outputs a corresponding face matching result of whether a face image to be processed and a tracking face image belong to the same object face based on the face position matching result and the face orientation matching result. Among them, the tracking face image is the face image of the tracked object. It can be understood that when an object in the current frame image is identified as a tracking object, the previous frame image or even multiple adjacent historical frame images contain the face image of the object. Therefore, the historical frame image in step S403 can be the previous frame image to be processed that is adjacent to the target frame image, or include multiple consecutive frames of frame images to be processed that are adjacent to the target frame image.

[0110] To reduce computational burden during face matching, this application converts the facial position and orientation information of a processed face image and a tracked face image into feature vectors (i.e., non-facial feature vectors, distinct from the facial feature vectors used in face recognition). This application then constructs a face matching model using an appropriate similarity / distance matching algorithm to perform matching calculations between the corresponding non-facial feature vectors to obtain a face matching result. This application does not restrict the types of matching algorithms supported by the face matching model; these algorithms may be used as needed.

[0111] In order to improve the reliability and stability of face matching, ensure the continuity and stability of the face presentation of the same object in the continuous frame images / videos finally output for the corresponding shooting scene, so as to solve the problem of objects entering the shooting scene (such as participants in a multi-person video conferencing scene) disappearing briefly (that is, leaving the image acquisition range of the image acquisition device, resulting in the corresponding frame image acquired no longer containing the object), and the problem of objects unrelated to the events in the shooting scene (such as a meeting) flashing (that is, briefly entering the image acquisition range of the image acquisition device) and then disappearing. That is, such objects will not always exist in multiple consecutive (the value can be dynamically configured or adjusted according to actual needs or experience. In order to ensure the reliability of matching, it can be configured to a larger value. The value is not restricted in this application) historical frame images. Such objects are not stable tracked objects. In order to avoid mistakenly identifying the face images to be processed of such objects as tracked face images, this application preferably performs face matching on the face images to be processed with the face images contained in multiple consecutive historical frame images to be processed, so as to combine the pending matching results corresponding to multiple historical frame images to accurately identify whether the face image to be processed belongs to a tracked object or an untracked object.

[0112] Based on this, an optional implementation method of the above-mentioned step S403 may include but is not limited to: based on the face matching model, using face position information and face orientation information, the corresponding face image to be processed is respectively matched with each tracking face image contained in the same historical frame image to obtain the pending matching result of the corresponding face image to be processed in the historical frame image; using the pending matching results of the same face image to be processed in multiple consecutive historical frame images to obtain the face matching result of the corresponding face image to be processed.

[0113] For example, the present application can determine a pending face image that successfully matches the face of multiple consecutive historical frame images (such as matching the same tracking face image to all historical frame images, or the number of frames that continuously match the same tracking face image is greater than a preset number threshold, and the number threshold is less than or equal to the number of frames of the historical frame images) as a tracking face image, without the need to perform face recognition on it; and determine a pending face image that fails to match the face of a historical frame image (such as not matching the tracking face image of any historical frame image; or the number of frames that successfully match continuously is less than a preset number threshold, etc.) as an untracked face image, thereby avoiding the situation in which, in a scenario where the object flashes and disappears, the pending face image of the object is mistakenly determined as a tracking face image when only one or two or other small number of historical frame images are successfully matched, and face recognition is stopped, resulting in the inability to track and detect the object, affecting subsequent tasks.

[0114] In some other embodiments, during the implementation of step S403, the face matching model may also first accurately identify each tracked face image, that is, after the same face image is contained in multiple consecutive historical frame images, the face image is determined to be a tracked face image, and then the face position information and face orientation information of the face image to be processed are matched with each tracked face image. If the match is successful, it is determined that the face image to be processed is a tracked face image of the same tracked object, and the identity of the object is known, and there is no need to perform object identity recognition through the face recognition algorithm; if the match fails, it is determined that the face image to be processed is an untracked face image, and its object identity is unknown, and subsequent face recognition steps need to be performed to determine the object identity of the untracked face image.

[0115] Step S405: Input the untracked face image into the face recognition model to perform face recognition, and obtain a face recognition result of the corresponding untracked object.

[0116] The face recognition model can be a model built based on a face recognition algorithm, which is used to extract discriminative feature vectors (i.e., face embedding vectors, also known as the face feature vectors described above) from face images and perform identity recognition by comparing the feature vectors. This application can select an appropriate face recognition model to implement step S405 based on actual scenario requirements and hardware resources. This application does not limit the type of face recognition model or its network structure.

[0117] In some embodiments, to meet the high-precision requirements of face recognition and achieve industrial-level recognition performance, face recognition models such as ArcFace (Additive Angular Margin Loss, enhancing inter-class separation) or CosFace (Cosine Margin Loss) can be selected for implementation. To meet real-time requirements and be suitable for low-power, mobile, and embedded device scenarios, face recognition models composed of lightweight convolutional neural networks such as MobileFaceNet (face recognition network on mobile devices) and ShuffleFaceNet (A Lightweight Face Architecture for Efficient and Highly-Accurate Face Recognition‌‌, a highly efficient and lightweight face recognition architecture) can be selected for implementation. To meet the requirements of 3D face recognition and be suitable for anti-counterfeiting and high-security scenarios, a face recognition model such as 3D-PatchNet (a deep learning-based 3D object detection project used to process 3D face point cloud features) can be combined with a depth sensor (used to obtain 3D point cloud or structured light data, etc.), or a cross-spectral method that jointly trains visible light and infrared images can be selected. FaceNet is a facial recognition model that supports nighttime surveillance scenarios. Of course, this application can also choose other facial recognition models based on deep learning, or facial recognition models built based on traditional methods (such as principal component analysis, linear discriminant analysis, maximizing inter-class differences, minimizing intra-class differences, etc.). This application does not provide detailed examples here.

[0118] In summary, in the embodiments of the present application, it is proposed to implement the corresponding processing steps in the face tracking processing method proposed in the present application by integrating a multi-model collaborative architecture of a face detection model, a face orientation detection model, a face matching model, and a face recognition model, so as to achieve efficient and stable face recognition and tracking. Even in a multi-person shooting scene (such as a complex environment with multiple participants), face tracking detection can be correctly and reliably achieved to maintain good tracking performance. Moreover, by selectively performing face recognition on all face images to be processed, the computational burden is reduced and the processing speed is improved. The present application can also dynamically adjust the model types according to actual scene requirements and / or hardware resources, thereby expanding the scope of application of the present application.

[0119] Reference Figure 5, is a flow chart of a face tracking method provided in the third embodiment of the present application. The present embodiment of the present application can describe an optional implementation process of performing face recognition on untracked face images in the face tracking method described above. For the method of determining untracked face images, please refer to the description of the corresponding part of the context embodiment. Figure 5 As shown, an optional implementation process of the above step S405 may include but is not limited to the following steps:

[0120] Step S501: Inputting an untracked face image into a face recognition model, performing feature extraction on the untracked face image through a feature extraction module included in the face recognition model to obtain a corresponding face feature vector;

[0121] Step S502: Calculate the distance between the facial feature vector and the pre-stored reference facial feature vector using a similarity calculation module included in the face recognition model to obtain a corresponding distance, and calculate the similarity between the facial feature vector and the pre-stored reference facial feature vector to obtain a corresponding similarity.

[0122] Step S503: The decision module included in the face recognition model determines, based on the distance and similarity between the face feature vector and the same reference face feature vector, that the face feature vector and the reference face feature vector belong to the same untracked object, and determines the pre-stored identity information of the untracked object as the face recognition result.

[0123] In the embodiments of the present application, the face recognition model may employ, but is not limited to, the model categories listed above. It may include a feature extraction module, a similarity calculation module, and a decision module to accurately identify untracked facial images in accordance with the face tracking processing method proposed in this application. Optionally, the face recognition model may also include an output module to output the face recognition results obtained by the decision module.

[0124] Among them, the feature extraction module is used to map the input untracked face image to a low-dimensional, high-discriminative feature space, and the obtained face embedding vector is used as the face feature vector of the untracked face image. The feature extraction process can make the feature distance of the same object close and the feature vectors of different objects far. According to the actual scene requirements, the appropriate category of feature extraction network can be selected as the feature extraction module in the face recognition model. The implementation process can refer to the selection and implementation method of the face recognition model in the above embodiment.

[0125] In the face recognition model, the similarity calculation module is used to measure the similarity between the facial feature vector of the person to be identified and the reference facial feature vector (which can be the facial feature vector of the facial image stored during object registration and obtained by the feature extraction module), thereby assisting the decision module in determining whether the person belongs to the same identity. Depending on actual needs, one or more combinations of similarity algorithms, such as cosine similarity, Euclidean distance, and Mahalanobis distance, can be used. To improve the reliability and accuracy of object identity recognition, it is preferable to construct a similarity calculation module based on multiple similarity algorithms, so that the decision module can integrate the similarity calculation results (such as similarity scores) of each similarity algorithm for the facial feature vector and the reference facial feature vector, or verify each other, to accurately determine the facial recognition result for the corresponding untracked facial image. This application does not limit the combination of multiple similarity algorithms, including but not limited to the two similarity algorithms described in step S502 that respectively implement similarity calculation and distance calculation, such as performing step S502 using a similarity calculation module based on a cosine similarity algorithm and a Euclidean distance algorithm.

[0126] In some embodiments, the above-mentioned similarity calculation module can also be a network based on metric learning, such as a similarity calculation module obtained by learning the similarity function through a dual-tower structure; it can also be a similarity calculation module based on the attention mechanism, to highlight the key area features in the facial feature vector through attention weights, and improve the robustness to occlusion and posture changes, etc. This application does not impose any restrictions on the similarity calculation methods supported by the similarity calculation module, which can be determined in combination with the hardware resources of the model deployment device and the processing requirements of the application scenario.

[0127] Following the above analysis, during the implementation of step S503, the decision module determines whether the facial feature vector of the untracked object and the reference facial feature vector belong to the same object based on the various similarity calculation results obtained by the similarity calculation module, such as the distance and similarity between the facial feature vector calculated in step S502 of this embodiment and the reference facial feature vector. The smaller the distance and the greater the similarity, the greater the probability that the facial feature vector and the reference facial feature vector belong to the same object. Based on this, in a possible implementation, the present application compares the distance and similarity with the corresponding pre-configured thresholds respectively. If it is determined that the similarity is greater than or equal to the similarity threshold, and the distance is less than the distance threshold, it can be considered that the facial feature vector and the reference facial feature vector belong to the same object, and the identity information stored corresponding to the reference facial feature vector is determined as the face recognition result; on the contrary, if it is determined that the similarity is less than the similarity threshold, or the distance is greater than or equal to the distance threshold, it can be considered that the facial feature vector and the reference facial feature vector do not belong to the same object, and the identity information of the corresponding untracked object can be identified by other means, such as determining the similarity between the two feature vectors based on other similarity algorithms, and identifying whether the two feature vectors belong to the same object by comparing with the corresponding configured threshold. Afterwards, the majority of the three or more recognition results are used as the final recognition result.

[0128] Optionally, the present application can also output the corresponding untracked facial image for the user or relevant management personnel to input their identity information and obtain the facial recognition result of the untracked facial image. Of course, the present application can also instruct the untracked object to register its identity and store its reference facial feature vector (which can be the facial feature vector processed this time, or the facial feature vector obtained by feature extraction of the facial image uploaded by the untracked object based on the facial recognition model, etc.), and determine the identity information registered by the untracked object as the facial recognition result to implement subsequent face tracking processing. Combined with the above analysis, the untracked object can be used as the object to be tracked. With the face detection and matching results of multiple consecutive frames of images in the future, if the facial image of the untracked object is detected in multiple consecutive frames of images in the future, the untracked object / object to be tracked can be updated to a tracked object, and there is no need to perform facial recognition on its facial image in the future.

[0129] Among them, the above threshold value can be a critical value for determining whether the facial feature vector of an untracked object and the reference facial feature vector belong to the same object. The threshold values ​​configured for different similarity algorithms can be different and can be configured based on actual needs or experience. This application does not limit the size of each threshold value. In actual applications, for similarity calculation modules with different performance and actual scenarios with requirements for face tracking detection accuracy, the threshold values ​​of the same category pre-configured in this application can be different. In other words, this application can dynamically configure threshold values ​​to improve face recognition accuracy.

[0130] In some other embodiments, during the implementation of step S503, the decision module may also perform a weighted summation of the normalized similarity calculation results to obtain a target similarity, and then compare the result with a corresponding threshold to determine whether the facial feature vector and the reference facial feature vector belong to the same untracked object. The weight coefficients corresponding to the different similarity algorithms are generally different. These weight coefficients may be configured based on the performance (e.g., accuracy, reliability, etc.) of the different similarity algorithms in actual scenarios. Similarity algorithms with higher performance may have correspondingly larger weight coefficients. This application does not impose any restrictions on the values ​​of the weight coefficients.

[0131] In order to improve data security, when a user registers and uploads a personal image, after processing it with the face recognition model, the user's reference face feature vector is obtained, which can be encrypted and stored, or written to read-only storage space for secure storage to protect user privacy and data security, avoid illegal adjustments to the reference face feature vector, and reduce the reliability and security of face tracking processing. This application does not limit the method for implementing secure storage of the reference face feature vector.

[0132] In summary, after screening out untracked face images from all the face images to be processed contained in the target frame image, such as face images detected for the first time, or face images that reappear after the face tracking of the tracked object is lost, etc., this application will perform face recognition on each untracked face image based on the face recognition model. In this face recognition process, it is proposed to use multiple similarity algorithms to measure whether the currently recognized face feature vector and the reference face feature vector belong to the same object. By mutual verification or fusion measurement, the similarity between the two face feature vectors is determined to accurately and reliably realize the identity recognition of the same object. This avoids the situation in which a single similarity algorithm is selected to realize face recognition, and the similarity calculation result obtained is unreliable because the similarity algorithm is not suitable for the actual application scenario, which affects the reliability and accuracy of the face recognition result obtained thereby, thereby improving the accuracy and reliability of face recognition.

[0133] Reference Figure 6, is a flow chart of a face tracking method provided in the fourth embodiment of the present application. The embodiment of the present application can describe an optional implementation process of how to perform face matching between the face image to be processed and the tracking face image based on the face matching model in the face tracking method described above, such as Figure 6 As shown, the optional implementation process may include but is not limited to:

[0134] Step S601, performing normalized weighted processing on the facial position information and facial orientation information corresponding to the same facial image to be processed, to obtain facial features to be matched for the corresponding facial image to be processed;

[0135] Because face position information and face orientation information have different information units and characteristics (such as domain and change speed), in order to participate in subsequent unified calculations and improve the robustness and accuracy of the model, they need to be normalized first and converted into information under the same measurement method. There is no restriction on the normalization implementation method.

[0136] In addition, the present application takes into account that the reliability and accuracy of various face position information and face orientation information detected from the same face image to be processed based on the face detection model and the face orientation detection model are different, resulting in different effects on the subsequent face matching results. The present application hopes to appropriately reduce the participation of information that has an adverse effect on the face matching result, or has a high degree of adverse effect, or has a low degree of favorable effect, in the subsequent face matching process, thereby reducing its degree of influence on the face matching result; while appropriately increasing the participation of information that has a favorable effect on the face matching result, or has a high degree of favorable effect, in the subsequent face matching process, thereby increasing its degree of influence on the face matching result, thereby improving the accuracy and reliability of face matching based on face position information and face orientation information.

[0137] For example, differences in shooting parameters such as the shooting angle of the image acquisition device during image capture can cause random variations in the facial size information of the facial image being processed, resulting in moderate accuracy and reliability of the size information (such as the width and height of the facial rectangle) detected in the facial position information. However, this does not interfere with the detection of the facial center point, resulting in high accuracy and reliability of the center point coordinates of the facial position information. Furthermore, the performance of the facial orientation detection model is affected, resulting in moderate accuracy and reliability of the estimated facial orientation information. To address this issue, the influence of the facial center point coordinates on the face matching results can be increased, while the influence of the facial size and orientation information can be appropriately reduced. This can be achieved by configuring different weights. The information with higher influence (i.e., higher accuracy and reliability) (each piece of facial position information and each piece of facial orientation information) is assigned a larger weight, with the sum of the weights being 1. This application does not impose any restrictions on the size of these weights or their configuration and implementation.

[0138] Therefore, after obtaining the normalized information corresponding to each facial position information and each facial orientation information of each facial image to be processed, the corresponding normalized information can be weighted summed based on the weights corresponding to each facial position information and each facial orientation information. After feature extraction, the facial features to be matched for the facial image to be processed can be obtained. If the normalized information is a normalized facial feature, the facial features to be matched for the facial image to be processed can be weighted summed based on the weights.

[0139] Step S602, calculating the matching degree of each to-be-matched facial feature with the tracked facial features of each tracked object to obtain a corresponding facial matching degree;

[0140] Step S603 : Based on the face matching degrees corresponding to the same face feature to be matched, the face image to be processed belonging to the untracked object is determined as the untracked face image.

[0141] Regarding the implementation process of step S602, reference can be made to the above description of the similarity calculation module. This embodiment can adopt one or more combined similarity algorithms, such as cosine similarity or vector distance calculation method, to calculate the matching degree of different facial features. The implementation process is not described in detail in this application.

[0142] According to the method described above, after determining the face matching degree between each face image to be processed and the tracked face image of each tracked object, each face matching degree can be compared with the matching threshold. If there is a face matching degree greater than or equal to the matching threshold, it means that the tracked face image corresponding to the face matching degree matches the face image to be processed, and can be considered to be the face images of the same object; if each face matching degree is less than the matching threshold, it means that the face image to be processed does not match any of the tracked face images, and it can be considered that the face image to be processed belongs to the face image of an untracked object, and it is determined to be an untracked face image, and the subsequent face recognition process is continued to identify the identity information of the untracked object.

[0143] Preferably, in order to further improve the reliability and accuracy of face matching, the present application can perform face matching on each to-be-processed face image in the target frame image with the tracking face image contained in each historical frame image in a plurality of consecutive historical frame images according to the method described in steps S601 to S603 above, and use the face matching degrees corresponding to each to-be-processed face image and each historical frame image to comprehensively determine whether the to-be-processed face image belongs to the face image of an untracked object, so as to avoid misidentifying the face image of an untracked object flashing in the historical frame image as the tracking face image, resulting in inaccurate face matching results for the unprocessed face image, and further resulting in continued face recognition of the face image of the tracked object, increasing the computational burden of the electronic device, or failing to perform face recognition on the face image of the untracked object, thus missing its face tracking.

[0144] It can be seen that in the face tracking processing method proposed in each embodiment of the present application, in order to ensure the continuity and stability of the presentation of the facial image of the object in the actual scene, so as to improve the presentation of details while ensuring the experience of the object participating in the scene event, such as the stable presentation of the facial image of the speaker in a multi-person video conference scene to ensure the conference experience, life cycle management can be introduced, such as maintaining a life cycle table to record the status of each object, such as the tracked state or the untracked state, to improve the reliability and accuracy of face matching in the case of a short disappearance of the object and the flash and disappearance of irrelevant objects in the scene. Among them, the life cycle can be the number of frames in which the object presents the facial image in the continuous frame images when determining that the object belongs to the tracking object. A sliding window with a length of this number of frames can be formed, and the multiple frames contained therein are dynamically updated to achieve dynamic update of the object state according to the face matching method described above.

[0145] Based on this, in a possible implementation, in response to the first number of frame images containing facial images of the same untracked object, the untracked object is updated to a tracked object. At this time, the facial image of the untracked object can be updated to a tracked facial image, and it can be added to the storage space for storing tracked objects and their tracked facial images; in response to the second number of frame images not containing facial images of the same tracked object, the tracked object is updated to an untracked object. At this time, the tracked facial image stored in the tracked object can be updated to an untracked facial image, and it can be deleted from the storage space to avoid subsequent face matching with unprocessed facial images in future frame images, thereby ensuring the reliability and accuracy of face tracking detection in future frame images.

[0146] Among them, the first number and the second number can be the same, such as the length of the sliding window mentioned above. In some embodiments, the first number and the second number may also be different, and can be determined based on the tracking accuracy requirements for tracked objects and untracked objects. For example, the higher the required tracking accuracy, the larger the corresponding configuration number. Since the larger the number, the greater the demand for computing resources, the present application can also consider the hardware resources of the electronic device and reasonably configure the respective values ​​of the first number and the second number. The present application does not limit the respective values ​​of the first number and the second number and their configuration implementation methods.

[0147] In some embodiments, in order to solve the problem of mismatching when faces in images are crossed and then separated, reduce the probability of mismatching faces, ensure the accuracy of face matching in complex scenes with multiple objects and maintain good tracking performance, the present application proposes to construct a face matching model based on the global nearest neighbor matching algorithm (GNN) and the similarity algorithm (also called the matching algorithm) to achieve face position matching and face orientation matching between the face image to be processed and the face images in the historical frame image.

[0148] Based on this, refer to Figure 7 , is a flowchart of a face tracking method provided in the fifth embodiment of the present application. The embodiment of the present application can further refine the face tracking method described above, such as Figure 7 As shown, the face tracking processing method proposed in this embodiment may include but is not limited to:

[0149] Step S701, performing face detection on the target frame image to be processed, and determining the face position information and face orientation information of each face image to be processed contained in the target frame image;

[0150] Regarding the implementation process of step S701, reference may be made to the description of the corresponding part of the above embodiment, and this embodiment will not be described in detail here.

[0151] Step S702: Based on the Mahalanobis distance module included in the face matching model, using the face position information, face orientation information, and the weights of each information, obtain the face matching degree between the corresponding face image to be processed and each tracked face image included in the historical frame image;

[0152] In the embodiments of this application, the Mahalanobis distance module is a calculation module that matches different facial images based on the Mahalanobis distance algorithm. Because the Mahalanobis distance algorithm considers the correlation between different features, when there is a linear correlation between features (such as the distribution of features under different postures or lighting conditions), other similarity algorithms such as the Euclidean distance algorithm are more able to reflect the actual sample differences (i.e., facial differences). Therefore, to improve the reliability and accuracy of face matching, this application chooses to use the Mahalanobis distance algorithm in the construction of the face matching model.

[0153] Combine Figure 8 As shown in the flowchart, the face position information of the face image to be processed obtained by the face detection model in this application may include: four dimensional information of the face center coordinates (x, y) and size (w, h), where w represents the width of the face rectangle frame and h represents the height of the face rectangle frame; the face orientation information of the face image to be processed obtained based on the face orientation detection model includes: three dimensional information of roll angle (roll), pitch angle (pitch) and yaw angle (yaw), and the seven dimensional information of the face image to be processed can be spliced ​​to form the geometric attributes of the face image to be processed, which can be expressed as (x, y, w, h, roll, pitch, yaw), and then can be input into the Mahalanobis distance module, which performs normalization processing based on the characteristics of the geometric attributes of each dimension, and then performs weighted fusion (i.e., summation) based on the information weights of the geometric attributes of each dimension to obtain the Mahalanobis distance MD of the seven-dimensional geometric attributes of the corresponding face image to be processed. After that, it can be subtracted from the Mahalanobis distance of each stored tracked face image (which is also calculated based on the Mahalanobis distance module). The obtained Mahalanobis distance difference represents the difference between the face image to be processed and the corresponding tracked face image, which can be recorded as cost. The present application can also convert the Mahalanobis distance difference into a face matching degree (such as a similarity score) between the face image to be processed and the corresponding tracked face image. The smaller the Mahalanobis distance difference, the higher the corresponding face matching degree. The present application does not impose any restrictions on the conversion relationship between the face matching degree and the Mahalanobis distance difference.

[0154] Exemplarily, according to the method described in the above embodiment, based on the Mahalanobis distance module, the normalized features of each geometric attribute (x0, y0, w0, h0, roll0, pitch0, yaw0) of each face image to be processed are weighted and summed to obtain the Mahalanobis distance MD0, that is, MD0 = A×f1(x0)+B×f1(y0)+C×f2(w0)+D×f2(h0)+E×f3(roll0)+F×f4(pitch0)+G×f5(yaw0); similarly, the normalized features of the geometric attributes (x1, y1, w1, h1, roll1, pitch1, yaw1) of each tracked face image are weighted and summed to obtain the Mahalanobis distance MD1 of the tracked face image, that is, MD1 =A×f1(x1)+B×f1(y1)+C×f2(w1)+D×f2(h1)+E×f3(roll1)+F×f4(pitch1)+G×f5(yaw1). A cost function constructed based on the Mahalanobis distance can then be used to obtain the Mahalanobis distance difference between different facial images (e.g., |MD0-MD1|), representing the cost difference between the two corresponding facial images. For example, cost(0,1) =ABS(MD1-MD0), where ABS represents the absolute value function. This application can use the cost to reflect the facial matching degree between the corresponding facial image to be processed and the tracked facial image, or, based on the inverse proportional relationship between the cost and the facial matching degree, convert it into a facial matching degree for subsequent processing.

[0155] Among them, A, B, C, D, E, F, and G represent the information weights of the geometric attributes of the corresponding dimensions, respectively. The higher the precision (such as the accuracy and reliability mentioned above), the greater the corresponding information weight of the geometric attribute. It can be seen that this information weight can reflect the importance of the corresponding geometric attribute and the degree of influence on the subsequent face matching results. This application does not restrict the numerical value of each information weight, and the information weight can be learned based on experience or historical data. For example, if the information weight is configured based on the characteristics of the camera image: A=B=1.0, C=D=0.6, E=F=0.2, G=0.4, etc. The above f1, f2, f3, f4, and f5 represent the normalization functions for normalizing the geometric attributes of the corresponding dimensions, respectively. They can be determined based on characteristics such as the domain of the corresponding information and the speed of change. This application does not restrict the category of each normalization function.

[0156] Step S703: Based on the graph matching module included in the face matching model, global nearest neighbor matching is performed on each face image to be processed and each tracking face image using the face matching degrees corresponding to the same face image to be processed, to determine a face matching result for each face image to be processed.

[0157] In actual applications, if the faces of tracked objects cross and then separate in historical frame images, since the faces of different tracked objects are close to or overlap, using traditional NN matching (Nearest Neighbor Matching), that is, calculating the distance between the face image to be processed and the face images in the previous historical frame image, and selecting the identity information of the face image closest to the distance as the object identity information of the unprocessed face image, it is easy to cause matching errors, thereby generating incorrect face tracking.

[0158] In order to improve the above problems, the present application proposes to construct a graph matching module based on global nearest neighbor matching GNN. For all the face images to be processed contained in the target frame image, based on the face matching degrees (or costs) corresponding to the tracked face images, all the tracked face images are conveniently combined and matched, and the combination with the smallest difference is determined, that is, the combination with the largest sum of face matching degrees (or the smallest total cost) between each face image to be processed and different tracked face images, as the optimal matching solution to obtain the face matching results of each face image to be processed, that is, whether each face image to be processed is successfully matched to the tracking face image (that is, whether it belongs to the face image of the tracked object).

[0159] Among them, the global nearest neighbor matching of the graph matching module is combined with the matching method of the graph neural network. It improves the robustness and accuracy of face matching by modeling the global relationship between features through the graph structure. The Mahalanobis distance MD corresponding to each face image to be processed and each stored tracked face image (the face feature vector obtained by calculating the seven-dimensional geometric properties of the face image based on the Mahalanobis distance module) is used as the feature of different nodes. The face matching degree (or Mahalanobis distance difference cost) between each face image to be processed and each tracked face image is used as the weight of the edge connecting the corresponding two nodes, thereby constructing a graph structure. Afterwards, the global nearest neighbor matching method can be used to search for combinations of each face image to be processed and each tracked face image. Each combination refers to a plurality of face image pairs formed by all the face images to be processed and different tracked face images. The face image matching degrees of each face image pair contained in each combination are summed up, and the combination corresponding to the maximum value of all the total matching degrees (or the minimum sum of the costs of the combined face image pairs, that is, the minimum total cost) is taken as the optimal solution for face matching, that is, to determine the best tracked face image that matches all the face images to be processed in the target frame image.

[0160] It can be seen that when the target frame image contains three unprocessed face images, such as unprocessed face image 1, unprocessed face image 2 and unprocessed face image 3, and the historical frame image has three tracking face images, such as tracking face image 1, tracking face image 2 and tracking face image 3, Figure 9 As shown, the Mahalanobis distance module can be used to obtain the facial matching degree or Mahalanobis distance difference cost between each unprocessed facial image and each tracked facial image. Optionally, to reduce the computational complexity of the subsequent image matching module, a facial image pair that clearly does not belong to the same object can be proposed. That is, a facial image to be processed and a tracked facial image can be a facial image pair whose corresponding facial matching degree is less than a matching threshold, or whose cost is greater than a corresponding distance threshold. In the subsequent optimal combination search, this facial image pair is no longer used for matching search, thereby reducing the number of combinations that need to be searched, improving facial matching efficiency, and reducing the probability of false matches.

[0161] Based on this, Figure 10 The flowchart of a face tracking method provided in the sixth embodiment of the present application is shown. The graph matching module in step S703 can implement but is not limited to the method described in steps 1001 to 1003 as follows:

[0162] Step S1001: In response to a face matching degree of any face being less than a matching threshold, determining that face matching of the corresponding face image pair has failed; the face image pair includes a face image to be processed and a tracking face image;

[0163] Step S1002: Remove the facial image pairs corresponding to failed face matching from all facial image pairs consisting of the facial images to be processed and the tracking facial images, to obtain facial image pairs to be matched;

[0164] Step S1003 , using the face matching degree corresponding to each to-be-matched face image pair, perform global nearest neighbor matching on the to-be-matched face image pairs corresponding to each to-be-processed face image, and determine the face matching result of each to-be-processed face image.

[0165] Among them, the above-mentioned matching threshold can be the minimum matching degree / similarity when the two face images in a face image pair belong to the same object, which can be determined based on actual experience or historical data or actual scene requirements, and this application does not limit the size of the matching threshold. It should be understood that if the Mahalanobis distance module outputs the Mahalanobis distance difference cost of each face image pair, as analyzed above, it can be determined that when the cost of the face image object is greater than the distance threshold, the corresponding two face images do not belong to the same object, that is, the face matching fails. After that, GNN global nearest neighbor matching can be performed based on the cost of the face image pair to be matched to accurately determine the face matching results of each face image to be processed. The matching implementation process can be determined in combination with the matching principle of GNN global nearest neighbor matching, which is not described in detail in this application.

[0166] Step S704 : In response to the face matching result being a face matching failure, determining the corresponding face image to be processed as an untracked face image belonging to an untracked object.

[0167] In the face matching process proposed in the embodiment of the present application, the seven-dimensional geometric attributes of the face image, namely (x, y, w, h, roll, pitch, yaw), are combined with the Mahalanobis distance to obtain the face matching degree of the face image pair. In this implementation process, the different accuracies of the geometric attributes of each dimension are taken into account, and reasonable information weights are configured to represent the accuracy of the corresponding geometric attributes, thereby improving the accuracy of the calculated face matching degree and reducing the false matching rate, thereby improving the tracking scene presentation experience.

[0168] Moreover, in multi-object scenarios, there are usually situations where the faces of different objects are close to each other and overlap. In order to ensure the correctness and reliability of face matching in this situation and to ensure tracking performance, this application adopts a GNN matching algorithm based on the seven-dimensional Mahalanobis distance, combined with face lifecycle management (such as the above-mentioned description of implementing dynamic updates of object status to ensure the continuity and stability of face presentation), to achieve reliable tracking and detection of object faces, avoid mismatching the pending face images of untracked objects as face images of tracked objects, and miss subsequent tracking detection based on face recognition; or mismatching the pending face images of tracked objects as face images of untracked objects, and repeating face recognition on the pending face images of known objects, which increases the computational burden and affects the processing speed.

[0169] Step S705: comparing the roll angle, pitch angle, and yaw angle in the facial orientation information corresponding to the untracked facial image with their corresponding angle thresholds;

[0170] Step S706 , in response to the roll angle, pitch angle, and yaw angle corresponding to the untracked facial image being less than their respective angle thresholds, performing facial recognition on the untracked facial image based on the facial recognition model to determine a facial recognition result of the corresponding untracked object.

[0171] In the embodiment of the present application, as analyzed above, from the various unprocessed facial images in the target frame image, the unprocessed facial images determined to belong to the tracked object have their object identity information known, and no further facial recognition is required. This avoids repeated recognition of the faces of the tracked objects, which increases the computational burden and reduces processing speed. The present application only performs facial tracking detection on unprocessed facial images determined to belong to untracked objects (referred to as untracked facial images, at which point the object identity information is unknown), thereby making the untracked objects become new tracked objects.

[0172] Among them, before performing face recognition on the untracked face image that will become the new tracked object, in order to avoid the situation where the face recognition result is wrong due to factors such as poor image quality or less facial features, in a possible implementation, the present application can first check whether the face orientation of each untracked face image meets the face recognition preset conditions (which can be determined based on the characteristics of the face recognition algorithm, and the present application does not limit the content of the conditions) before the face recognition model runs, such as the method described in steps S705 and S706 above, to determine whether the face roll angle, pitch angle and yaw angle in the untracked face image are all less than the corresponding angle threshold. If they are all less than the corresponding angle threshold, it means that the face recognition preset conditions are met, and the face recognition model can be triggered to execute the face recognition process for the untracked face image. Figure 8 As shown, the face recognition model can output the identity information of the corresponding untracked objects (such as participants in a video conference scenario), such as name, position, etc. Afterwards, based on the face recognition results, scene presentation can be realized, such as performing face tracking and detection on the speaker in the video conference and presenting the speaker's face image to improve the meeting experience.

[0173] On the contrary, that is, in response to at least one of the roll angle, pitch angle and yaw angle corresponding to the untracked facial image being greater than or equal to the corresponding angle threshold, it indicates that the preset conditions for facial recognition are not met, and the facial recognition model is no longer triggered to perform the facial recognition process for the untracked facial image, that is, the processing of the untracked facial image is terminated. In some scenarios, corresponding prompt information can also be output to notify the corresponding untracked object to adjust the facial orientation, or to handle the untracked object through management personnel, etc., depending on the situation.

[0174] For example, when the camera captures the target frame image, when the angle of the face of an untracked face image facing the camera (i.e., yaw) is less than its preset angle threshold (such as 20°, etc.), and the other two angles (i.e., roll and pitch) are both less than the corresponding angle thresholds (such as 30°, etc.), it means that the face orientation information of the untracked face image meets the preset conditions for face recognition, each face deflection angle is within a reasonable range, and the facial features presented by the untracked face image are sufficient to support face recognition, the face recognition process is triggered, which reduces the waste of computing resources caused by invalid face recognition under conditions of poor face deflection angles and improves the reliability of face recognition results.

[0175] It should be understood that for the method described above, facial images belonging to untracked objects, i.e., untracked facial images, are screened out from the unprocessed facial images in the target frame image. After obtaining the object identity information through facial recognition, the untracked object can be directly used as a new tracked object, and face tracking detection of the new tracked object can be continued in the face tracking processing of future frame images. Preferably, in order to ensure the continuity and stability of face presentation, the present application will also combine face lifecycle management, and the untracked object can be used as the object to be tracked. The face tracking processing method of the present application is executed on multiple future frame images (new target frame images), and the facial image of the untracked object is continuously detected, that is, the same object is continuously presented in multiple consecutive frame images, and then the object to be tracked is updated to a tracked object. Similarly, for each tracked object, if its facial image is not matched in multiple consecutive frame images, it is determined that the tracked object is lost and deleted from the storage space to ensure the continuity and stability of face presentation.

[0176] The above describes a face tracking method provided by an embodiment of the present application. The following describes a device for executing the above face tracking method.

[0177] Reference Figure 11 , is a structural diagram of a face tracking processing device provided in an embodiment of the present application, such as Figure 11 As shown, the face tracking processing device may include:

[0178] A detection module 111 is configured to perform face detection on a target frame image to be processed, and determine face position information and face orientation information of each face image to be processed contained in the target frame image;

[0179] a matching module 112 configured to perform face matching between the face image to be processed and the tracked face image of the tracked object based on the face position information and the face orientation information, and determine that the face image to be processed that belongs to the untracked object in the target frame image is an untracked face image;

[0180] The recognition module 113 is configured to perform face recognition on the untracked face image and determine a face recognition result corresponding to the untracked object.

[0181] In some embodiments, the matching module 112 may include:

[0182] a face matching unit configured to perform face matching on the face image to be processed and each tracked face image contained in the historical frame image based on the face matching model and using the face position information and the face orientation information, thereby obtaining a face matching result for the face image to be processed; the tracked face image is a face image of the tracked object;

[0183] The first determining unit is configured to, in response to the face matching result being a face matching failure, determine the corresponding face image to be processed as an untracked face image belonging to an untracked object.

[0184] In one possible implementation, the face matching unit may include:

[0185] a first matching unit, configured to perform face matching on the corresponding face image to be processed with each tracked face image contained in the same historical frame image based on a face matching model and using the face position information and the face orientation information, to obtain a pending matching result of the corresponding face image to be processed in the historical frame image;

[0186] The first obtaining unit is used to obtain a face matching result corresponding to the face image to be processed by using the pending matching results of the same face image to be processed in a plurality of consecutive historical frame images.

[0187] In a possible implementation, the face matching unit may also include:

[0188] a second matching unit, configured to obtain, based on a Mahalanobis distance module included in a face matching model, a face position information, a face orientation information, and weights of the information, a face matching degree between the face image to be processed and each tracked face image included in the historical frame image;

[0189] The third matching unit is used to perform global nearest neighbor matching on each of the face images to be processed and each of the tracked face images based on the graph matching module included in the face matching model and using the face matching degrees corresponding to the same face image to be processed, so as to determine the face matching results of each of the face images to be processed.

[0190] Optionally, the third matching unit may include:

[0191] a second determining unit, configured to determine that face matching of the corresponding face image pair fails in response to any of the face matching degrees being less than a matching threshold; the face image pair comprising one of the face image to be processed and one of the tracked face images;

[0192] a second obtaining unit, configured to remove the facial image pairs corresponding to the facial matching failures from all facial image pairs consisting of the facial images to be processed and the facial images to be tracked, to obtain facial image pairs to be matched;

[0193] The third determining unit is used to perform global nearest neighbor matching on the face image pairs to be matched corresponding to each face image to be processed by using the face matching degree corresponding to each face image to be matched, and determine the face matching result of each face image to be processed.

[0194] In some further embodiments, the matching module 112 may also include:

[0195] a first processing unit, configured to perform normalized weighted processing on the face position information and the face orientation information corresponding to the same face image to be processed, respectively, to obtain face features to be matched corresponding to the face image to be processed;

[0196] A fourth matching unit is configured to calculate a matching degree between each of the facial features to be matched and the tracked facial features of each tracked object to obtain a corresponding facial matching degree;

[0197] The fourth determining unit is configured to determine, based on the face matching degrees corresponding to the same face feature to be matched, that the face image to be processed, which belongs to an untracked object, is an untracked face image.

[0198] Based on the above analysis, in some embodiments, the face tracking processing device may further include:

[0199] a first updating module, configured to update the untracked object to a tracked object in response to a first number of frames of images containing facial images of the same untracked object;

[0200] The second updating module is configured to update the tracked object to an untracked object in response to a second number of image frames not including a face image of the same tracked object.

[0201] In some embodiments, the detection module 111 may include:

[0202] A face detection unit, configured to perform face detection on the target frame image to be processed based on a face detection model, and determine face position information of each face image to be processed contained in the target frame image;

[0203] a face orientation detection unit, configured to perform face orientation detection on each of the face images to be processed based on a face orientation detection model, and determine the roll angle, pitch angle, and yaw angle of the face orientation image acquisition device in the corresponding face image to be processed as face orientation information of the face image to be processed;

[0204] Based on this, the above-mentioned face tracking processing device may further include:

[0205] a comparison module, configured to compare the roll angle, the pitch angle, and the yaw angle corresponding to the untracked facial image with their corresponding angle thresholds;

[0206] a first triggering module configured to trigger the recognition module 113 to perform face recognition on the untracked face image in response to the roll angle, the pitch angle, and the yaw angle corresponding to the untracked face image being smaller than their respective angle thresholds, and to determine a face recognition result corresponding to the untracked object;

[0207] and an end processing module, configured to end processing of the untracked facial image in response to at least one of the roll angle, the pitch angle, and the yaw angle corresponding to the untracked facial image being greater than or equal to a corresponding angle threshold.

[0208] In some embodiments, the identification module 113 may include:

[0209] a feature extraction unit, configured to input the untracked face image into a face recognition model, so as to extract features from the untracked face image using a feature extraction module included in the face recognition model to obtain a corresponding face feature vector;

[0210] a similarity calculation unit, configured to perform distance calculation between the facial feature vector and a pre-stored reference facial feature vector using a similarity calculation module included in the facial recognition model to obtain a corresponding distance, and perform similarity calculation between the facial feature vector and the pre-stored reference facial feature vector to obtain a corresponding similarity;

[0211] A decision unit is configured to determine, through a decision module included in the face recognition model, based on the distance and similarity between the face feature vector and the same reference face feature vector, that the face feature vector and the reference face feature vector belong to the same untracked object, and determine the pre-stored identity information of the untracked object as a face recognition result.

[0212] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the face tracking processing methods provided in the embodiments of the present application.

[0213] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any face tracking processing method provided in the embodiment of the present application.

[0214] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0215] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general-purpose hardware, and of course can also be implemented by means of dedicated hardware including dedicated integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. In other words, the above embodiments can be implemented in whole or in part by means of software, hardware, firmware, or any combination thereof.

Claims

1. A face tracking method, comprising: Performing face detection on the target frame image to be processed, and determining face position information and face orientation information of each face image to be processed contained in the target frame image; performing face matching on the corresponding face image to be processed and the tracked face image of the tracked object based on the face position information and the face orientation information, and determining that the face image to be processed belonging to the untracked object in the target frame image is an untracked face image; Performing face recognition on the untracked face image to determine a face recognition result corresponding to the untracked object.

2. The method according to claim 1, wherein the step of performing face matching on the to-be-processed face image and the tracked face image based on the face position information and the face orientation information, and determining that the to-be-processed face image in the target frame image, which belongs to an untracked object, is an untracked face image, comprises: Based on the face matching model, using the face position information and the face orientation information, face matching is performed on the corresponding face image to be processed with each tracking face image contained in the historical frame image to obtain a face matching result for the corresponding face image to be processed; the tracking face image is a face image of the tracked object; In response to the face matching result being a face matching failure, the corresponding face image to be processed is determined as an untracked face image belonging to an untracked object.

3. The method according to claim 2, wherein the face matching model is based on the face position information and the face orientation information, and the face matching is performed on the corresponding face image to be processed with each tracked face image contained in the historical frame image to obtain the face matching result of the corresponding face image to be processed, comprising: Based on the face matching model, using the face position information and the face orientation information, face matching is performed on the corresponding face image to be processed with each tracked face image contained in the same historical frame image, to obtain a pending matching result of the corresponding face image to be processed in the historical frame image; The face matching result corresponding to the face image to be processed is obtained by utilizing the pending matching results of the same face image to be processed in a plurality of consecutive historical frame images.

4. The method according to claim 1, wherein the step of performing face matching on the to-be-processed face image and the tracked face image of the tracked object based on the face position information and the face orientation information, and determining that the to-be-processed face image belonging to the untracked object in the target frame image is the untracked face image, comprises: performing normalized weighted processing on the face position information and the face orientation information corresponding to the same face image to be processed, respectively, to obtain face features to be matched corresponding to the face image to be processed; Calculating the matching degree of each of the facial features to be matched and the tracked facial features of each tracked object to obtain a corresponding facial matching degree; Based on the face matching degrees corresponding to the same face feature to be matched, the face image to be processed belonging to the untracked object is determined to be an untracked face image.

5. The method according to claim 2, wherein the face matching model is based on the face position information and the face orientation information, and the face matching is performed on the corresponding face image to be processed with each tracked face image contained in the historical frame image to obtain the face matching result of the corresponding face image to be processed, comprising: Based on the Mahalanobis distance module included in the face matching model, the face position information, the face orientation information and the weight of each information are used to obtain the face matching degree between the corresponding face image to be processed and each tracked face image included in the historical frame image; Based on the graph matching module included in the face matching model, and using the face matching degrees corresponding to the same face image to be processed, global nearest neighbor matching is performed on each face image to be processed and each tracked face image to determine the face matching results of each face image to be processed.

6. The method according to claim 5, wherein the step of performing global nearest neighbor matching on each of the facial images to be processed and each of the tracking facial images using the facial matching degrees corresponding to the same facial image to be processed to determine a facial matching result for each of the facial images to be processed comprises: In response to any of the face matching degrees being less than a matching threshold, determining that face matching of the corresponding face image pair fails; The face image pair includes a face image to be processed and a tracked face image; Eliminating the facial image pairs corresponding to the facial matching failures from all facial image pairs formed by the facial images to be processed and the facial images to be tracked, to obtain facial image pairs to be matched; Using the face matching degree corresponding to each pair of face images to be matched, global nearest neighbor matching is performed on the pair of face images to be matched corresponding to each face image to be processed, to determine a face matching result for each face image to be processed.

7. The method according to any one of claims 1 to 6, further comprising: In response to the first number of frames of images containing facial images of the same untracked object, updating the untracked object to a tracked object; In response to the second number of image frames not including the facial image of the same tracked object, the tracked object is updated to an untracked object.

8. The method according to any one of claims 1 to 6, wherein the step of performing face detection on the target frame image to be processed and determining face position information and face orientation information of each face image to be processed contained in the target frame image comprises: Performing face detection on the target frame image to be processed based on the face detection model, and determining face position information of each face image to be processed contained in the target frame image; Performing face orientation detection on each of the face images to be processed based on the face orientation detection model, and determining the roll angle, pitch angle, and yaw angle of the face orientation image acquisition device in the corresponding face image to be processed as the face orientation information of the face image to be processed; The method further comprises: comparing the roll angle, the pitch angle, and the yaw angle corresponding to the untracked face image with their corresponding angle thresholds; In response to the roll angle, the pitch angle, and the yaw angle corresponding to the untracked facial image being less than their respective corresponding angle thresholds, performing step 15 of performing facial recognition on the untracked facial image to determine a facial recognition result corresponding to the untracked object; In response to at least one of the roll angle, the pitch angle, and the yaw angle corresponding to the untracked facial image being greater than or equal to a corresponding angle threshold, processing of the untracked facial image is terminated.

9. The method according to any one of claims 1 to 6, wherein performing face recognition on the untracked facial image to determine a face recognition result corresponding to the untracked object comprises: Inputting the untracked face image into a face recognition model, performing feature extraction on the untracked face image through a feature extraction module included in the face recognition model to obtain a corresponding face feature vector; Performing a distance calculation between the facial feature vector and a pre-stored reference facial feature vector by a similarity calculation module included in the facial recognition model to obtain a corresponding distance, and performing a similarity calculation between the facial feature vector and the pre-stored reference facial feature vector to obtain a corresponding similarity; The decision module included in the face recognition model determines, based on the distance and the similarity between the face feature vector and the same reference face feature vector, that the face feature vector and the reference face feature vector belong to the same untracked object, and determines the pre-stored identity information of the untracked object as the face recognition result.

10. An electronic device, comprising: at least one communication element, at least one memory, and at least one processor, wherein: The communication element is used to receive the target frame image to be processed; The memory is used to store a plurality of computer instructions; The processor is configured to execute the computer instructions to implement the following steps: Performing face detection on the target frame image to determine face position information and face orientation information of each face image to be processed contained in the target frame image; performing face matching on the corresponding face image to be processed and the tracked face image of the tracked object based on the face position information and the face orientation information, and determining that the face image to be processed belonging to the untracked object in the target frame image is an untracked face image; Performing face recognition on the untracked face image to determine a face recognition result corresponding to the untracked object.

Images