Face capture method, system and computer readable storage medium

Abstract

Provided are a face capture method, system, and computer-readable storage medium. The method includes: capturing faces in a monitoring area to obtain an initial frame; performing face detection in the initial frame to obtain multiple faces; establishing a tracking chain list; in a detection frame after the initial frame, updating the first part, the second part, and the third part of the tracking information of each face according to the detection result; in a tracking frame after the initial frame, updating the second part and the third part of the tracking information of the first face tracked using a first tracker, and updating the second part of the tracking information of the second face tracked using a second tracker. In the initial frame, a tracking chain list is established for the multiple faces captured to record the tracking information of the multiple faces; and in the tracking process, the tracking chain list is dynamically updated according to the actual situation, thereby achieving continuous and efficient tracking of dynamically changing faces.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and more specifically, to a face capture method, system and computer-readable storage medium. Background Art

[0002] With the development of face tracking and face recognition technology, the application of face capture system is becoming more and more widespread.

[0003] The face capture system can use the camera to continuously capture and track faces in the monitoring area. In crowded scenes, such as train stations, a captured image often contains multiple faces. As the capture process proceeds, some of the multiple faces will leave the monitoring area, and some new faces will appear in the monitoring area. In other words, during the capture process, the number and location of captured faces often change dynamically. How to continuously and efficiently track dynamically changing faces is an urgent problem to be solved. Summary of the invention

[0004] The present application provides a face capture method, system and computer-readable storage medium to continuously and efficiently track dynamically changing faces.

[0005] In a first aspect, a face capture method is provided, comprising: capturing faces in a monitoring area to obtain an initial frame; performing face detection in the initial frame to obtain multiple faces; establishing a tracking link list, the tracking link list including multiple tracking information corresponding to the multiple faces, the tracking information of each of the multiple faces including a first part, a second part and a third part, the first part is used to indicate the tracking state of the face, the second part is used to indicate the position of the boundary box of the face in the image frame, and the third part is used to indicate the information of the highest quality image of the face, wherein the tracking state of the multiple faces is the first state; in a detection frame after the initial frame, the first part, the second part and the third part of the tracking information of each face are updated according to the detection result; in a tracking frame after the initial frame, the second part and the third part of the tracking information of the first face tracked by using the first tracker are updated, and the second part of the tracking information of the second face tracked by using the second tracker is updated.

[0006] In one embodiment, the third part of the tracking information includes the highest score of the image of the face, the spatial position of the image of the face with the highest score in the image frame, and the temporal position of the image of the face with the highest score in the captured image sequence. The updating of the third part of the tracking information of the first face includes: determining a first score of the bounding box of the first face in a current frame, wherein the current frame is a detection frame or a tracking frame for tracking the first face using a first tracker; comparing the first score with the highest score of the image of the first face recorded in the third part of the tracking information of the first face; if the first score is higher than the highest score, updating the highest score in the third part of the tracking information of the first face to the first score, updating the spatial position in the third part of the tracking information of the first face to the position of the bounding box of the first face in the current frame, and updating the temporal position in the third part of the tracking information of the first face to the frame number of the current frame.

[0007] In one embodiment, in a detection frame after the initial frame, it is determined whether a third face satisfies a tracking end condition, wherein the third face is any one of the multiple faces; if the tracking end condition is met, a capture result of the third face is output, and the tracking information of the third face is deleted from the tracking linked list, wherein the capture result is the third part of the tracking information of the third face.

[0008] In one embodiment, the tracking end condition includes: the tracking state of the third face is the second state, and the time duration of the third face in the second state reaches a preset time duration, and the second state is used to indicate that the second tracker is used to track the face; and / or, the tracking state of the third face is the third state, and the detection frame does not detect the third face, and the third state is used to indicate that the face is at the edge of the monitoring area.

[0009] In one embodiment, updating the second part of the tracking information includes: in response to obtaining the position of the bounding box of the first face in the current frame, updating the second part of the tracking information of the first face, the current frame being a detection frame or a tracking frame.

[0010] In one embodiment, in a detection frame, determine whether a new face is detected; if a new face exists, add tracking information of the new face to a tracking list, wherein the first part of the added tracking information of the new face is updated to a fourth state; determine the image quality of the new face; when the image quality of the new face is less than a preset threshold, delete the tracking information of the new face from the tracking list; when the image quality of the new face is greater than or equal to the preset threshold, convert the tracking state of the new face from the fourth state to the first state.

[0011] In one embodiment, updating the first part of the tracking information includes: in response to detecting in the detection frame that the first face is not located at the edge of the detection frame, determining that the first part of the tracking information of the first face is in a first state; in response to not detecting the first face in the detection frame, updating the first part of the tracking information of the first face to a second state; in response to detecting in the detection frame that the first face is located at the edge of the detection frame and moving toward the edge, updating the first part of the tracking information of the first face to a third state.

[0012] In one embodiment, the first tracker is a tracker for human faces, and the second tracker is a tracker for non-specific types of objects.

[0013] In one embodiment, the first tracker is a deep learning model based tracker, and the second tracker is a kernel correlation filter tracker.

[0014] In a second aspect, a face capture system is provided, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the first aspect or any embodiment of the first aspect.

[0015] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect or any embodiment of the first aspect are performed.

[0016] According to a fourth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the steps of the method described in the first aspect or any embodiment of the first aspect.

[0017] In the initial frame, a tracking list is established for the captured multiple faces to record the tracking information of the multiple faces; then, during the tracking process, the tracking list is dynamically updated according to the actual situation, thereby achieving continuous and efficient tracking of dynamically changing faces. In addition, during the process of tracking a certain face, different tracking states can be assigned to the face, so that different trackers can be used to track it, which improves the flexibility of tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic flow chart of the face capture method provided in an embodiment of the present application.

[0019] Figure 2 This is an example diagram of the initial frame provided in an embodiment of the present application.

[0020] Figure 3 It is a schematic flow chart of the tracking information updating method provided in an embodiment of the present application.

[0021] Figure 4 It is a schematic diagram of a detection frame provided in an embodiment of the present application.

[0022] Figure 5 It is a schematic diagram of another detection frame provided in an embodiment of the present application.

[0023] Figure 6 It is a schematic flowchart of the updating method of the tracking linked list provided in the embodiment of the present application.

[0024] Figure 7 It is a schematic structural diagram of the face capture system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The face capture method provided in the embodiment of the present application can be executed by a face capture system. The face capture system can be implemented in many ways, which are not specifically limited in the embodiment of the present application.

[0026] As an example, the face capture system may include a camera at the front end and a server at the back end. The camera may be installed in a monitoring area, such as a train station or a store. The camera may capture or sample the monitored area at a preset time interval (the time interval may be set according to actual needs, such as a few milliseconds or a few seconds).

[0027] As another example, the face capture system can be located in a mobile terminal, such as a vehicle-mounted mobile terminal, or a handheld mobile terminal (such as a smart phone). For example, a camera on the mobile terminal can be used to capture a certain area of ​​interest (hereinafter referred to as a monitoring area), and the processor of the mobile terminal can use a tracking algorithm to track the captured face and dynamically maintain the information obtained during the tracking process.

[0028] Capturing faces in the monitoring area will form an image sequence. The image sequence may include multiple frames of images captured at different times, and the initial frame may be the first frame of the multiple frames, and the initial frame will be set as the detection frame.

[0029] In the tracking frame, a tracker (or tracking algorithm) can be used to track the face, and in the detection frame, a detector (or detection algorithm) can be used to detect the face. The detector can be a detector based on a deep learning model or other types of detectors.

[0030] The interval between the tracking frame and the detection frame can be set according to actual needs, and the present application does not limit this. It is understandable that, since the algorithm for performing face detection is more complex and time-consuming than the algorithm for face tracking, the number of tracking frames can be set to be much larger than the number of detection frames, for example, the number of tracking frames is several times (5-20, etc.) of the detection frames, for example, 10 times. On the other hand, in order to ensure the accuracy of tracking by the tracking frame, one or several detection frames can be set after each continuous number of tracking frames. Optionally, several continuous tracking frames (which can be called tracking frame sequences) can be set after each detection frame, that is, tracking frames are set between two adjacent detection frames, and the interval can be uniform. For example, a detection frame can be set every 10 image frames, and the remaining image frames can be set to tracking frames.

[0031] As an example, the initial frame may be the 0th frame, and the frames that are integer multiples of 10 thereafter are detection frames, such as the 10th frame, the 20th frame, ... The remaining frames are tracking frames, such as the 1st to 9th frames, the 11th to 19th frames, ...

[0032] In order to maintain facial images whose quantity and quality change dynamically, the embodiment of the present application proposes a tracking linked list that can be dynamically updated.

[0033] The tracking linked list may include multiple pieces of tracking information corresponding to multiple faces currently being tracked. The tracking information of each face in the multiple faces may include a first part, a second part, and a third part. The first part is used to indicate the tracking state of the face, the second part is used to indicate the position of the bounding box of the face in the image frame, and the third part is used to indicate the information of the highest quality image of the face.

[0034] It should be noted that the present application does not limit the maximum number of tracking information in the tracking list, and can be flexibly selected according to the hardware conditions of the face capture system, the response speed requirements, etc. For example, the maximum number P of tracking information in the tracking list can be 30 or 60. In this way, the embodiment of the present application can track up to P faces at the same time, which can meet the needs of most application scenarios and meet the real-time requirements.

[0035] The following takes the tracking information of any one of the multiple faces (for the convenience of description, the present application will record any one of the multiple faces as FACE) as an example to illustrate the composition of the tracking information in detail.

[0036] The structure of the tracking information is similar and will not be described in detail later.

[0037] The first part of the tracking information of FACE can be used to indicate the tracking state of FACE. The tracking state may include at least one of a first state, a second state, a third state, and a fourth state. The first state can be used to indicate that the first tracker is used to track the face, the second state can be used to indicate that the second tracker is used to track the face, the third state can be used to indicate that the face is at the edge of the monitoring area, and the fourth state can be used to indicate that the detected face is a new face other than the multiple faces. The first tracker and the second tracker can be different tracking algorithms, and different tracking algorithms usually have different tracking capabilities or tracking effects.

[0038] As an example, the first state may be TRACK, and the second state may be PENDING.

[0039] For example, the first tracker may be a tracker for human faces, and the second tracker may be a tracker for non-specific types of targets. The second tracker may be a general tracker, which may track human faces as well as other types of targets (such as human bodies or vehicles).

[0040] For another example, the first tracker can be a tracker based on a deep learning model. This type of tracker can easily track multiple targets in parallel, but has high requirements for the characteristics of the tracked targets. The second tracker can be a kernel correlation filter tracker. This tracker requires a lot of computing resources to achieve parallel tracking, but the advantage of this tracker is that it has low requirements for the tracked targets and is not easy to lose the target.

[0041] The tracking state of FACE can be set according to the actual situation. For example, when the image quality of FACE is high (such as higher than a certain quality threshold), the tracking state of FACE can be set to the first state to track it using the first tracker; when the image quality of FACE is low (such as lower than the above quality threshold), the tracking state of FACE can be set to the second state, so that it can be tracked using a second tracker different from the first tracker. In other words, the actual situation of different faces is different, and different trackers have different tracking effects on faces in different situations. Selecting a FACE tracker according to the actual situation can increase the flexibility of tracking and improve the tracking effect.

[0042] For another example, when FACE is a new face, that is, a face that has just entered the monitoring area, its tracking state can be set to the first state, so that it can be tracked using the first tracker. If FACE is not detected in a subsequent detection frame, the tracking state of FACE can be converted from the first state to the second state. In other words, when a face is a new face, its tracking state can be uniformly set to the first state, and the first tracker can be used to track it; when the detection of a face fails, the second tracker can be used to track it instead. It can be seen that when the first tracker loses track of the face, this example does not immediately output the tracking result, but uses another tracker to continue tracking it, thereby avoiding the problem of a high rate of repeated output of the same face due to missed detection to a certain extent.

[0043] It should be noted that the tracker will track the face in the tracking frame, so the above-mentioned "the first state is used to indicate the use of the first tracker for tracking" means: if the tracking state of a face is the first state, the first tracker is used to track it in the tracking frame. "The second state is used to indicate the use of the second tracker for tracking" is the same as "the first state is used to indicate the use of the first tracker for tracking", which will not be repeated here.

[0044] The second part of the tracking information of FACE can be used to indicate the position of the bounding box of FACE in the image frame. Exemplarily, the tracking information of FACE can be used to record the coordinates (or pixel positions) of the bounding box of FACE in the image frame. As the tracking process proceeds and the number of captured images increases, the information in the second part of the tracking information of FACE will also be dynamically updated in real time to record the latest position of the position of FACE in the monitoring area. That is, the second part can include the position of the bounding box of the face detected or tracked in real time in the current frame.

[0045] In one embodiment, the second part may include four parameters, such as (x1, y1, x2, y2) or (x1, y1, w, h), where (x1, y1) may represent the coordinates of the lower left corner or upper left corner of the bounding box, (x2, y2) may represent the coordinates of the upper right corner or lower right corner relative to (x1, y1), and (w, h) may represent the width and height of the bounding box, respectively.

[0046] The third part of the tracking information of FACE may be used to indicate information of the image with the highest quality of FACE. Exemplarily, the third part may include at least one of the following information: the highest score of the image of FACE, the spatial position of the image with the highest score of FACE in the image frame, and the temporal position of the image with the highest score of FACE in the captured image sequence.

[0047] It is understandable that when the tracking list is created in the initial frame, the third part may be information of the image in the initial frame. For example, it may include: the quality score of the image in the area defined by the bounding box (x1, y1, x2, y2), (x1, y1, x2, y2), and "0". Among them, "0" represents the frame number of the initial frame.

[0048] As the tracking process proceeds and the number of captured images increases, the third part of the FACE tracking information may be replaced by subsequent FACE images with higher quality.

[0049] When the tracking of the face is finished (for example, the face leaves the monitoring area), the third part of the tracking information of the face can be output as the capture result of the face. For example, the highest quality image of the face can be displayed on the display interface of the face capture system.

[0050] In combination with the above description, the following table 1 shows an example of a tracking chain table:

[0051] Table 1

[0052]

[0053] Combine the following Figure 1 The face capture process shown in the figure is explained in more detail with examples.

[0054] Figure 1 It is a schematic flow chart of the face capture method provided in an embodiment of the present application. Figure 1 The method can be executed by the face capture system mentioned above. Figure 1 The method may include steps S110 to S150.

[0055] In step S110, a human face in the monitoring area is captured to obtain an initial frame.

[0056] Figure 2 An example of an initial frame.

[0057] In step S120, face detection is performed on the initial frame to obtain multiple faces.

[0058] from Figure 2 It can be seen that at least three faces, face0, face1 and face2, are detected in the initial frame.

[0059] In step S130, a tracking chain list is established.

[0060] still Figure 2 For example, a tracking list can be created, and tracking information corresponding to face0, face1, and face2 can be added to the tracking list.

[0061] For example, the tracking states of face0, face1, and face2 may all be set to the first state, and recorded in the first part of the tracking information corresponding to face0, face1, and face2 respectively.

[0062] In addition, the coordinates of the bounding boxes of face0, face1, and face2 in the image frame can be recorded in the second part of the tracking information corresponding to face0, face1, and face2. It should be understood that as face0, face1, and face2 move in the monitoring area, the positions of the bounding boxes of face0, face1, and face2 in the image frame change dynamically. Therefore, the second part of the tracking information of face0, face1, and face2 is continuously updated as the number of captured image frames increases.

[0063] Then, due to Figure 2 is the initial frame, then the information of the highest quality image of face0, face1, and face2 is Figure 2 Of course, as the number of captured image frames increases, the highest quality images of face0, face1, and face2 may be replaced by the images that appear later. Figure 3 , the replacement or update process of the third part of the tracking information is explained in detail with examples, which will not be described in detail here.

[0064] In step S140, in a detection frame after the initial frame, the first part, the second part and the third part of the tracking information of each face are updated according to the detection result.

[0065] It should be noted that the detection frame after the initial frame (for the convenience of description, the detection frame after the initial frame is referred to as the first detection frame in this application) can be the next detection frame of the initial frame, or the nth detection frame (n is greater than 1) of the initial frame. When the first detection frame is the nth detection frame after the initial frame, the detection results of the face in other detection frames between the initial frame and the first detection frame can be a successful detection or a failed detection, or the detection result can also be the position of the detected face in the detection frame, the image quality, etc.

[0066] In step S150, in a tracking frame after the initial frame, the second part and the third part of the tracking information of the first face tracked by the first tracker are updated, and the second part of the tracking information of the second face tracked by the second tracker is updated.

[0067] It is understandable that the three parts of the tracking information may not be updated in all image frames. The three parts of the tracking information are described below respectively.

[0068] The first part is used to indicate the tracking state of the face, and the tracking state may include at least one of the first state, the second state, the third state and the fourth state. The first state can be used to indicate the use of the first tracker for face tracking, the second state can be used to indicate the use of the second tracker for face tracking, the third state can be used to indicate that the face is at the edge of the monitoring area, and the fourth state can be used to indicate that the detected face is a new face other than the multiple faces. For example, the first state can be represented by "TRACK", the second state can be represented by "PENDING", the third state can be represented by "OUTSIDE" or "PENDING_OUTSIDE", and the fourth state can be represented by "NEWDETECT". How to update the tracking state will be described in detail in the following content.

[0069] The update of the first part can be performed in the detection frame, and the tracking state of the target can be updated according to the detection situation of the target. The detection situation may include, for example, whether a face is detected, the position of the detected face, etc. It should be noted that in the tracking frame, the first part remains unchanged, that is, the state will not change with the tracking result of the tracking frame.

[0070] The second part is used to indicate the position of the bounding box of the face in the image frame. Since the position of the face is in dynamic change, the second part can be updated in each frame during the face tracking process. In response to obtaining the position of the bounding box of the face in the current frame, the second part of the face tracking information is updated, wherein the current frame can be a detection frame or a tracking frame. When the current frame is a detection frame, the position of the bounding box can be the position of the detection box of the detected face. When the current frame is a tracking frame, the position of the bounding box can be the position of the tracking box of the tracked face in the tracking frame.

[0071] The third part is used to indicate the information of the highest quality image of the human face, and can be updated in the image frame with higher quality of the human face image.

[0072] For example, in the detection frame, if a face can be detected, the quality can be calculated in the detection frame, and the third part of the tracking information of the detected face can be updated. The face detection method can be implemented using an existing detection algorithm, for example, it can be implemented based on the Hungarian algorithm.

[0073] Alternatively, when the first face is tracked using the first tracker in a certain tracking frame, the quality of the first face can be similarly calculated in the tracking frame in which the first face is tracked using the first tracker, and the third part of the tracking information of the first face can be updated. On the contrary, for the second face tracked using the second tracker, since the second tracker is a tracker for a non-specific target, although the tracking result can be obtained using the second tracker, the second face may not actually exist in the tracking frame, or even if the second face exists, the position accuracy indicated by the tracking result using the second tracker is inaccurate, so the image quality of the second face is not calculated in the tracking frame in which the second face is tracked using the second tracker, that is, the third part of the tracking information of the second face is not updated.

[0074] It should be noted that, when tracking multiple faces in the same tracking frame, the quality of the multiple faces may not be the same, and there may be a situation where multiple faces are tracked using different trackers. Therefore, for the same tracking frame, the quality of some faces may be calculated or the third part of the tracking information may be updated, and the quality of some faces may not be calculated or the third part of the tracking information may not be updated. For example, when tracking a first face and a second face at the same time in a tracking frame, the quality of the first face tracked by the first tracker may be calculated, and the third part of the tracking information of the first face may be updated; the quality of the second face tracked by the second tracker may not be calculated, and the third part of the tracking information of the second face may not be updated.

[0075] Optionally, a deep learning model or other model (or algorithm) can be used to calculate the quality of the face image. Since the calculation of face quality requires computing resources, this application does not calculate the face quality during the face tracking process using the second tracker, and does not update the third part, which can effectively improve the computing efficiency of the face capture system, improve the response time of the capture, and avoid the situation of erroneous capture output.

[0076] It can be seen that the embodiment of the present application establishes a tracking list for the multiple faces captured in the initial frame to record the tracking information of the multiple faces; and during the tracking process, the tracking list is dynamically updated according to the actual situation, thereby achieving continuous and efficient tracking of dynamically changing faces. In addition, in the process of tracking a certain face, different tracking states can be assigned to the face, that is, different trackers are used to track it, thereby improving the flexibility of tracking.

[0077] The third part of the face tracking information may be information of the image with the highest quality, including at least one of the following information: the highest score of the first face image, the spatial position of the first face image with the highest score in the image frame, and the time position of the first face image with the highest score in the captured image sequence. Figure 3, taking the first face as an example, the updating process of the third part of the tracking information is illustrated in detail.

[0078] Figure 3 The method includes steps S310 to S350.

[0079] In step S310, a first score of an image of a first human face in a current frame is determined, where the current frame is a detection frame or a tracking frame in which the first human face is tracked using a first tracker.

[0080] The score of the image can be calculated using a quality judgment model, which can be a neural network model.

[0081] In step S320, the first score is compared with the highest score of the first face image recorded in the tracking list.

[0082] In step S330, if the first score is higher than the highest score, the highest score in the tracking list is updated to the first score.

[0083] In step S340, if the first score is higher than the highest score, the spatial position of the highest quality image of the first face in the tracking list in the image is updated to the position of the image of the first face in the current frame.

[0084] In step S350, if the first score is higher than the highest score, the time position of the highest quality image of the first face in the tracking list in the captured image sequence is updated to the frame number of the current frame in the captured image sequence. For example, assuming that the current frame is the 29th frame in the captured image sequence, the time position of the highest quality image of the first face in the tracking list in the captured image sequence can be updated to 29.

[0085] It can be understood that after step S320, S330, S340 and S350 are executed simultaneously or partially simultaneously, that is, the third part of the tracking information is updated.

[0086] The face capture method involved in this application is based on face tracking, and the output capture result is based on tracking information. In the detection frame after the initial frame, it is determined whether any one of the multiple faces (for the convenience of description, this application refers to the third face) meets the tracking end condition; if the tracking end condition is met, the capture result of the third face is output, and the tracking information of the third face is deleted from the tracking linked list, wherein the capture result is the third part of the tracking information of the third face.

[0087] When the face tracking is finished, an embodiment of the present application outputs the third part of the face tracking information to obtain the face capture result, that is, the image information with the highest quality in the face tracking process is used as the face capture result. It can be seen that the face capture result obtained in this way is the image of the face capture result with the highest quality in the face tracking process, that is, the face image with a better angle and higher clarity during the dynamic change of the face. Through the method provided by the present application, users can be provided with face capture images with higher image quality, which effectively improves the user experience of face capture.

[0088] If a face leaves the monitoring area, the face tracking can be terminated. Alternatively, if a face is not detected and continued tracking fails to obtain a good quality snapshot result, the face tracking can be terminated. For example: the tracking state of the third face is the second state, and the duration of the third face in the second state reaches a preset duration; and / or the tracking state of the third face is the third state, and the third face is not detected in the detection frame.

[0089] Optionally, the duration that the third face is in the second state (also referred to as the duration that the third face is tracked by the second tracker) can be calculated by the number of consecutive frames. The preset duration can be a set number of consecutive frames. For example, the preset duration can be 50 frames. When the third face is not detected in 50 consecutive image frames, that is, the second tracker is used for tracking during this period, the tracking task of the third face is terminated, the capture result of the third face is output, and the tracking information of the third face is deleted from the tracking list.

[0090] Optionally, the tracking duration of the second tracker can be calculated by the number of consecutive detection frames in which the third face is not detected. Exemplarily, the tracking duration can also be referred to as the tracking age of the second tracker. For example, the preset duration can be 4. When the third face is not detected in 4 consecutive detection frames during the use of the second tracker, it means that the tracking duration reaches the preset duration, the tracking of the third face is stopped, and the capture result of the third face is output.

[0091] It can be seen that in some embodiments, if a face is not detected, it is not necessary to immediately delete the tracking information of the face from the tracking list, but the first face can continue to be tracked. If the first face is not detected for a continuous period of time, the corresponding tracking information is deleted from the tracking list. Continuing to track after no face is detected can avoid repeated output of face capture results caused by missed detection. Ending the tracking after no face is detected can effectively reduce the waste of resources caused by continuous tracking of some faces, which generally have problems such as poor image quality and poor shooting angles.

[0092] As an implementation method, when the tracking state of the third face is the third state, that is, the third face is at the edge of the image frame, in the next image frame, the third face may leave the monitoring area, but it may also wander at the edge, or return to the inside of the image frame (the image frame can detect the face and the face is not at the edge of the image frame). In order to avoid erroneous termination of tracking in the latter two cases, for the third face in the third state, if the third face is not detected in the next detection frame, the application can determine that the third face has left the scene, then terminate the tracking of the third face and output the snapshot result. In this way, the accuracy of judging whether the face has left the scene where the image frame is located can be improved, and the erroneous premature termination of the tracking of some faces that have not left the monitoring area can also be avoided. The following is a detailed description of the updating method of the first part of the tracking information.

[0093] In response to detecting in the detection frame that the first face is not located at the edge of the detection frame, determining that the first part of the tracking information of the first face is in the first state; in response to not detecting the first face in the detection frame, updating the first part of the tracking information of the first face to the second state; in response to detecting in the detection frame that the first face is located at the edge of the detection frame and the moving direction of the first face is toward the edge, updating the first part of the tracking information of the first face to the third state; in response to detecting a new face other than multiple faces in the detection frame, adding the tracking information of the new face to the tracking linked list, and determining the first part of the tracking information of the new face to be in the fourth state.

[0094] For example, for a first face in a first state, if the first face is detected in a detection frame and the detected first face is not located at an edge, the first part of its tracking information may be kept in the first state unchanged.

[0095] Combine the following Figure 4 and Figure 5 Detailed description. Figure 4 and Figure 5 It is any detection frame in the video sequence, for example, it can be a detection frame after the initial frame, or it can also be the initial frame.

[0096] exist Figure 4 In the above figure, two faces, face1 and face2, are detected. Figure 4 Use box 1 and box 2 to mark them. And face1 and face2 are not located in Figure 4 As shown, the edge of the detection frame, therefore, the first part of the tracking state of face1 and face2 can be updated to the first state, for example, marked as TRACK.

[0097] Continue with Figure 4For example, face face0 (marked with box 0 for ease of explanation) is not detected, so the first part of the tracking information of face0 can be updated to the second state.

[0098] Figure 5 is another detection frame after the initial detection frame. In this detection frame, face0 is detected (marked with box 0) and face0 is located at Figure 5 If the left edge of face0 is present, the first part of the tracking information of face0 can be updated to a third state, for example, marked as OUTSIDE.

[0099] It should be noted that the update of the first part here can maintain at least one state at the same time. Figure 5 For example, if face0 is in Figure 5 The previous status was TRACK. Figure 5 The detection frame shown in the figure, the first part of face0 can be updated to TRACK+OUTSIDE; if face0 is in Figure 5 The previous status was PENDING. Figure 5 In the detection frame shown, the first part of face0 can be updated to PENDING+OUTSIDE.

[0100] For a detected face, one possible implementation is to directly add the tracking information of the new face to the tracking list and update the first part to the first state; or, the present application proposes that the face may be first quality tested, and after the quality test is qualified, the first part may be updated to the first state, and tracked in subsequent image frames, which can effectively avoid false alarms and reduce resource waste. As an implementation, in a detection frame, when a new face is detected, the tracking information of the detected face is created, and before the first part of the tracking information of the new face is set to the first state, the first part of the tracking information may be updated to a fourth state, for example, NEWDETECT.

[0101] Combine the following Figure 6 Taking the third face as an example, the processing of updating tracking information of the new face is illustrated, which may include steps S131 to S138.

[0102] In step S131, it is determined whether a new face is detected.

[0103] In step S132, if there is a new face, the tracking information of the new face is added to the tracking chain list, and the first part of the tracking information of the new face is determined to be in the fourth state.

[0104] The fourth state is a tracking state set for a new face. The fourth state can be used to indicate that the corresponding face is a new face. The fourth state can be NEWDETECT, for example.

[0105] In step S134, the image quality of the new face is determined.

[0106] In step S136, when the image quality of the new face is less than the preset threshold, the new face is deleted from the tracking list.

[0107] Tracking information;

[0108] In step S138, when the image quality of the new face is greater than or equal to a preset threshold, the tracking state of the new face is converted from the fourth state to the first state.

[0109] by Figure 2 For example ( Figure 2 The initial frame shown is also the detection frame), assuming that the new face is Figure 2 (face3 is not actually a face, but a handbag. The color of the handbag is similar to the skin color of the face, so it is mistakenly detected as a face.) When face3 is detected, the tracking information of face3 can be generated in the tracking list first, and the tracking state of face3 is set to NEWDETECT. Then, the image quality of face3 can be tested. After the test, it is found that the image quality of face3 is lower than the test qualification standard. Therefore, the tracking information of face3 is deleted from the tracking list, which can reduce the waste of computing resources.

[0110] Similarly, it can be understood that in an alternative implementation, when multiple faces are detected in the initial frame to establish a tracking list in the above S130, the tracking state of the tracking information of each face in the tracking list is initially set to the fourth state, i.e., NEWDETECT. Subsequently, in response to the face quality scores of the multiple faces being greater than or equal to a preset threshold, the tracking state of the tracking information of each face in the multiple faces can be further updated from the fourth state to the first state, i.e., TRACK.

[0111] This application also provides a face capture system, such as Figure 7 As shown, it includes a memory, a processor and a computer program stored in the memory. And when the processor executes the computer program, the above combination can be achieved Figure 1 The steps of the method.

[0112] The memory may be a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM), etc.

[0113] The processor may be a general-purpose CPU, a microprocessor, an application specific integrated circuit (ASIC) or one or more integrated circuits to execute relevant programs to implement the method of the embodiment of the present application.

[0114] The processor may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the method of the present application may be completed by an integrated logic circuit of hardware in the processor or by instructions in the form of software. The above-mentioned processor may also be a general-purpose processor, a digital signal processor (digital signal processing, DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the functions required to be performed by the units included in the system of the embodiment of the present application in combination with its hardware, or executes the method of the embodiment of the method of the present application.

[0115] Optionally, the system may further include a communication interface and a bus. The communication interface uses a transceiver such as, but not limited to, a transceiver to achieve communication with other devices or networks. For example, multiple images can be acquired from an image acquisition device through a communication interface, image processing results can be transmitted to other external devices through a communication interface, and so on. The bus may include a path for transmitting information between various components of the device (e.g., memory, processor, communication interface).

[0116] Understandable, Figure 7 The face capture system in the embodiment may be a computer system, a computer device, etc. In some embodiments, the face capture system may be a mobile terminal, such as a handheld mobile terminal, for example, a mobile phone.

[0117] In addition, an embodiment of the present invention further provides a computer storage medium on which a computer program is stored. When the computer program is executed by a computer or a processor, the aforementioned combination Figure 1 The steps of the method described. For example, the computer storage medium is a computer readable storage medium.

[0118] In one embodiment, the computer program instructions, when executed by a computer or a processor, cause the computer or the processor to perform the following steps: capturing a face in a monitoring area to obtain an initial frame; performing face detection in the initial frame to obtain multiple faces; establishing a tracking chain list, the tracking chain list including multiple tracking information corresponding to the multiple faces, the tracking information of each face in the multiple faces including a first part, a second part and a third part, the first part is used to indicate the tracking state of the face, the second part is used to indicate the position of the bounding box of the face in the image frame, and the third part is used to indicate the information of the highest quality image of the face, wherein the tracking state of the multiple faces is all the first state; in a detection frame after the initial frame, updating the first part, the second part and the third part of the tracking information of each face according to the detection result; in a tracking frame after the initial frame, updating the second part and the third part of the tracking information of the first face tracked using the first tracker, and updating the second part of the tracking information of the second face tracked using the second tracker.

[0119] Computer storage media may include, for example, a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. A computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0120] In addition, an embodiment of the present invention further provides a computer program product, which includes a computer program or an instruction. When the computer program or the instruction is executed by a computer or a processor, the above-mentioned combination is executed. Figure 1 The steps of the method are described.

[0121] It can be seen that the present application can establish a tracking chain table to maintain the tracking information of multiple faces being tracked. On the one hand, the tracking chain table can record the tracking status and the bounding box position of the face in real time. On the other hand, the tracking chain table also records the information of the highest quality image in the current tracking sequence in real time, which is convenient for the output of the captured image. Through the tracking chain table, the efficiency and accuracy of capturing faces in the video can be improved.

[0122] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present invention thereto. Various changes and modifications may be made therein by one of ordinary skill in the art without departing from the scope and spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as required by the appended claims.

[0123] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

Claims

1. A face capture method, characterized in that: include: Capture the face of a person in the monitoring area to obtain an initial frame; Performing face detection on the initial frame to obtain multiple faces; Establishing a tracking linked list, the tracking linked list including a plurality of tracking information corresponding to the plurality of faces one by one, the tracking information of each face in the plurality of faces including a first part, a second part and a third part, the first part being used to indicate a tracking state of the face, the second part being used to indicate a position of a bounding box of the face in an image frame, and the third part being used to indicate information of an image of the highest quality of the face, wherein the tracking states of the plurality of faces are all in the first state; In a detection frame after the initial frame, the first part, the second part and the third part of the tracking information of each face are updated according to the detection result; In a tracking frame after the initial frame, updating the second part and the third part of the tracking information of the first face tracked by the first tracker, and updating the second part of the tracking information of the second face tracked by the second tracker; The third part of the tracking information includes the highest score of the face image, the spatial position of the face image with the highest score in the image frame, and the temporal position of the face image with the highest score in the captured image sequence; The updating of the third part of the tracking information of the first face includes: determining a first score for a bounding box of the first face in a current frame, wherein the current frame is a detection frame or a tracking frame in which the first face is tracked using a first tracker; comparing the first score with a highest score of the images of the first face recorded in the third part of the tracking information of the first face; If the first score is higher than the highest score, updating the highest score in the third part of the tracking information of the first face to the first score, updating the spatial position in the third part of the tracking information of the first face to the position of the bounding box of the first face in the current frame, and updating the temporal position in the third part of the tracking information of the first face to the frame number of the current frame; In a detection frame after the initial frame, determining whether a third face satisfies a tracking end condition, wherein the third face is any one of the multiple faces; If the tracking end condition is met, outputting the capture result of the third face, and deleting the tracking information of the third face from the tracking linked list, wherein the capture result is the third part of the tracking information of the third face; The tracking state of the third face is the second state, and the duration of the third face being in the second state reaches a preset duration, and the second state is used to indicate that the second tracker is used to track the face; and / or, The tracking state of the third face is a third state, and the detection frame does not detect the third face, and the third state is used to indicate that the face is at the edge of the monitoring area; The method further comprises: In the detection frame, determining whether a new face is detected; If the new face exists, adding the tracking information of the new face to the tracking linked list, wherein the first part of the added tracking information of the new face is in the fourth state; determining an image quality of the new face; When the image quality of the new face is less than a preset threshold, deleting the tracking information of the new face from the tracking chain list; When the image quality of the new face is greater than or equal to the preset threshold, the tracking state of the new face is converted from the fourth state to the first state.

2. The method according to claim 1, characterized in that The second part of the tracking information is updated, including: In response to acquiring the position of the bounding box of the first face in a current frame, updating the second part of the tracking information of the first face, the current frame being a detection frame or a tracking frame.

3. The method according to claim 1, characterized in that: The first part of the tracking information is updated, including: In response to detecting in the detection frame that the first human face is not located at an edge of the detection frame, determining that a first portion of the tracking information of the first human face is in the first state; In response to not detecting the first face in the detection frame, updating a first part of the tracking information of the first face to a second state; In response to detecting in the detection frame that the first human face is located at an edge of the detection frame and the first human face moves toward the edge, updating a first part of the tracking information of the first human face to a third state.

4. The method according to claim 1, characterized in that: The first tracker is a tracker based on a deep learning model, and the second tracker is a kernel correlation filter tracker.

5. A face capture system, comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of any one of the methods of claims 1 to 4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images