A method and device for optimizing face selection based on video

Abstract

The present invention discloses a method and device for optimizing faces based on video, the method comprising: performing face tracking detection on video frames of compressed video to obtain face thumbnails; scoring the face thumbnails according to the clarity and noise coefficient of the face thumbnails, and for face thumbnails whose scores reach a set threshold, saving the face detection data of the face thumbnails and the timestamp of the video frames in correspondence; obtaining the position information of the face thumbnails in the video frames of the compressed video, and obtaining a large face image from the corresponding position of the original video according to the position information of the face thumbnails. The present invention improves the rate of face detection by performing face tracking detection on compressed video, can quickly find the position of the face in the compressed video, and then obtains a large face image from the corresponding position of the original video to obtain detailed face features.

Description

Technical Field

[0001] The present invention belongs to the technical field of video face detection, and in particular, relates to a video-based face detection method and device. Background Art

[0002] Video surveillance technology is widely used in the field of security. Video surveillance includes front-end cameras, transmission cables, and video surveillance platforms. The front-end camera transmits the captured video images to the control host, which then distributes the video signals to each monitor and recording device. With the development of network communication technology, video is transmitted through the network in video surveillance, realizing remote monitoring. In order to ensure the real-time nature of remote monitoring, the size of the transmitted video is limited, which affects the clarity of the monitoring video.

[0003] Face recognition technology is a biometric technology that uses facial features to identify people. It uses a camera or a webcam to collect images or video streams containing faces, and automatically detects and tracks faces in the images. Since surveillance videos often contain multiple faces, real-time face recognition of surveillance videos requires a lot of system resources.

[0004] In view of this, the present invention is proposed. Summary of the invention

[0005] The first object of the present invention is to provide a video-based face optimization method, which detects a small face image from a compressed video and obtains a large face image from the original video according to the position of the small face image, so as to reduce the system resources occupied in the face recognition process. The second object of the present invention is to provide a video-based face optimization method, which cuts out the same face multiple times within a certain period of time and scores them again, so as to improve the accuracy and breadth of face recognition.

[0006] In order to solve the above technical problems, the basic concept of the technical solution adopted by the present invention is:

[0007] A face optimization method based on video, which performs face tracking and detection on video frames of compressed video to obtain a face thumbnail;

[0008] Scoring the face thumbnails according to their clarity and noise coefficient, and for face thumbnails whose scores reach a set threshold, saving the face detection data of the face thumbnails and the timestamp of the video frame in correspondence;

[0009] The position information of the small face image in the video frame of the compressed video is obtained, and the large face image is obtained from the corresponding position of the original video according to the position information of the small face image.

[0010] Preferably, the face tracking detection on the video frames of the compressed video includes obtaining a visible code stream and an infrared code stream from the compressed video, wherein the visible code stream is a real-time code stream used for detection, and the infrared code stream is a real-time infrared code stream used for binocular living body detection;

[0011] Face tracking detection is performed on the video frames of the visible code stream and the infrared code stream respectively to obtain all the face thumbnails contained in the video frames. A binocular liveness detection algorithm is used to determine whether the detected face thumbnails are live, and all the live face thumbnails contained in the video frames are obtained.

[0012] Preferably, after obtaining all the face thumbnails contained in the video frame, the size of each face thumbnail is obtained, and the face thumbnails whose size is smaller than a set threshold are filtered out.

[0013] Preferably, the corresponding saving of the face detection data of the face thumbnail and the timestamp of the video frame includes obtaining the total number of faces in the video frame, creating multiple face tracking identifiers based on the obtained total number of faces, each face tracking identifier corresponding to a person, saving the face detection data of the face thumbnail and the matching face tracking identifier in correspondence, and saving the timestamp of the video frame where the face thumbnail is located and the face detection data of the face thumbnail in correspondence.

[0014] Preferably, the method of obtaining a large face image from a corresponding position of the original video based on the position information of the small face image includes: selecting a face tracking identifier, obtaining a small face image corresponding to the selected face tracking identifier, calculating the position ratio of the small face image in the video frame of the compressed video based on the position of the small face image in the video frame of the compressed video and the resolution of the compressed video, and cutting out the image from the corresponding position of the original video based on the resolution of the original video and the calculated position ratio of the small face image in the video frame of the compressed video to obtain the large face image.

[0015] Preferably, after obtaining the large face image from the corresponding position of the original video, face detection is performed on the large face image to obtain the size of the large face image, and large face images with a size smaller than a set threshold are filtered out.

[0016] Preferably, after filtering the large face image, a score is calculated for the large face image based on its clarity and noise coefficient. If the score of the large face image is greater than the highest score, the angle deviation of the face is detected, the blur score is calculated, and a clear frontal face image is obtained. The highest score is the highest score among the scores of all small face images corresponding to the currently selected face tracking identifier.

[0017] Preferably, if the score of the large face image is greater than the highest score, the facial attributes contained in the large face image are saved, and the facial attributes include at least skin color information, hair information, and gender information. The facial attributes contained in the saved large face image are compared with the face database to find the face.

[0018] Preferably, after obtaining the large face image, obtain the video frame timestamp of the original video containing the large face image, and determine whether the interval between the obtained timestamp and the earliest timestamp corresponding to the currently selected face tracking identifier reaches a preset time threshold. If so, reselect the large face image.

[0019] Another object of the present invention is to provide a device using the above-mentioned video-based face optimization method, comprising:

[0020] An acquisition module is used to acquire visible code stream and infrared code stream from compressed video;

[0021] A face detection module is used to detect the video frames of the visible code stream and the video frames of the infrared code stream for face tracking detection, and is connected to the acquisition module;

[0022] A face scoring module is used to score face images according to their clarity and noise coefficient, and is connected to the face detection module;

[0023] A face feature extraction module is used to extract face features contained in face images and is connected to the face scoring module;

[0024] The storage module is used to store face detection data and face features, and is connected to the face detection module and the face feature extraction module.

[0025] After adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art:

[0026] 1. The present invention improves the rate of face detection by detecting face images from compressed videos, can quickly find the position of the face in the compressed video, and then obtain the large face image at the corresponding position from the original video to obtain detailed face features, thereby reducing the system resources occupied during the face recognition process.

[0027] 2. The present invention recognizes multiple faces in the same video separately, cuts out the same face multiple times within a certain period of time, and scores them again to obtain the face picture with the highest total score for comparison with the face library, effectively improving the recognition accuracy and breadth.

[0028] The specific implementation modes of the present invention are further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings are part of the present invention and are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but do not constitute an improper limitation of the present invention. Obviously, the drawings described below are only some embodiments. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the accompanying drawings:

[0030] Figure 1 It is a flow chart of a video-based face optimization method of the present invention;

[0031] Figure 2 It is a flow chart of face tracking detection of the present invention;

[0032] Figure 3 It is a schematic diagram of the linked list structure of the present invention;

[0033] Figure 4 This is a flow chart of face optimization according to the present invention.

[0034] It should be noted that these drawings and textual descriptions are not intended to limit the conceptual scope of the present invention in any way, but are intended to illustrate the concept of the present invention for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. The following embodiments are used to illustrate the present invention but are not used to limit the scope of the present invention.

[0036] In the description of the present invention, it should be noted that the directions or positional relationships indicated by terms such as “upper”, “lower”, “front”, “back”, “left”, “right”, “vertical”, “inside” and “outside” are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as a limitation on the present invention.

[0037] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0038] like Figure 1 As shown, the present invention introduces a video-based face optimization method, comprising:

[0039] Face detection and tracking: During the face detection and tracking process, face thumbnails and related face detection data contained in the compressed video are detected, and scores are calculated for the face thumbnails;

[0040] Face optimization: During the face optimization process, a large face image is cut out from the corresponding position in the original video according to the position of the small face image in the compressed video, and the large face image is comprehensively scored. Finally, the cutout feature with the highest score is compared with the features of the face library to find the face.

[0041] The present invention can detect small face images contained in compressed videos through face detection and tracking, and score the small face images. In the face optimization process, large face images are cut out from the original video and scored again. The face with the highest total score is compared with the face library, which can reduce the system resources occupied in the face recognition process and effectively improve the accuracy and breadth of face recognition.

[0042] Embodiment 1

[0043] like Figure 2 As shown, this embodiment introduces a video-based face optimization method, which scores face thumbnails during face detection and tracking, filters out faces with low scores, and reduces the system resources required to be consumed.

[0044] The face detection and tracking process includes the following steps:

[0045] S101, obtaining a frame of visible code stream and infrared code stream.

[0046] S102, performing face tracking detection on the visible code stream and the infrared code stream respectively, filtering out edge faces and faces with too small a size within a certain threshold, and obtaining the number of faces and handle information in the video image.

[0047] S103: Match the face data detected by the visible code stream and the infrared code stream.

[0048] S104. Use a binocular liveness algorithm to determine whether the detected face is alive. If it is not alive, there is no need to waste resources on face recognition, and the face information is discarded.

[0049] S105 . For a detected living face, the position of the face in the video frame is detected and converted into a width-to-height ratio of 10,000 for adapting to various resolution conversions.

[0050] S106: Obtain a face tracking ID for tracking multiple faces in the picture, where each face tracking ID corresponds to a different person.

[0051] S107, obtain 81 facial feature points such as the eye area, nose, mouth, etc., score the clarity and noise coefficient, put the faces whose scores reach a certain threshold into linked list 1, and use the face tracking ID as the key for optimization. In order to meet the requirement that there are multiple face information of the same person that pass the scoring threshold, the second parameter of linked list 1 is the frame time as the key, and linked list 2 such as the position of the face frame detected at that time is saved.

[0052] The corresponding linked list is defined as follows:

[0053] std::map<face tracking ID, std::map<frame timestamp, detected face related information>>.

[0054] In step S101, the visible code stream and the infrared code stream are obtained from the compressed video. The visible code stream is a real-time code stream for detection, with a resolution of 720P and a frame rate of 15 frames. It is a YUV code stream that has not been coded. The infrared code stream is a real-time infrared code stream for binocular liveness detection, with a resolution of 720P and a frame rate of 15 frames. It is a YUV code stream that has not been coded. In the surveillance video, the resolution of the compressed video is lower than that of the original video. Since the resolution of the compressed video is low, the information contained in the video frame of the compressed video is relatively small, and the area where the face is located can be quickly found from the compressed video. Then, the corresponding position of the original video is cut out. The original video has a high resolution, and very detailed facial features can be obtained from the original video, such as skin color, hair, whether to wear a mask, etc. The facial features obtained from the original video are compared with the face database to find the face and confirm the identity.

[0055] In step S102, face tracking detection is performed on the video frames of the visible code stream and the infrared code stream to obtain all face thumbnails contained in the video frames, obtain the size of each face thumbnail, and filter out face thumbnails whose size is smaller than a set threshold.

[0056] In some scenarios, such as access control systems, only a single face in the captured video needs to be identified. However, the camera cannot guarantee that only one face is captured during shooting. Often, there are multiple faces in the captured video image, and the sizes of the captured faces are not exactly the same. The face closest to the camera is the largest. When a user is identified in front of the access control, he is close to the camera, and the user's face is the largest in the video image captured by the camera. At the same time, other faces captured are far from the camera source, and other faces in the video image captured by the camera are relatively small. The present invention improves the accuracy of face recognition and avoids identifying unnecessary faces by filtering face images according to their size.

[0057] The face thumbnails may include edge face images, which are face images at the edge of the video frame image and with incomplete face information. Due to the existence of boundaries during video shooting, face images with incomplete information may be captured. For example, an edge face image only includes the mouth and chin, while the nose, eyes and other areas above the mouth are not captured. The present invention also filters the edge face images, filters out the smaller edge face images, and improves the efficiency of face recognition.

[0058] In step S104, since there are posters, photos and other images in reality, and there are faces on these images, these faces do not represent real people, and these face information is usually not needed in monitoring scenarios. If the monitoring video captures these images, detecting the faces on these images will cause a waste of resources. Users hope to detect real people captured in the monitoring video, so faces that are not living are directly discarded and do not need to be recognized, avoiding a waste of resources. Figure 3 As shown, face tracking IDs are created based on the total number of people in the video, and each face tracking ID corresponds to one person. Create linked list 1 and use the face tracking ID as the key of linked list 1. For example, if the number of video frames is 100 and a person appears in the entire video, then the number of face images of this person in the video should be 100, with one face image in each frame; if a person only appears in a certain part of the video and does not appear in the rest of the video, then the number of face images of this person should be less than 100. Each face tracking ID corresponds to a linked list 2, and the frame time is used as the key of linked list 2. Information such as the position of the face frame detected at that time is saved in linked list 2.

[0059] When shooting surveillance videos, multiple people are often captured in the surveillance videos at the same time, and face tracking and recognition needs to be performed for each person separately. A video sequence contains multiple frames of pictures, and the faces of the same person contained in the multiple frames are often different. The present invention performs face detection on multiple different faces of the same person in the video, obtains multiple groups of different face detection data, and saves the face detection data in correspondence with the timestamp of the video frame, so as to facilitate finding the preferred face.

[0060] Embodiment 2

[0061] like Figure 4 As shown, this embodiment introduces a video-based face optimization method, which includes a face optimization process. In the face optimization process, a large face image is cut out from the corresponding position in the original video according to the position of the small face image in the compressed video, and the score of the large face image is calculated again, and the face with the highest total score is compared with the face library, which can effectively improve the recognition accuracy and breadth.

[0062] In the face selection process, it is necessary to determine whether the interval between the earliest face frame timestamp in the linked list and the current frame timestamp reaches 400ms, and loop to read the face tracking ID from linked list 1 to find the cutout with the highest score for this ID.

[0063] Determine whether the earliest face frame timestamp in the linked list is 400ms away from the current frame timestamp. The frame timestamp is used to prevent inaccurate time caused by system time modification. The earliest face frame time in the linked list is compared with the current frame time to ensure that too much computing power is not wasted on frames without face data, and that faces can be quickly recognized.

[0064] like Figure 4 As shown in FIG. 1 , the face selection process specifically includes the following steps:

[0065] S201, read the face tracking ID from linked list 1, cyclically read a face detection data from linked list 2, and extract the face image at the corresponding position from the original video according to the position information.

[0066] S202, performing face detection on the face cutout, filtering out edge faces with a certain threshold and faces with too small a size, and obtaining detected face information.

[0067] S203, detect 81 facial feature points such as the periphery of the eyes, nose, and mouth, and make a comprehensive score for clarity and noise coefficient. If the score of the face reaches the set threshold and is greater than the highest score of the current face tracking ID, continue.

[0068] S204, detecting the angle deviations of the face in the X, Y, and Z axes, calculating the blur score, obtaining a clear front face, and continuing if passed.

[0069] S205. Obtain facial attributes, including but not limited to age, gender, mask, skin color, hair, and other information.

[0070] S206: Save the current score as the highest score of the current face tracking ID, and save the relevant features of the face for comparison with the features of the face library.

[0071] S207: Compare the highest-scoring cutout feature obtained by the current face tracking ID with the features in the face database to find the face.

[0072] In step S201, since the resolution of the compressed video is different from that of the original video, the position ratio of the small face image in the video frame of the compressed video is first calculated, and then the large face image is cut out from the corresponding position of the video frame of the original video according to the calculated position ratio and the resolution of the original video. For example, in this embodiment, the resolution of the compressed video is 720P, the size of the video frame is 1280*720, the lower left corner vertex of the video frame is the coordinate origin, the width of the video frame is the X axis, the height of the video frame is the Y axis, and the video frame is located in the first quadrant of the coordinate system. After face detection, a rectangular face frame surrounding the face is obtained. The vertex coordinates of the lower left corner of the face frame are (128, 72), and the upper right corner of the face frame are (256, 144). The resolution of the original video is 1080P, and the video frame size is 1920*1080. Then the vertex coordinates of the lower left corner of the face frame in the original video are (192, 108), and the vertex coordinates of the upper right corner of the face frame in the original video are (384, 216).

[0073] In the surveillance video, the face of a person is not necessarily facing the surveillance camera, resulting in the face of the person being offset in the surveillance video. The present invention detects the offset of each angle of the face and obtains a clear front face for optimization.

[0074] In the face selection process, the present invention cuts out the same face multiple times within a certain period of time, scores them again, and compares the face picture with the highest total score with the face library, thereby effectively improving the accuracy and breadth of face recognition.

[0075] An embodiment of the present invention is a device that adopts the video-based face optimization method as described above, including: an acquisition module, used to acquire a visible code stream and an infrared code stream from a compressed video; a face detection module, used to perform face tracking detection on video frames of the visible code stream and video frames of the infrared code stream, respectively, and connected to the acquisition module; a face scoring module, used to score face images according to the clarity and noise coefficient of the face images, and connected to the face detection module; a face feature extraction module, used to extract face features contained in the face images, and connected to the face scoring module; a storage module, used to store face detection data and face features, and connected to the face detection module and the face feature extraction module; a face recognition module, used to compare face features extracted from the face images with face features stored in a face database, and connected to the face feature extraction module and the storage module.

[0076] In the process of real-time face tracking and recognition of surveillance video, the surveillance video is shot by an infrared binocular camera. Get a frame of visible code stream and infrared code stream. The visible code stream is a real-time code stream used for detection, with a resolution of 720P and a frame rate of 15 frames. The visible code stream is the collected yuv code stream that has not been encoded. The infrared code stream is a real-time infrared code stream used for binocular liveness detection. The resolution and frame rate of the infrared code stream are the same as those of the visible code stream.

[0077] Perform face tracking detection on the visible code stream video frame and the infrared code stream video frame respectively, obtain all the face thumbnails contained in the video frame, obtain the size of each face thumbnail, and filter out the face images whose size is smaller than the set threshold. At the same time, edge faces are also filtered out. Edge faces refer to faces at the edge of the video frame. When shooting a video, since the camera can only shoot videos within a certain range, if there is someone at the shooting boundary, an edge face will be formed. The edge face may only contain partial facial information. For example, if a person walks from outside the shooting boundary to inside the shooting boundary, there may be only a chin in one frame of the video, but a mouth appears in the next frame.

[0078] All face thumbnails detected from the visible code stream video frames and the infrared code stream video frames are matched, and a binocular liveness detection algorithm is used to determine whether the detected face thumbnails are alive. If the detected face is not alive, the face information is discarded, and all alive face thumbnails are obtained.

[0079] Create face tracking IDs based on the total number of people in the video, with each face tracking ID corresponding to one person. Create a main linked list, which is used to save the correspondence between the face tracking ID and all the faces of the corresponding person in the video. For example, if the number of video frames is 100 and a person appears throughout the video, then the number of face images of this person in the video should be 100, with one face image in each frame; if a person only appears in a certain part of the video and does not appear in the rest of the video, then the number of face images of this person should be less than 100. Use the face tracking ID as the key of the main linked list, and the face tracking ID corresponds to a sub-linked list. The sub-linked list is used to save the detected face image data and frame timestamps, and the frame timestamp is used as the key of the sub-linked list.

[0080] Multiple facial feature points such as the eye area, nose, and mouth are obtained from the face thumbnail, and the face thumbnail is scored according to its clarity and noise coefficient. The face thumbnails with scores exceeding a preset scoring threshold are stored in a main linked list. The second parameter of the main linked list is a sub-linked list, which stores the timestamp of the video frame and the face detection data. The face detection data includes the position information of the face thumbnail in the video frame of the compressed video.

[0081] A face tracking ID is selected, and a small face image is read from the corresponding sub-link list in a loop. According to the face position information, a large face image is obtained by cutting out the image from the corresponding position in the original video. Since the resolution of the compressed video and the original video is different, the position ratio of the small face image in the video frame of the compressed video is first calculated, and then the large face image is obtained by cutting out the image from the corresponding position of the video frame of the original video according to the calculated position ratio and the resolution of the original video. For example, in this embodiment, the resolution of the compressed video is 720P, the video frame size is 1280*720, the lower left corner vertex of the video frame is the coordinate origin, the width of the video frame is the X-axis, the height of the video frame is the Y-axis, and the video frame is located in the first quadrant of the coordinate system. After face detection, a rectangular face frame surrounding the face is obtained. The vertex coordinates of the lower left corner of the face frame are (128, 72), and the upper right corner of the face frame are (256, 144). The resolution of the original video is 1080P, and the video frame size is 1920*1080. Then the vertex coordinates of the lower left corner of the face frame in the original video are (192, 108), and the vertex coordinates of the upper right corner of the face frame in the original video are (384, 216).

[0082] Perform face detection on the large face image to obtain all the faces contained in the large face image, filter out faces whose size is smaller than a preset size threshold, and also filter out edge faces.

[0083] Detect multiple facial feature points such as the eye area, nose, mouth, etc. in the large face image, and score the large face image according to the clarity and noise coefficient. If the score of the large face image is greater than the highest score, detect the angle deviation of the face, calculate the blur score, and obtain a clear frontal face image. The highest score is the highest score among the scores of all small face images corresponding to the currently selected face tracking ID.

[0084] The facial features contained in the face image are obtained, the facial features at least including skin color information, hair information, gender information, and other information. The facial features are compared with the face database to find the face and confirm the identity.

[0085] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0086] All features disclosed in this specification (including accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this specification (including accompanying claims, abstracts and drawings) may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless otherwise expressly stated, each feature disclosed in this specification (including accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0087] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present invention and to form different embodiments.

[0088] The above description is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this patent can make some changes or modifications to equivalent embodiments of equivalent changes by using the technical content suggested above without departing from the scope of the technical solution of the present invention. The implementation scheme in the above embodiment can also be further combined or replaced. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of the solution of the present invention.

Claims

1. A face optimization method based on video, characterized in that: Perform face tracking and detection on the video frames of the compressed video to obtain a small face image; Scoring the face thumbnails according to their clarity and noise coefficient, and for face thumbnails whose scores reach a set threshold, saving the face detection data of the face thumbnails and the timestamp of the video frame in correspondence; Obtaining position information of the small face image in the video frame of the compressed video, and obtaining a large face image from a corresponding position of the original video according to the position information of the small face image; The face tracking detection on the video frames of the compressed video includes obtaining a visible code stream and an infrared code stream from the compressed video, wherein the visible code stream is a real-time code stream used for detection, and the infrared code stream is a real-time infrared code stream used for binocular living body detection; Perform face tracking detection on the video frames of the visible code stream and the infrared code stream respectively to obtain all face thumbnails contained in the video frames, use the binocular liveness detection algorithm to determine whether the detected face thumbnails are live, and obtain all live face thumbnails contained in the video frames; After obtaining the large face image, obtain the video frame timestamp of the original video where the large face image is located, and determine whether the interval between the obtained timestamp and the earliest timestamp corresponding to the currently selected face tracking identifier reaches a preset time threshold. If so, reselect the large face image.

2. The video-based face selection method according to claim 1, characterized in that: After obtaining all face thumbnails contained in the video frame, the size of each face thumbnail is obtained, and face thumbnails whose size is smaller than a set threshold are filtered out.

3. The video-based face selection method according to claim 1, characterized in that: The corresponding saving of the face detection data of the face thumbnail and the timestamp of the video frame includes obtaining the total number of faces in the video frame, creating multiple face tracking identifiers according to the obtained total number of faces, each face tracking identifier corresponding to a person, and saving the face detection data of the face thumbnail and the matching face tracking identifier in correspondence, and saving the timestamp of the video frame where the face thumbnail is located and the face detection data of the face thumbnail in correspondence.

4. The video-based face selection method according to claim 3, characterized in that: The method of obtaining a large face image from a corresponding position of an original video based on position information of a small face image includes: selecting a face tracking identifier, obtaining a small face image corresponding to the selected face tracking identifier, calculating a position ratio of the small face image in a video frame of the compressed video based on the position of the small face image in a video frame of the compressed video and a resolution of the compressed video, and cutting out a picture from a corresponding position of the original video based on the resolution of the original video and the calculated position ratio of the small face image in a video frame of the compressed video to obtain a large face image.

5. The video-based face selection method according to claim 4, characterized in that: After obtaining the large face image from the corresponding position of the original video, face detection is performed on the large face image to obtain the size of the large face image, and the large face images whose size is smaller than the set threshold are filtered out.

6. The method for selecting faces based on video according to claim 5, characterized in that: After filtering the large face image, the score of the large face image is calculated based on its clarity and noise coefficient. If the score of the large face image is greater than the highest score, the angle deviation of the face is detected, the blur score is calculated, and a clear frontal face image is obtained. The highest score is the highest score among the scores of all small face images corresponding to the currently selected face tracking identifier.

7. The video-based face selection method according to claim 6, characterized in that: If the score of the large face image is greater than the highest score, the facial features contained in the large face image are saved, and the facial features include at least skin color information, hair information, and gender information. The facial features contained in the saved large face image are compared with the face database to find the face.

8. A device using the video-based face optimization method as described in any one of claims 1 to 7, characterized in that: include: An acquisition module is used to acquire visible code stream and infrared code stream from compressed video; A face detection module, used for performing face tracking detection on the video frames of the visible code stream and the video frames of the infrared code stream respectively, and connected to the acquisition module; A face scoring module is used to score face images according to their clarity and noise coefficient, and is connected to the face detection module; A face feature extraction module is used to extract face features contained in face images and is connected to the face scoring module; A storage module, used for storing face detection data and face features, connected with the face detection module and the face feature extraction module; The face recognition module is used to compare the face features extracted from the face image with the face features stored in the face database, and is connected to the face feature extraction module and the storage module.

Images