A method, system and device for acquiring face sample set

By detecting and tracking faces on short videos, combining face picture sets with high similarity and automatically labeling, the problem of low efficiency and insufficient accuracy of face sample set acquisition in the existing technology is solved, and efficient and automated Face sample set acquisition.

CN112488072BActive Publication Date: 2024-11-22广州海格星航信息科技有限公司
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202011520727.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-12-21
Publication Date
2024-11-22
Estimated Expiration
2040-12-21

AI Technical Summary

Technical Problem

When obtaining face sample sets, existing facial recognition technology relies on manual annotation, which is costly and inefficient, making it difficult to meet complex environments and multi-scale needs, affecting the generalization ability of the model.

Method used

By performing face detection and tracking on each frame of a short video, a high-similar face picture collection is merged, facial features are extracted and automatically labeled, reducing manual participation and improving labeling efficiency.

Benefits of technology

It realizes automated face sample set acquisition, reduces manual labeling costs, improves labeling efficiency and accuracy, and adapts to complex environments and multi-scale needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112488072B_ABST
    Figure CN112488072B_ABST
Patent Text Reader

Abstract

The present invention discloses a method, system and device for acquiring a face sample set. The present invention performs face detection and face tracking on each frame of each short video in a video set to obtain a first object picture collection corresponding to each short video. According to the similarity between the face picture collections corresponding to each character object in each first object picture collection, the face picture collections corresponding to each character object in all short videos are merged to obtain a final face picture collection of each character object. The face features in the final face picture collection of each character object are extracted and annotated to obtain a face sample set. In the process of obtaining the face sample set, the embodiment of the present invention does not require human participation. The face picture collections are merged by calculating the similarity between the face pictures, and the face features are extracted to achieve the annotation of the face picture of each object, thereby reducing the labor cost of manual annotation and improving the image annotation efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of face recognition, and in particular to a method, system and device for acquiring a face sample set. Background Art

[0002] At present, with the rapid development of the Internet and mobile terminals, the Internet has penetrated into all aspects of people's lives. With the emergence of short video software, more and more people are keen to post their short videos of life on the Internet, which is a huge opportunity for face recognition. If we can make full use of these massive network resources and apply them to face recognition data sets, it will be of great significance to the improvement of face recognition technology.

[0003] As a key technology in the current security field, face recognition has very high requirements for recognition accuracy. The current mainstream face recognition technologies are all based on deep learning methods, using convolutional neural networks to train a feature extractor, and finally calculating the distance between faces to determine whether the faces match. The high accuracy of face recognition is very dependent on a large amount of face data, but the manual processing of a large amount of face data is very expensive, and the current booming development of short videos has greatly facilitated the collection of face data. If we can make full use of short video data and combine face-related algorithms to automatically generate face image samples, it will be conducive to the development of face recognition technology.

[0004] At present, the methods for labeling faces are usually the following:

[0005] 1) In a fixed environment, shoot faces from various angles and expressions.

[0006] The collection of facial data usually relies on fixed video recording equipment to record faces in multiple poses in the same environment, save them as a video, and use a face detection algorithm to extract face samples from the video to generate a new ID. This method is suitable for situations with small amounts of data.

[0007] 2) In an open environment, personnel cooperate with monitoring equipment to take pictures.

[0008] In a real surveillance environment, one or more people simulate the specified actions in the scenario, collect facial data, and then manually annotate the surveillance video.

[0009] 3) Image search engines retrieve celebrity face data or extract them from film and television videos.

[0010] Downloading celebrity photos or film and television works from major search engines is a common solution for public data sets.

[0011] However, the current face annotation methods have the following shortcomings:

[0012] The solution under a fixed environment has too simple a face collection environment, which makes it difficult to meet the influence of complex environments in actual scenes; and the size of the recorded faces is fixed, which makes it difficult to meet the multi-scale requirements for faces in actual scenes; the changes in facial posture and angle are not rich enough, resulting in a large number of similar face samples.

[0013] Solutions in open scenarios require huge human and material resources, and the dataset production cycle is very long.

[0014] The celebrity pictures downloaded by the search engine retrieval solution are very different from the pictures of the actual scenes, which will affect the generalization ability of the final face recognition model. Summary of the invention

[0015] The present invention provides a method, system and device for acquiring a face sample set. The present invention does not require human participation in the process of obtaining the face sample set, reduces the labor cost of manual labeling, and improves the efficiency of image labeling.

[0016] In order to solve the above technical problem, an embodiment of the present invention provides a method for obtaining a face sample set, comprising the following steps:

[0017] Performing face detection and face tracking on each frame of each short video in the video set to obtain a first object image collection corresponding to each short video; wherein the video set includes multiple short videos; each first object image collection includes a face image collection corresponding to multiple human objects;

[0018] According to the similarity between the face picture sets corresponding to each character object in each first object picture collection, the face picture sets corresponding to each character object in all short videos are merged to obtain the second object picture collection corresponding to each character object, and the final face picture set of each character object is obtained based on the second object picture collection corresponding to each character object; wherein each second object picture collection includes multiple subsets corresponding to the same character object, and each subset contains multiple face pictures corresponding to the same character object;

[0019] The facial features of each face image in the final face image set of each character object are extracted respectively, and each face image is labeled in turn according to the extracted facial features to obtain the face sample set corresponding to each character object, and then the total face sample set of all character objects is obtained.

[0020] Preferably, face detection and face tracking are performed on each frame of each short video in the video set to obtain a collection of first object images corresponding to each short video as follows:

[0021] Perform face detection on each frame in each short video to obtain the face frame of each character in each frame;

[0022] The facial features of the face frame of each character in each frame of each short video are extracted, and the facial features of each frame in each short video are tracked using a face tracking algorithm. Each frame with the same facial features tracked in the same short video is saved as a face image set of the same character object, and a first object face image collection corresponding to each short video is obtained.

[0023] Preferably, according to the similarity between the face picture sets corresponding to the character objects in each first object picture collection, the face picture sets corresponding to the character objects in all the short videos are merged to obtain the second object picture collection corresponding to each character object as follows:

[0024] Calculate the similarity between the face picture sets corresponding to the various character objects in the first object picture collection of each short video; when the similarity between the face picture set corresponding to any character object and the face picture set corresponding to other character objects is greater than a preset first threshold, merge the face picture sets corresponding to the two character objects;

[0025] The similarity between the face picture sets corresponding to each character object in all short videos is calculated. When the similarity between the face picture sets corresponding to any two character objects in different short videos is greater than a preset second threshold, the face picture sets corresponding to the two character objects are merged as subsets corresponding to the same character object. When all the character objects in all short videos are merged, a second object picture collection corresponding to each character object is obtained.

[0026] Preferably, the similarity between the face image sets is the cosine distance between the face image sets.

[0027] Preferably, the specific process of obtaining the final face picture set of each character object based on the second object picture set corresponding to each character object is as follows:

[0028] A subset with the largest number of face images is selected from the second object image collection corresponding to each person object, and is used as the final face image set for each person object.

[0029] Preferably, the facial features of each face picture in the final face picture set of each character object are extracted respectively, and each face picture is labeled in turn according to the extracted facial features to obtain the face sample set corresponding to each character object, and then the specific process of obtaining the total face sample set of all character objects is as follows:

[0030] The face pose estimation algorithm, face attribute recognition algorithm and face area brightness algorithm are used to extract the face features of each face picture in the final face picture set of each character object. According to the extracted face features, each face picture is marked as a difficult sample or an easy sample in turn, so as to obtain the face sample set corresponding to each character object, and then obtain the total face sample set of all character objects.

[0031] Preferably, a face posture estimation algorithm is used to extract whether the face in each face image is a frontal face or a side face; a face attribute recognition algorithm is used to extract the face gender, expression, whether the face is wearing a mask, and whether the face is wearing sunglasses in each face image; a face area brightness algorithm is used to calculate the average brightness of each face image, and determine whether the face image is a strong light face or a low light face based on the average brightness.

[0032] Preferably, if the face image in the final face image set of each object has one of the features of side face, wearing a mask, wearing sunglasses, face in strong light, and face in low light, the face image is marked as a difficult sample, otherwise it is marked as an easy sample.

[0033] In order to further illustrate the solution of the present invention, the present invention also provides a face sample set acquisition system, including a picture collection generation module, a merging module and a labeling module;

[0034] The image collection generation module is used to perform face detection and face tracking on each frame of each short video in the video set, and obtain a first object image collection corresponding to each short video; wherein the video set includes multiple short videos; each first object image collection includes a face image collection corresponding to each of multiple character objects;

[0035] The merging module is used to merge the face picture sets corresponding to the character objects in all the short videos according to the similarity between the face picture sets corresponding to the character objects in each first object picture collection, and obtain the second object picture collection corresponding to each character object, so as to obtain the final face picture set of each character object; wherein each second object picture collection includes multiple subsets corresponding to the same character object, and each subset contains face pictures corresponding to the same character object;

[0036] The labeling module is used to extract the facial features of each face image in the final face image set of each character object, and label each face image in turn according to the extracted facial features to obtain the face sample set corresponding to each character object, and then obtain the total face sample set of all character objects.

[0037] In order to further illustrate the solution of the present invention, the present invention also provides a face sample set acquisition device, including a processor and a memory;

[0038] The memory is used to store program code and transmit the program code to the processor;

[0039] The processor is used to execute the above-mentioned method for acquiring a face sample set according to the instructions in the program code.

[0040] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0041] 1. The present invention performs face detection and face tracking on each frame of each short video in the video collection to obtain a first object picture collection corresponding to each short video, and merges the face picture collections corresponding to each character object in all short videos according to the similarity between the face picture collections corresponding to each character object in each first object picture collection to obtain a final face picture collection for each character object, extracts and annotates the face features in the final face picture collection of each character object, and obtains a face sample set. The embodiment of the present invention does not require human participation in the process of obtaining a face sample set, and merges the face picture collections by calculating the similarity between face pictures, and extracts face features to annotate the face pictures of each object, thereby reducing the labor cost of manual annotation and improving the image annotation efficiency.

[0042] 2. In the process of labeling the face picture of each object, the embodiment of the present invention extracts the features of each face picture through the face posture estimation algorithm, the face attribute recognition algorithm and the face area brightness algorithm, including whether each face is a frontal face or a side face, the gender, expression, whether wearing a mask and sunglasses of each face picture, and whether each face picture is a strong light face or a low light face; thereby, each face picture can be divided into a finer granularity, greatly improving the accuracy of face picture labeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 : A method flow chart of a method, system and device for obtaining a face sample set provided in an embodiment of the present invention.

[0044] Figure 2 : This is an example diagram of a short video downloaded from the Internet based on keywords according to an embodiment of the present invention.

[0045] Figure 3 : Generate an example image for a face image set of a single person object according to an embodiment of the present invention.

[0046] Figure 4 : is a schematic diagram of the merging strategy of different character objects in an embodiment of the present invention.

[0047] Figure 5 : is a schematic diagram of the merging strategy between different videos according to an embodiment of the present invention.

[0048] Figure 6 : A method flow chart of a method, system and device for obtaining a face sample set provided in an embodiment of the present invention.

[0049] Figure 7 : A schematic diagram of a final face picture set of each character object obtained according to an embodiment of the present invention.

[0050] Figure 8 : A flowchart for extracting facial features of each facial image in the final facial image set of each person object and marking each facial image in an embodiment of the present invention.

[0051] Fig. 9 : A system structure diagram of a face sample set acquisition method, system and device provided in an embodiment of the present invention.

[0052] Fig.10 : A device framework diagram of a method, system and device for acquiring a face sample set provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] Embodiment 1

[0055] Please refer to Figure 1 , is a method for obtaining a face sample set provided by an embodiment of the present invention. The method is applicable to short videos pre-downloaded from the Internet according to keywords. It should be further explained that, due to the variety and complexity of short video types in short video platforms, several situations are listed below:

[0056] Some short video platforms mostly have users recording works from a first-person perspective, which is the preferred data;

[0057] Some short video platform users will direct and act in small TV series by themselves. The number of people is large, but the personnel is relatively fixed. This is the alternative data;

[0058] Some short video platform users only take photos of others or scenery, and rarely take selfies, which is unnecessary data;

[0059] Some short video platform users only forward videos, most of which are related to recent hot topics, including recent national events, hot topics, celebrity anecdotes, etc. This is the data that needs to be cleaned;

[0060] Some short video platform users like to post videos related to animation and games, which are data that are difficult to clean.

[0061] Therefore, before downloading a video, it is necessary to determine the type of short video content, so as to filter out most of the unnecessary short videos to a certain extent. In this embodiment, multiple short videos are downloaded from the network in advance according to keywords, such as Figure 2 shown.

[0062] An embodiment of the present invention provides a method for obtaining a face sample set, comprising the following steps:

[0063] S101: Since each short video is composed of a set of continuous frame sequences, the processing of the short video can be converted into processing each frame of the short video; first, face detection is performed on each frame of each short video in the video set. If no face exists, jump to the next frame. Since the short video may not be limited to a single face, it is necessary to use a tracking algorithm to track multiple human objects. A subfolder is established according to the tracked human objects. The subfolder stores the face images corresponding to the target human objects. At this point, a single video is converted into a set of face images corresponding to different human objects, such as Figure 3 As shown, a first object picture collection corresponding to the short video is obtained;

[0064] S102: However, the face picture sets corresponding to multiple person objects are not unique, that is, the most important difference between short videos and surveillance videos is that the pictures of short videos are not continuous. There are multiple cross-switching of video perspectives in short videos, which greatly affects the tracking algorithm, resulting in the face picture set corresponding to the same person object being divided into face picture sets corresponding to different person objects due to the conversion of the video picture. Therefore, it is necessary to merge the face picture sets corresponding to multiple different person objects, by calculating the similarity between the face picture sets corresponding to each person object in each first object picture collection in each short video, and merging the face picture sets corresponding to each person object in all short videos according to the similarity between the face picture sets corresponding to each person object in each first object picture collection, respectively obtaining the second object picture collection corresponding to each person object, and obtaining the final face picture set of each person object based on the second object picture collection corresponding to each person object; wherein each second object picture collection includes multiple subsets corresponding to the same person object, and each subset contains multiple face pictures corresponding to the same person object, such as Figure 4 , Figure 5 shown.

[0065] S103: After obtaining the final face picture set of each object, extract the facial features of each face picture in the final face picture set of each character object respectively, and label each face picture in turn according to the extracted facial features to obtain the face sample set corresponding to each character object, and then obtain the total face sample set of all character objects.

[0066] The embodiment of the present invention performs face detection and face tracking on each frame of each short video in the video collection to obtain a first object picture collection corresponding to each short video, merges the face picture collections corresponding to each character object in all short videos according to the similarity between the face picture collections corresponding to each character object in each first object picture collection, obtains a final face picture collection for each character object, extracts and annotates the face features in the final face picture collection of each character object, and obtains a face sample set. The embodiment of the present invention does not require human participation in the process of obtaining the face sample set, merges the face picture collections by calculating the similarity between the face pictures, and extracts the face features to annotate the face pictures of each object, thereby reducing the labor cost of manual annotation and improving the image annotation efficiency.

[0067] Embodiment 2

[0068] Please refer to Figure 6 , a method for obtaining a face sample set provided by an embodiment of the present invention, the method is applicable to a plurality of short videos pre-downloaded from the network according to keywords, and comprises the following steps:

[0069] S201: Since each short video is composed of a set of continuous frame sequences, the processing of the short video can be converted into processing each frame of the short video; first, face detection is performed on each frame of each short video in the video set. If there is no face, jump to the next frame. Since the short video may not be limited to a single face, it is necessary to use a tracking algorithm to track multiple human objects. A subfolder is established according to the tracked human object. The subfolder stores the face image corresponding to the target human object. At this point, a single video is converted into a set of face images corresponding to different human objects, such as Figure 3 As shown, a first object picture collection corresponding to the short video is obtained;

[0070] It should be further explained that the specific process of performing face detection and face tracking on each frame of each short video in the video set to obtain the first object picture collection corresponding to each short video is as follows:

[0071] Perform face detection on each frame in each short video to obtain the face frame of each character in each frame;

[0072] The face detection algorithm can adopt the existing face detection algorithm. The common face detection algorithm is basically a process of "scanning" and "discrimination", that is, the algorithm scans within the image range, and then determines whether the candidate area is a face one by one. Therefore, the calculation speed of the face detection algorithm will be related to the image size and image content. In this embodiment, the algorithm can be accelerated by setting the "input image size", or the "minimum face size limit", or the "upper limit on the number of faces". In this embodiment, the face detection algorithm (such as MTCNN, CenterFace, LFFD, etc.) is trained based on a deep learning method. By reading each frame of each short video, preprocessing each frame, and inputting it into the face detection algorithm model, the face detection algorithm model outputs the face coordinates in each frame (upper left corner [x1, y1] coordinates and lower right corner [x2, y2] coordinates) and the confidence of the face.

[0073] The facial features of the face frame of each character in each frame of each short video are extracted, and the facial features of each frame in each short video are tracked using a face tracking algorithm, and each frame with the same facial features tracked in the same short video is saved as a face picture set of the same character object, and a first object face picture collection corresponding to each short video is obtained. It should be further explained that, in this embodiment, the face feature extractor is trained using the ResNet series backbone network, the face frame of each character in each frame in the previous step is detected and preprocessed, and the pixels of the face frame of each character (including the pixels of the face) are input into the face feature extraction model, and the face feature extraction model automatically extracts the features of the face and outputs a feature vector of a fixed dimension.

[0074] S202: However, the face picture sets corresponding to multiple human objects are not unique, that is, the most important difference between short videos and surveillance videos is that the pictures of short videos are not continuous. There are multiple cross-switching of video perspectives in short videos, which greatly affects the tracking algorithm, resulting in the face picture set corresponding to the same human object being divided into face picture sets corresponding to different human objects due to the conversion of the video picture. Therefore, it is necessary to merge the face picture sets corresponding to multiple different human objects, by calculating the similarity between the face picture sets corresponding to each human object in each first object picture collection in each short video, and merging the face picture sets corresponding to each human object in all short videos according to the similarity between the face picture sets corresponding to each human object in each first object picture collection, and respectively obtaining the second object picture collection corresponding to each human object, and selecting the subset with the largest number of face pictures from the second object picture collection corresponding to each human object, and using it as the final face picture set for each human object, such as Figure 7As shown, each second object image collection includes multiple subsets corresponding to the same person object, and each subset includes multiple face images corresponding to the same person object;

[0075] It should be further explained that, according to the similarity between the face picture sets corresponding to the various character objects in each first object picture collection, the face picture sets corresponding to the various character objects in all short videos are merged to obtain the specific process of the second object picture collection corresponding to each character object as follows:

[0076] Calculate the similarity between the face picture sets corresponding to the various character objects in the first object picture collection of each short video; when the similarity between the face picture set corresponding to any character object and the face picture set corresponding to other character objects is greater than a preset first threshold, merge the face picture sets corresponding to the two character objects;

[0077] The similarity between the face picture sets corresponding to each character object in all short videos is calculated. When the similarity between the face picture sets corresponding to any two character objects in different short videos is greater than a preset second threshold, the face picture sets corresponding to the two character objects are merged as subsets corresponding to the same character object. When all the character objects in all short videos are merged, a second object picture collection corresponding to each character object is obtained.

[0078] The distances between faces are represented by Euclidean distance and cosine similarity. In this embodiment, cosine similarity is used as an example. Cosine similarity, also called cosine distance, uses the cosine value of the angle between two vectors in a vector space as a measure of the size of the difference between two individuals. When the angle θ between two vectors approaches 0, the closer the two vectors are, the smaller the difference is. At this time, cosθ≈1, that is, the closer to 1, the more similar the faces are. In this embodiment, the cosine distance between the face picture sets corresponding to each character object is used as the similarity, and the cosine distance is calculated as follows:

[0079]

[0080] Among them, A and B respectively represent the facial features of picture A and picture B in the same face picture set, i is the i-th feature dimension of the picture, and n is the feature dimension of the picture.

[0081] S203: After obtaining the final face picture set of each object, extract the facial features of each face picture in the final face picture set of each character object respectively, and label each face picture in turn according to the extracted facial features to obtain the face sample set corresponding to each character object, and then obtain the total face sample set of all character objects.

[0082] It should be further explained that the facial features of each face image in the final face image set of each character object are extracted respectively, and each face image is labeled in turn according to the extracted facial features to obtain the face sample set corresponding to each character object, and then the specific process of obtaining the total face sample set of all character objects is as follows:

[0083] The facial pose estimation algorithm, the facial attribute recognition algorithm and the facial region brightness algorithm are respectively used to extract the facial features of each facial image in the final facial image set of each person object. According to the extracted facial features, each facial image is marked as a difficult sample or an easy sample in turn, so as to obtain the facial sample set corresponding to each person object, and then obtain the total facial sample set of all person objects;

[0084] Since the final face image set of each person object may have great differences, such as different shooting scenes in different videos, inconsistent lighting, and variability in face angles (i.e., frontal and side faces), which will affect the quality of the image set. Therefore, the following three strategies are used to classify the images: face pose estimation algorithm, face attribute recognition algorithm, and face area brightness algorithm are used to extract the facial features of each face image in the final face image set of each person object, and each face image is marked as a difficult sample or an easy sample according to the facial features, such as Figure 8 shown.

[0085] Among them, the face posture estimation algorithm is used to extract whether the face of each face image is a front face or a side face; the face posture estimation algorithm is mainly used to estimate the three-dimensional Euler angle of the face of each face image. Posture estimation can be used in many business scenarios. For example, in the face recognition system, posture estimation can assist in the screening of input samples; in some scenarios where the face orientation is required as an important business basis, the face posture algorithm is also indispensable, such as the driver's left and right detection in fatigue driving products. In this embodiment, the image pixels containing only the face in the face of a face image are extracted, and the image containing only the face is preprocessed. After preprocessing, it is input into the face posture estimation model. The face posture estimation model outputs the three-dimensional Euler angle of the face of each face image, and the face type is judged based on the angle, whether it is a front face or a side face.

[0086] Algorithms for face pose estimation can be roughly divided into two categories: one is an algorithm that estimates 3D pose information through 2D calibration information, such as first calculating the key points of the face, then selecting a reference system (the key points of the average frontal face), calculating the transformation matrix of the key points and the reference system, and then estimating the face pose through an iterative optimization algorithm (refer to the SolvePnP algorithm in Opencv); the other is to train a regressor in a data-driven way, and the regressor makes a direct prediction of the input face block.

[0087] The face attribute recognition algorithm is used to extract the gender, expression, whether the face is wearing a mask, and whether the face is wearing sunglasses for each face image; the face area brightness algorithm is used to calculate the average brightness of each face image, and determine whether the face image is a strong light face or a low light face based on the average brightness. The specific process of the face area brightness algorithm is as follows:

[0088] Convert the RGB face image to HSV, extract the brightness V, and calculate the average brightness in the face area. The calculation formula is as follows:

[0089]

[0090] Among them, m represents the total pixel value, v represents the brightness of the face area, face represents the RGB data of the face area, and i represents the i-th pixel. Similarly, you need to pre-set an interval to represent faces under normal lighting, a value less than the lower limit of the interval represents a face under low lighting, and a value greater than the upper limit of the interval represents a face under strong lighting, and mark it in the form of a file name.

[0091] After extracting the features of the face images in the final face image set of each object, if the face images in the final face image set of each person object have one of the features of side face, wearing a mask, wearing sunglasses, face in strong light, and face in low light, then the face image is marked as a difficult sample, otherwise it is marked as an easy sample.

[0092] Thereby, a face sample set corresponding to each character object is obtained, and the sample set corresponding to each character object includes a difficult sample set and an easy sample set, and then the total set of face samples of all character objects is obtained.

[0093] In the process of labeling the face picture of each object, the embodiment of the present invention extracts the features of each face picture through a face posture estimation algorithm, a face attribute recognition algorithm, and a face area brightness algorithm, so that each face picture can be divided into a finer granularity, greatly improving the accuracy of face picture labeling.

[0094] The embodiment of the present invention performs face detection and face tracking on each frame of each short video in the video collection to obtain a first object picture collection corresponding to each short video, merges the face picture collections corresponding to each character object in all short videos according to the similarity between the face picture collections corresponding to each character object in each first object picture collection, obtains a final face picture collection for each character object, extracts and annotates the face features in the final face picture collection of each character object, and obtains a face sample set. The embodiment of the present invention does not require human participation in the process of obtaining the face sample set, merges the face picture collections by calculating the similarity between the face pictures, and extracts the face features to annotate the face pictures of each object, thereby reducing the labor cost of manual annotation and improving the image annotation efficiency.

[0095] Embodiment 3

[0096] To further describe the present invention, Fig. 9 As shown, this embodiment also provides a face sample set acquisition system, including a picture collection generation module 301, a merging module 302 and a labeling module 303;

[0097] The picture collection generation module 301 is used to perform face detection and face tracking on each frame of each short video in the video collection, and obtain a first object picture collection corresponding to each short video; wherein the video collection includes multiple short videos; each first object image collection includes a face picture collection corresponding to each of multiple character objects;

[0098] The merging module 302 is used to merge the face picture sets corresponding to the character objects in all the short videos according to the similarity between the face picture sets corresponding to the character objects in each first object picture collection, and obtain the second object picture collection corresponding to each character object, so as to obtain the final face picture set of each character object; wherein each second object picture collection includes multiple subsets corresponding to the same character object, and each subset contains face pictures corresponding to the same character object;

[0099] The labeling module 303 is used to extract the facial features of each face image in the final face image set of each character object, and label each face image in turn according to the extracted facial features to obtain the face sample set corresponding to each character object, and then obtain the total face sample set of all character objects.

[0100] In order to further describe the present invention, this embodiment also provides a face sample set acquisition device, such as Fig.10 As shown, a face sample set acquisition device 40 includes a processor 400 and a memory 401;

[0101] The memory 401 is used to store program code 402 and transmit the program code 402 to the processor;

[0102] The processor 400 is used to execute the steps in the above-mentioned embodiment of the method for obtaining a face sample set according to the instructions in the program code 402.

[0103] Exemplarily, the computer program 402 may be divided into one or more modules / units, which are stored in the memory 401 and executed by the processor 400 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 402 in the terminal device 40.

[0104] The terminal device 40 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The terminal device may include, but is not limited to, a processor 400 and a memory 401. Those skilled in the art will appreciate that Fig.10 It is only an example of the terminal device 40 and does not constitute a limitation of the terminal device 40. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device may also include input and output devices, network access devices, buses, etc.

[0105] The processor 400 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0106] The memory 401 may be an internal storage unit of the terminal device 40, such as a hard disk or memory of the terminal device 40. The memory 401 may also be an external storage device of the terminal device 40, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 40. Further, the memory 401 may also include both an internal storage unit and an external storage device of the terminal device 40. The memory 401 is used to store the computer program and other programs and data required by the terminal device. The memory 401 may also be used to temporarily store data that has been output or is to be output.

[0107] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0108] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0109] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0110] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0111] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0112] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for acquiring a face sample set, characterized in that: include: Performing face detection and face tracking on each frame of each short video in the video set to obtain a first object image collection corresponding to each short video; wherein the video set includes multiple short videos; each first object image collection includes a face image collection corresponding to multiple human objects; According to the similarity between the face picture sets corresponding to each character object in each first object picture collection, the face picture sets corresponding to each character object in all short videos are merged to obtain the second object picture collection corresponding to each character object, and the final face picture set of each character object is obtained based on the second object picture collection corresponding to each character object; wherein each second object picture collection includes multiple subsets corresponding to the same character object, and each subset contains multiple face pictures corresponding to the same character object; Extract the facial features of each face image in the final face image set of each person object respectively, and label each face image in turn according to the extracted facial features to obtain the face sample set corresponding to each person object, and then obtain the total face sample set of all person objects; According to the similarity between the face picture sets corresponding to each character object in each first object picture collection, the face picture sets corresponding to each character object in all short videos are merged to obtain the second object picture collection corresponding to each character object as follows: Calculate the similarity between the face picture sets corresponding to the various character objects in the first object picture collection of each short video; when the similarity between the face picture set corresponding to any character object and the face picture set corresponding to other character objects is greater than a preset first threshold, merge the face picture sets corresponding to the two character objects; Calculate the similarity between the face picture sets corresponding to each character object in all short videos. When the similarity between the face picture sets corresponding to any two character objects in different short videos is greater than a preset second threshold, merge the face picture sets corresponding to the two character objects as subsets corresponding to the same character object. When all the character objects in all short videos are merged, obtain a second object picture collection corresponding to each character object. The facial features of each face image in the final face image set of each character object are extracted respectively, and each face image is labeled in turn according to the extracted facial features to obtain the face sample set corresponding to each character object, and then the specific process of obtaining the total face sample set of all character objects is as follows: The face pose estimation algorithm, face attribute recognition algorithm and face area brightness algorithm are used respectively to extract the facial features of each face image in the final face image set of each character object. According to the extracted facial features, each face image is marked as a difficult sample or an easy sample in turn, so as to obtain the face sample set corresponding to each character object, and then obtain the total face sample set of all character objects.

2. The method for acquiring a face sample set according to claim 1, characterized in that: The specific process of performing face detection and face tracking on each frame of each short video in the video set to obtain the first object image collection corresponding to each short video is as follows: Perform face detection on each frame in each short video to obtain the face frame of each character in each frame; The facial features of the face frame of each character in each frame of each short video are extracted, and the facial features of each frame in each short video are tracked using a face tracking algorithm. Each frame with the same facial features tracked in the same short video is saved as a face image set of the same character object, and a first object face image collection corresponding to each short video is obtained.

3. The method for acquiring a face sample set according to claim 1, characterized in that: The similarity between the face image sets is the cosine distance between the face image sets.

4. The method for acquiring a face sample set according to claim 3, characterized in that: The specific process of obtaining the final face picture set of each person object based on the second object picture set corresponding to each person object is as follows: A subset with the largest number of face images is selected from the second object image collection corresponding to each person object, and is used as the final face image set for each person object.

5. The method for acquiring a face sample set according to claim 1, characterized in that: The face pose estimation algorithm is used to extract whether the face in each face image is frontal or side profile; the face attribute recognition algorithm is used to extract the gender, expression, whether wearing a mask, and whether wearing sunglasses in each face image; the face area brightness algorithm is used to calculate the average brightness of each face image, and determine whether the face image is a strong light face or a low light face based on the average brightness.

6. The method for acquiring a face sample set according to claim 5, characterized in that: If the face image in the final face image set of each object has one of the characteristics of side face, wearing a mask, wearing sunglasses, face in strong light, and face in low light, then the face image is marked as a difficult sample, otherwise it is marked as an easy sample.

7. A face sample set acquisition system, characterized in that: It includes picture collection generation module, merging module and annotation module; The image collection generation module is used to perform face detection and face tracking on each frame of each short video in the video set, and obtain a first object image collection corresponding to each short video; wherein the video set includes multiple short videos; each first object image collection includes a face image collection corresponding to each of multiple character objects; The merging module is used to merge the face picture sets corresponding to the character objects in all the short videos according to the similarity between the face picture sets corresponding to the character objects in each first object picture collection, and obtain the second object picture collection corresponding to each character object, so as to obtain the final face picture set of each character object; wherein each second object picture collection includes multiple subsets corresponding to the same character object, and each subset contains face pictures corresponding to the same character object; The annotation module is used to extract the facial features of each face image in the final face image set of each person object, and annotate each face image in turn according to the extracted facial features to obtain the face sample set corresponding to each person object, and then obtain the total face sample set of all person objects; According to the similarity between the face picture sets corresponding to each character object in each first object picture collection, the face picture sets corresponding to each character object in all short videos are merged to obtain the second object picture collection corresponding to each character object as follows: Calculate the similarity between the face picture sets corresponding to the various character objects in the first object picture collection of each short video; when the similarity between the face picture set corresponding to any character object and the face picture set corresponding to other character objects is greater than a preset first threshold, merge the face picture sets corresponding to the two character objects; Calculate the similarity between the face picture sets corresponding to each character object in all short videos. When the similarity between the face picture sets corresponding to any two character objects in different short videos is greater than a preset second threshold, merge the face picture sets corresponding to the two character objects as subsets corresponding to the same character object. When all the character objects in all short videos are merged, obtain a second object picture collection corresponding to each character object. The facial features of each face image in the final face image set of each character object are extracted respectively, and each face image is labeled in turn according to the extracted facial features to obtain the face sample set corresponding to each character object, and then the specific process of obtaining the total face sample set of all character objects is as follows: The face pose estimation algorithm, face attribute recognition algorithm and face area brightness algorithm are used respectively to extract the facial features of each face image in the final face image set of each character object. According to the extracted facial features, each face image is marked as a difficult sample or an easy sample in turn, so as to obtain the face sample set corresponding to each character object, and then obtain the total face sample set of all character objects.

8. A face sample set acquisition device, characterized in that: including a processor and a memory; The memory is used to store program code and transmit the program code to the processor; The processor is used to execute a method for acquiring a face sample set according to any one of claims 1 to 6 according to the instructions in the program code.

Citation Information

Patent Citations

  • Human face image identity labeling method and device

    CN106503691A

  • Face recognition method, device and system based on cloud edge fusion

    CN109190532A

  • A method and device for obtaining a target person based on a video

    CN109598223A

  • A video face labeling method and system based on community discovery clustering

    CN109684913A

  • Face recognition method and device, electronic equipment and storage medium

    CN111914649A