Method, apparatus and system for training a face feature extraction model

Abstract

The present application discloses a method, device, and system for training a face feature extraction model, belonging to the technical field of machine learning. The method includes: obtaining a sample face image pair, corresponding face comparison result data, and a gaze heat map of a person during the face comparison corresponding to the face comparison result data; training an attention network in the face feature extraction model based on the sample face image pair and the gaze heat map; and training the face feature extraction model based on the sample face image pair and the corresponding face comparison result data. By separately supervising and training the attention network, the training effect of the face feature extraction model is better in the present application.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and particularly relates to a method, apparatus, and system for training a face feature extraction model. Background Art

[0002] Currently, face comparison technology is widely used in fields such as security and transportation. The commonly used face comparison technology is to use a face feature extraction model to extract features from the captured face image to be compared, and compare it with the feature data of the reference face image in the database to obtain the comparison result. Before using the face feature extraction model, it needs to be trained with a large number of samples.

[0003] The training of the face feature extraction model usually uses a large number of sample face images and the comparison result data between two sample face images to train the face feature extraction model.

[0004] In the process of implementing this application, the inventors found that the prior art has at least the following problems:

[0005] The face feature extraction model includes an attention network. Only using the comparison result data corresponding to the face and the comparison result data between two sample face images to globally train and adjust has a poor training effect on the attention network. Summary of the Invention

[0006] Embodiments of this application provide a method, apparatus, and system for training a face feature extraction model, which can solve the problem of poor training effect on the attention network in the face feature extraction model in related technologies.

[0007] The technical solutions are as follows:

[0008] In a first aspect, a method for training a face feature extraction model is provided. The method includes:

[0009] Obtain sample face image pairs, corresponding face comparison result data, and the gaze heat map of the comparison personnel during the face comparison corresponding to the face comparison result data;

[0010] Based on the sample face image pairs and the gaze heat map, train the attention network in the face feature extraction model;

[0011] Based on the sample face image pairs and the corresponding face comparison result data, train the face feature extraction model.

[0012] In a possible implementation manner,

[0013] Obtaining the sample face image pair, the corresponding face comparison result data, and the gaze heat map of the person during the face comparison process corresponding to the face comparison result data includes:

[0014] During the process of the person comparing the face image pair, obtaining the face behavior video of the person;

[0015] Obtaining the face comparison result data of the person for the face image pair;

[0016] Based on the face behavior video and the gaze estimation model, determining the gaze heat map of the person during the process of comparing the face image pair;

[0017] If the gaze heat map meets the preset entropy condition, then taking the face image pair as the sample face image pair.

[0018] In a possible implementation manner,

[0019] The determining the gaze heat map of the person during the process of comparing the face image pair based on the face behavior video and the gaze estimation model includes:

[0020] For each frame of image in the face behavior video, obtaining the left eye image, the right eye image, and the face image in the image;

[0021] Inputting the left eye image, the right eye image, and the face image into the gaze estimation model to obtain the gaze direction information corresponding to the image;

[0022] Based on the gaze direction information corresponding to each frame of image in the face behavior video, generating the gaze heat map of the person during the process of comparing the face image pair.

[0023] In a possible implementation manner, the if the gaze heat map meets the preset entropy condition, then taking the face image pair as the sample face image pair includes:

[0024] Calculating the entropy of the gaze heat map;

[0025] If the entropy of the gaze heat map is less than the preset threshold, then taking the face image pair as the sample face image pair.

[0026] In a possible implementation manner, before obtaining the face behavior video of the person during the process of the person comparing the face image pair, the method further includes:

[0027] Obtaining a face image pair composed of two face images whose corresponding face feature data meet the preset similarity condition.

[0028] In a possible implementation, obtaining a pair of face images composed of two face images whose corresponding face feature data meet a preset similarity condition includes:

[0029] Obtaining a face image to be compared captured by a monitoring device;

[0030] Performing face feature extraction on the face image to be compared to obtain the face feature data of the face image to be compared;

[0031] If there is target reference face feature data in the reference face feature data of the face feature database that meets the preset similarity condition with the face feature data of the face image to be compared, then obtaining the target reference face image corresponding to the target reference face feature data, and using the face image to be compared and the target reference face image as a pair of face images.

[0032] In a second aspect, there is provided a device for training a face feature extraction model, and the device includes:

[0033] An acquisition module, configured to acquire a pair of sample face images, corresponding face comparison result data, and a gaze heat map of a person during the face comparison process corresponding to the face comparison result data;

[0034] A training module, configured to train an attention network in the face feature extraction model based on the pair of sample face images and the gaze heat map; and train the face feature extraction model based on the pair of sample face images and the corresponding face comparison result data.

[0035] In a possible implementation, the acquisition module is configured to:

[0036] During the process of a person comparing a pair of face images, acquiring a face behavior video of the person;

[0037] Obtaining the face comparison result data of the person for the pair of face images;

[0038] Based on the face behavior video and a gaze estimation model, determining the gaze heat map of the person during the process of comparing the pair of face images;

[0039] If the gaze heat map meets a preset entropy condition, then using the pair of face images as a pair of sample face images.

[0040] In a possible implementation, the acquisition module is configured to:

[0041] For each frame image in the face behavior video, obtaining the left eye image, right eye image, and face image in the image;

[0042] Input the left-eye image, the right-eye image, and the face image into the gaze estimation model to obtain the gaze direction information corresponding to the images;

[0043] Generate a gaze heat map of the comparison person during the face comparison of the face image pair based on the gaze direction information corresponding to each frame image in the face behavior video.

[0044] In a possible implementation, the obtaining module is configured to:

[0045] Calculate the entropy of the gaze heat map;

[0046] If the entropy of the gaze heat map is less than a preset threshold, use the gaze heat map as a sample gaze heat map and the face image pair as a sample face image pair.

[0047] In a possible implementation, the apparatus further includes a comparison module, configured to:

[0048] Obtain a face image pair composed of two face images whose corresponding face feature data meet a preset similarity condition.

[0049] In a possible implementation, the comparison module is configured to:

[0050] Obtain a face image to be compared captured by a monitoring device;

[0051] Extract face features from the face image to be compared to obtain the face feature data of the face image to be compared;

[0052] If there is target reference face feature data in the reference face feature data of the face feature database that meets the preset similarity condition with the face feature data of the face image to be compared, obtain the target reference face image corresponding to the target reference face feature data, and use the face image to be compared and the target reference face image as a face image pair.

[0053] In a third aspect, a system for training a face comparison model is provided. The system includes a management device and a training device, where:

[0054] The management device is configured to obtain a sample face image pair, the corresponding face comparison result data, and the gaze heat map of the comparison person during the face comparison corresponding to the face comparison result data, and send the obtained sample face image pair, the corresponding face comparison result data, and the gaze heat map of the comparison person during the face comparison corresponding to the face comparison result data to the training device;

[0055] The training device is used to train the attention network in the face feature extraction model based on the sample face image pairs and the gaze heat map, and train the face feature extraction model based on the sample face image pairs and the corresponding face comparison result data.

[0056] In a possible implementation manner, the system further includes an image acquisition device, and the image acquisition device is used for:

[0057] During the process of the comparison personnel performing face comparison on the face image pairs, acquiring the face behavior video of the comparison personnel;

[0058] The management device is used to obtain the face comparison result data of the comparison personnel for the face image pairs, determine the gaze heat map of the comparison personnel during the process of performing face comparison on the face image pairs based on the face behavior video and the gaze estimation model, and if the gaze heat map meets the preset entropy condition, use the face image pairs as the sample face image pairs.

[0059] In a possible implementation manner, the management device is used for:

[0060] For each frame of image in the face behavior video, acquiring the left eye image, right eye image and face image in the image;

[0061] Inputting the left eye image, the right eye image and the face image into the gaze estimation model to obtain the gaze direction information corresponding to the image;

[0062] Generating the gaze heat map of the comparison personnel during the process of performing face comparison on the face image pairs based on the gaze direction information corresponding to each frame of image in the face behavior video.

[0063] In a possible implementation manner, the management device is used for:

[0064] Calculating the entropy of the gaze heat map;

[0065] If the entropy of the gaze heat map is less than a preset threshold, using the gaze heat map as the sample gaze heat map and the face image pairs as the sample face image pairs.

[0066] In a possible implementation manner, the management device is further used for:

[0067] Obtaining a face image pair composed of two face images whose corresponding face feature data meet the preset similarity condition.

[0068] In a possible implementation manner, the management device is used for:

[0069] Obtain the face image to be compared captured by the monitoring device;

[0070] Extract the face features of the face image to be compared to obtain the face feature data of the face image to be compared;

[0071] If there is a target reference face feature data in the reference face feature data of the face feature database that satisfies the preset similarity condition with the face feature data of the face image to be compared, obtain the target reference face image corresponding to the target reference face feature data, and use the face image to be compared and the target reference face image as a pair of face images.

[0072] In a fourth aspect, a management device is provided. The decision controller includes a processor and a memory. At least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to implement the method for training the face feature extraction model as described in the first aspect above.

[0073] In a fifth aspect, a computer-readable storage medium is provided. At least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the method for training the face feature extraction model as described in the first aspect above.

[0074] The beneficial effects brought by the technical solution provided by the embodiments of the present application are:

[0075] In the process of training the face feature extraction model provided by the embodiments of the present application, not only the sample face image pairs and the corresponding face comparison result data are used to globally train the face feature extraction model, but also the sample face image pairs and the gaze heat map of the comparison personnel during the face comparison of the sample face image pairs are used to separately train the attention network in the face feature extraction model. In this way, by separately training the attention network in the face feature extraction model, its training effect is better. Description of the Drawings

[0076] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0077] Figure 1 is a flowchart of a method for training a face feature extraction model provided by an embodiment of the present application;

[0078] Figure 2 is a schematic diagram of a gaze estimation model provided by an embodiment of the present application;

[0079] Figure 3 It is a schematic structural diagram of a device for training a face feature extraction model provided by an embodiment of the present application;

[0080] Figure 4 It is a schematic structural diagram of a management device provided by an embodiment of the present application;

[0081] Figure 5 It is a schematic diagram of a system for training a face feature extraction model provided by an embodiment of the present application;

[0082] Figure 6 It is a flowchart of a method for training a face feature extraction model provided by an embodiment of the present application. Detailed implementation manners

[0083] To make the objectives, technical solutions, and advantages of the present application clearer, the following will further describe the embodiments of the present application in detail with reference to the accompanying drawings.

[0084] The embodiment of the present application provides a method for training a face feature extraction model, and this method can be implemented by a management device. In this method, during the process of training the face feature extraction model, the management device not only uses the sample face image pairs and the corresponding face comparison result data to globally train the face feature extraction model, but also uses the sample face image pairs and the gaze heat map of the person during the face comparison process of the sample face image pairs to separately supervise and train the attention network in the face feature extraction model, so that the training effect of the attention network is better.

[0085] Figure 1 It is a flowchart of a method for training a face feature extraction model provided by an embodiment of the present application. Refer to Figure 1 The processing flow of this method may have the following steps:

[0086] Step 101, obtain sample face image pairs, the corresponding face comparison result data, and the gaze heat map of the person during the face comparison corresponding to the face comparison result data.

[0087] In implementation, before the sample face image pairs, high-similarity face image pairs can be obtained first, and the person performs face comparison on the high-similarity faces, and then the gaze tracking result of the person on the face images, such as a gaze heat map, can be obtained.

[0088] The monitoring device can capture images of the monitoring range and send the captured images to the management device. The management device performs face recognition on the images captured by the monitoring device to obtain the face images to be compared, and inputs the face images to be compared into the face feature extraction model to obtain the face feature data corresponding to the face images to be compared.

[0089] Alternatively, the monitoring device can capture images of the monitored area, perform face recognition on the captured images to obtain the face images to be compared, and then send the face images to be compared to the monitoring device. The monitoring device inputs the face images to be compared into the face feature extraction model to obtain the face feature data corresponding to the face images to be compared.

[0090] Then, the management device can obtain the reference face feature data from the face feature database and calculate the similarity between the reference face feature data and the face feature data corresponding to the face images to be compared. The similarity can be represented by Euclidean distance, cosine distance, etc.

[0091] Here, according to different actual usage scenarios, technicians can store the reference face feature data of different objects in the face feature database. For example, the monitoring device can be deployed at the access control of relatively private places such as offices. In these places, only specified objects are allowed to enter. Then, the reference face feature data of these specified objects can be stored in the face feature database. Another example is that the monitoring device can be deployed in public places such as streets, intersections, and shopping malls. Security managers need to trace specified objects. Then, the reference face feature data of these specified objects can be stored in the face feature database.

[0092] In addition, in the face feature database, a reference face image is stored corresponding to each reference face feature data. Correspondingly, the target reference face image corresponding to the reference face feature data with a similarity greater than the preset threshold of the face feature data corresponding to the face image to be compared can be obtained from the face feature database. The target reference face image and the face image to be compared can form a face image pair.

[0093] The face image to be compared and the target reference face image in the face image pair can be displayed side by side on the display screen of the management device to facilitate the comparer to perform face comparison on the face image to be compared and the target reference face image.

[0094] The comparer can manually perform face comparison on the face image pair to determine whether the two face images in the face image pair are of the same object. An image acquisition device can be installed above the display screen of the management device, and the image acquisition device can be a camera. During the process of the comparer manually comparing the face image pair, the image acquisition device can be used to capture the face of the comparer to obtain the face behavior video of the comparer during the process of face comparison of the face image pair.

[0095] On the display screen of the management device, options for starting face comparison and ending face comparison can also be displayed. When the person to be compared starts face comparison, they can select the option for starting face comparison, and then the management device will activate the image acquisition device to start taking pictures of the person to be compared. When the person to be compared finishes face comparison, they can select the option for ending face comparison, and then the management device will control the image acquisition device to stop taking pictures of the face of the person to be compared. In this way, a face behavior video of the person to be compared during the face comparison of the face image pair can be obtained.

[0096] After the person to be compared selects the option for ending face comparison, a face comparison result selection interface can also be displayed on the display screen of the management device. In this face comparison result selection interface, a first option for indicating that the two images in the face image pair are of the same object and a second option for indicating that the two images in the face image pair are of different objects can be displayed. The person to be compared can select the first option or the second option according to the face comparison result they obtain. The specific display effects of the first option and the second option are not limited in the embodiments of the present application. If the person to be compared selects the first option, it means that the person to be compared believes that the face image to be compared and the target reference face image in the face image pair are of the same object. If the selection is different, it means that the person to be compared believes that the face image to be compared and the target reference face image in the face image pair are of different objects.

[0097] After the person to be compared finishes face comparison of the face image pair, a gaze heat map of the person to be compared during the face comparison of the face image pair can be determined according to the face behavior video obtained by the camera and the pre-trained gaze estimation model. Specifically, the method for determining the gaze heat map can be as follows:

[0098] Perform face recognition on each frame of the face behavior video to obtain the face image of the person to be compared in each frame. Then, perform face feature point localization on the face image of the person to be compared in each frame, and crop the left eye image, right eye image, and face image according to the feature points. Then, the left eye image, right eye image, and face image can be input into the gaze estimation model.

[0099] The gaze estimation model may include three branch networks, a first branch network for processing the left-eye image, a second branch network for processing the right-eye image, and a third branch network for processing the face image. Among them, the first branch network and the second branch network may be the same. The first branch network and the second branch network are mainly composed of Conv (Convolution) layers, RES blocks (residual blocks), and pool (pooling) layers. The third branch network is mainly composed of Conv layers, pooling layers, and Norm (Normalization) layers. Since the third branch network needs to process the face image, in order to better perceive the global information of the face, the convolutional kernels in the third branch network can be relatively larger than those in the first branch network and the second branch network. In this gaze estimation model, in addition to the above three branch networks, there are also a concat (concatenation) layer and an FC (fully connected) layer.

[0100] When in use, the left-eye image can be input into the first branch network, the right-eye image can be input into the second branch network, and the face image can be input into the third branch network, and this gaze estimation model can output the corresponding gaze direction information. This gaze direction information includes pitch (vertical deflection angle) and yaw (horizontal deflection angle). In this way, for each frame of the face behavior video of the comparison person, which includes an image of the comparison person's face, the corresponding gaze direction information can be obtained.

[0101] As Figure 2 shown is a possible gaze estimation model. In this gaze estimation model, the pooling layers in the first branch network and the second branch network are avgpooling (average pooling) layers. After the first branch network, the second branch network, and the third branch network, there are a concat layer, an FC1 layer, and an FC2 layer.

[0102] Then, a preset three-dimensional face model can be used to estimate the three-dimensional coordinates (X, Y, Z) of the center point of the comparison person's eye corners in the camera coordinate system of the camera. Here, the center point of the eye corners is roughly located at the middle position between the two eyes. It is also possible to determine the plane where the display screen of the above management device is located in the camera coordinate system of the camera. Since the positions of the management device and the camera are relatively fixed, then, the plane where the display screen of the management device is located in the camera coordinate system of the camera can be determined in advance before the comparison person conducts a face comparison.

[0103] For each frame in the face behavior video, which includes an image pair of the face of the person to be compared along with the line-of-sight direction information, a ray can be drawn starting from the three-dimensional coordinates (X, Y, Z) of the center point of the eye corner in the camera coordinates along the implementation direction corresponding to the line-of-sight direction information, and the intersection point of the plane where the display screen of the management device is located in the camera coordinate system and the ray can be obtained, and the intersection point coordinates (x m ,y m ,x m ) are recorded. This intersection point can be considered as the attention point of the person to be compared on the display screen. In this way, for each frame in the face behavior video that includes an image pair of the face of the person to be compared, the corresponding intersection point coordinates can be obtained. Thus, based on the intersection point coordinates corresponding to each frame image, a line-of-sight heat map of the person to be compared during the comparison process of the face image pair can be obtained. The value of each pixel point in the line-of-sight heat map reflects the frequency at which the corresponding pixel points in the face image to be compared and the target reference face image are concerned by the person to be compared. In this line-of-sight heat map, a first line-of-sight heat map corresponding to the face image to be compared and a second line-of-sight heat map corresponding to the target reference face image can be obtained.

[0104] Finally, the above face image pair can be used as a sample face image pair, and the sample face image pair, the corresponding face comparison result data, the first line-of-sight heat map, and the second line-of-sight heat map are packaged as a set of training samples.

[0105] In a possible implementation manner, before using the face image pair as a sample face image pair, it can be first calculated whether the entropy of the line-of-sight heat map corresponding to the sample image pair meets a preset condition. If it meets the condition, the face image pair can be used as a sample face image.

[0106] In implementation, the entropy of the first line-of-sight heat map and the entropy of the second line-of-sight heat map can be calculated respectively. If the entropy of the first line-of-sight heat map and the entropy of the second line-of-sight heat map are both less than a preset threshold, the face image pair can be used as a sample face image, and the sample face image pair, the corresponding face comparison result data, the first line-of-sight heat map, and the second line-of-sight heat map are packaged as a set of training samples.

[0107] The calculation formula for the entropy of the line-of-sight heat map can be as follows:

[0108] [[ID=..]]

[0109] where x i is the value of the i-th pixel point in the line-of-sight heat map, and n is the number of pixel points in the line-of-sight heat map.

[0110] Step 102: Based on the sample face image pair and the line-of-sight heat map, train the attention network in the face feature extraction model.

[0111] In implementation, an attention network can be included in the feature extraction network, and the attention network can be in a higher layer of the face feature extraction model.

[0112] For the training of both the face feature extraction model and the attention network in the face feature extraction model, a large number of training samples are required. Then, training can be started after the number of groups of the obtained training samples reaches a preset quantity threshold.

[0113] For each group of training samples, when training the attention network in the face feature extraction model, the face image to be compared and the first line-of-sight heat map therein can be used as the first training sample pair, and the target reference face image and the second line-of-sight heat map can be used as the second training sample pair.

[0114] For the first training sample pair, the face image to be compared is input into the feature extraction network, and the predicted attention heat map output by the attention network is obtained. Then, the loss value L of the attention network is calculated. attention 。

[0115] The loss value L of the attention network attention can use L1 loss (L1 loss value), and the calculation formula can be as follows:

[0116]

[0117] where a m is the value of the m-th pixel point in the predicted line-of-sight heat map output by the attention network, b m is the value of the m-th pixel point in the first line-of-sight heat map, and N is the number of pixel points in the predicted line-of-sight heat map or the first line-of-sight heat map output by the attention network.

[0118] Similarly, for the second sample pair, the loss value of the attention network when the target reference face image is input can be calculated. For each training sample pair of each group of training samples obtained, the corresponding loss value can also be calculated. According to the calculated loss value, the values of the adjustable parameters in the attention network can be adjusted, and the adjustment direction is the direction in which the loss value decreases.

[0119] It should be noted here that the sizes of the predicted line-of-sight heat map, the first line-of-sight heat map, and the second line-of-sight heat map may be different. Since the size of the predicted line-of-sight heat map is known, after obtaining the first line-of-sight heat map and the second line-of-sight heat map in step 101, the first line-of-sight heat map and the second line-of-sight heat map can be scaled by the bilinear interpolation method to the same size as the predicted line-of-sight heat map, and then packed together with the face image pair and the comparison result data as training samples. That is, the first line-of-sight heat map and the second line-of-sight heat map used in this step 102 have been scaled. Alternatively, it is also possible not to scale the first line-of-sight heat map and the second line-of-sight heat map, but after obtaining the predicted line-of-sight heat map, first scale the predicted line-of-sight heat map by the bilinear interpolation method to the same size as the first line-of-sight heat map or the second line-of-sight heat map, and then calculate the loss value of the attention network.

[0120] Step 103: Train the face feature extraction model based on the sample face image pairs and the corresponding face comparison result data.

[0121] In implementation, after the number of obtained training sample groups reaches the preset quantity threshold, the training of the face feature extraction model can be started. When training the face feature extraction model, the face image to be compared and the target reference face image can be respectively input into the face feature extraction model to obtain the first face feature data corresponding to the face image to be compared and the second face feature data corresponding to the target reference face image. Then, the similarity between the first face feature data and the second face feature data can be calculated. If the similarity is higher than the threshold, the predicted face comparison result data is output as 1. If the similarity is lower than the threshold, the predicted face comparison result data is output as 0.

[0122] Then, according to the predicted face comparison result data corresponding to the sample image pair, the face comparison result data in the training sample, the first face feature data, and the second face feature data, calculate the loss value L of the face feature extraction model contrastive . This loss value L contrastive can be calculated using the commonly used contrastive loss (comparison loss). Based on the calculated loss value, the adjustable parameters in the face feature extraction model can be adjusted, and the adjustment direction is the direction in which the loss value decreases.

[0123] Based on the loss value L of the face feature extraction model contrastive Adjust the parameters in the face feature extraction model, which can be executed after adjusting the parameters in the attention network based on the loss value L of the attention network attention or can be executed before adjusting the parameters in the attention network based on the loss value L of the attention network attention The embodiment of the present application does not limit the execution order of the two.

[0124] In a possible implementation, after the face feature extraction model and the attention network are trained, the trained face feature extraction model can be updated to the management device, so that when the management device performs face feature extraction subsequently, it can be implemented using the updated face feature extraction model.

[0125] In addition, when the face feature extraction model in the management device is continuously updated, the reference face feature data in the face feature database can also be updated regularly. When updating the reference face feature data in the face feature database, the updated face feature extraction model deployed in the management device can be used to extract features from the reference face images to obtain the corresponding face feature data, and the corresponding reference face feature data in the face feature database can be replaced with the face feature data obtained at this time.

[0126] In the process of training the face feature extraction model provided by the embodiments of the present application, not only the sample face image pairs and the corresponding face comparison result data are used to globally train the face feature extraction model, but also the sample face image pairs and the gaze heat map of the comparison personnel during the face comparison of the sample face image pairs are used to separately train the attention network in the face feature extraction model. In this way, by separately training the attention network in the face feature extraction model, its training effect is better. All the above optional technical solutions can be combined arbitrarily to form the optional embodiments of the present disclosure, which will not be elaborated here one by one.

[0127] In the optional embodiment of the present application, when the similarity between two face pictures is too high, the target face picture can be selected from the two face pictures by the gaze tracking of the comparison personnel.

[0128] See Figure 6 , another method for training a face feature extraction model provided by the embodiments of the present application. This method can be implemented by a management device, and this method may include the following processing procedures:

[0129] Step 601, obtain a pair of face images composed of two face images whose corresponding face feature data meet the preset similarity condition.

[0130] In implementation, two face images whose corresponding face feature data meet the preset similarity condition can both be face images obtained from a database. For example, the management device can obtain face images in pairs from multiple face images in the database, extract the respective face feature data of the two obtained face images for similarity calculation. If the calculated similarity is greater than the preset threshold, it can be considered that these two face images meet the preset similarity condition, form a pair of face images, and continue with the processing of step 602.

[0131] In addition, one of the two face images whose corresponding face feature data meet the preset similarity condition can be a face image captured in real time by a monitoring device, and the other can be a reference face image obtained from the database. In this case, the method for obtaining the pair of face images is the same as the method for obtaining the pair of face images in step 101 above, and will not be elaborated here.

[0132] Step 602: During the process of a comparison person performing a face comparison on the above-mentioned pair of face images, obtain the face behavior video of the comparison person, and obtain the face comparison result data of the comparison person for the pair of face images.

[0133] Step 603: Based on the face behavior video and the gaze estimation model, determine the gaze heat map of the comparison person during the process of performing a face comparison on the pair of face images.

[0134] Step 604: If the above-mentioned gaze heat map meets the preset entropy condition, then use the above-mentioned pair of face images as a sample pair of face images.

[0135] Step 605: Obtain the sample pair of face images, the corresponding face comparison result data, and the gaze heat map of the comparison person during the face comparison corresponding to the face comparison result data.

[0136] Step 606: Based on the sample pair of face images and the gaze heat map, train the attention network in the face feature extraction model.

[0137] Step 607: Based on the sample pair of face images and the corresponding face comparison result data, train the face feature extraction model.

[0138] It should be noted that the specific implementation manners of the above steps 602 - 607 are the same as or similar to the corresponding specific implementation manners of the above steps 101 - 103, and will not be elaborated here.

[0139] See Figure 3 , based on the same technical concept, the embodiment of the present application also provides a device for training a face feature extraction model, and the device includes:

[0140] An acquisition module 310, configured to acquire a sample face image pair, corresponding face comparison result data, and a gaze heat map of a person during the face comparison process corresponding to the face comparison result data;

[0141] A training module 320, configured to train an attention network in a face feature extraction model based on the sample face image pair and the gaze heat map; and train the face feature extraction model based on the sample face image pair and the corresponding face comparison result data.

[0142] In a possible implementation manner, the acquisition module 310 is configured to:

[0143] During the process of a person comparing a face image pair, acquire a face behavior video of the person;

[0144] Acquire face comparison result data of the person for the face image pair;

[0145] Based on the face behavior video and a gaze estimation model, determine a gaze heat map of the person during the process of comparing the face image pair;

[0146] If the gaze heat map meets a preset entropy condition, use the face image pair as a sample face image pair.

[0147] In a possible implementation manner, the acquisition module 310 is configured to:

[0148] For each frame of image in the face behavior video, acquire a left eye image, a right eye image, and a face image in the image;

[0149] Input the left eye image, the right eye image, and the face image into the gaze estimation model to obtain gaze direction information corresponding to the image;

[0150] Based on the gaze direction information corresponding to each frame of image in the face behavior video, generate a gaze heat map of the person during the process of comparing the face image pair.

[0151] In a possible implementation manner, the acquisition module 310 is configured to:

[0152] Calculate the entropy of the gaze heat map;

[0153] If the entropy of the gaze heat map is less than a preset threshold, use the gaze heat map as a sample gaze heat map and use the face image pair as a sample face image pair.

[0154] In a possible implementation manner, the device further includes a comparison module, configured to:

[0155] Obtain a pair of face images composed of two face images whose corresponding face feature data meet a preset similarity condition.

[0156] In a possible implementation manner, the comparison module is configured to:

[0157] Obtain a face image to be compared captured by a monitoring device;

[0158] Extract face features from the face image to be compared to obtain the face feature data of the face image to be compared;

[0159] If there is target reference face feature data in the reference face feature data of the face feature database that meets the preset similarity condition with the face feature data of the face image to be compared, obtain the target reference face image corresponding to the target reference face feature data, and use the face image to be compared and the target reference face image as a pair of face images.

[0160] It should be noted that: when the device for training the face feature extraction model provided in the above embodiment performs face feature extraction, only the division of the above functional modules is used for illustration. In actual applications, the above functions can be assigned to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device for training the face feature extraction model provided in the above embodiment and the method embodiment for training the face feature extraction model belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0161] Figure 4 It is a schematic structural diagram of a management device provided by an embodiment of the present application. The management device 400 includes a processor 401 and a memory 402.

[0162] The processor 401 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. The processor 401 may be implemented in at least one of the following hardware forms: DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 401 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the wake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 401 may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 401 may further include an AI (Artificial Intelligence) processor, and the AI processor is used to process computational operations related to machine learning.

[0163] The memory 402 may include one or more computer-readable storage media, and the computer-readable storage media may be non-transitory. The memory 402 may further include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. In some embodiments, the non-transitory computer-readable storage media in the memory 402 is used to store at least one instruction, and the at least one instruction is used to be executed by the processor 401 to implement the method for training a face feature extraction model provided in the method embodiments of the present application.

[0164] In some embodiments, the management device 400 may further optionally include: a peripheral device interface 403 and at least one peripheral device. The processor 401, the memory 402, and the peripheral device interface 403 may be connected through a bus or signal lines. Each peripheral device may be connected to the peripheral device interface 403 through a bus, signal lines, or a circuit board. Specifically, the peripheral device includes: a touch display screen 404.

[0165] The peripheral device interface 403 can be used to connect at least one I / O (Input / Output) related peripheral device to the processor 401 and the memory 402. In some embodiments, the processor 401, the memory 402, and the peripheral device interface 403 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 401, the memory 402, and the peripheral device interface 403 can be implemented on separate chips or circuit boards, and this embodiment does not limit this.

[0166] The display screen 404 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 404 is a touch display screen, the display screen 404 also has the ability to collect touch signals on or above the surface of the display screen 404. The touch signal can be input to the processor 401 as a control signal for processing. At this time, the display screen 404 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there can be one display screen 404, which is arranged on the front panel of the management device 400; in some other embodiments, there can be at least two display screens 404, which are respectively arranged on different surfaces of the management device 400 or in a foldable design; in still some other embodiments, the display screen 404 can be a flexible display screen, which is arranged on the curved surface or the folding surface of the management device 400. Even, the display screen 404 can also be set as an irregular non-rectangular shape, that is, a special-shaped screen. The display screen 404 can be prepared using materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0167] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, and the above instructions can be executed by a processor in the management device to complete the method of face feature extraction in the above embodiments. The computer-readable storage medium can be non-transitory. For example, the computer-readable storage medium can be a ROM (Read-Only Memory), a RAM (Random Access Memory), a CD-ROM (Compact Disc Read-Only Memory), magnetic tape, a floppy disk, and an optical data storage device, etc.

[0168] See Figure 5, this application also provides a system for training a face feature extraction model. The system includes a management device and a training device. Among them, the management device can be a notebook, a desktop computer, etc., and the training device can be a server. In the above Figure 1 In the face feature extraction model training method shown, the acquisition of the gaze heat map, the training of the face feature extraction model, and the training of the attention network in the face feature extraction model are all implemented by the management device. Considering the performance of the management device, the training of the face feature extraction model and the training of the attention network in the face feature extraction model can be implemented by a separate training device. In this system, the processing performed by the management device and the training device can be as follows:

[0169] The management device is used to: obtain a sample face image pair, the corresponding face comparison result data, and the gaze heat map of the person being compared during the face comparison corresponding to the face comparison result data, and send the obtained sample face image pair, the corresponding face comparison result data, and the gaze heat map of the person being compared during the face comparison corresponding to the face comparison result data to the training device. Specifically, it can implement the processing performed by the management device in step 101 above, and the processing performed by the management device in each alternative implementation of step 101.

[0170] The training device is used to: train the attention network in the face feature extraction model based on the sample face image pair and the gaze heat map, and train the face feature extraction model based on the sample face image pair and the corresponding face comparison result data. Specifically, it can implement the processing performed by the management device in step 102 and step 103 above, and the processing performed by the management device in each alternative implementation of step 102 and step 103.

[0171] Those of ordinary skill in the art can understand that all or part of the steps to implement the above embodiments can be completed by hardware, or can be completed by a program instructing relevant hardware. The program can be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk, an optical disc, etc.

[0172] The above are only alternative embodiments of this application and are not intended to limit this application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application shall be included within the protection scope of this application.

Claims

1. A method for training a face feature extraction model, characterized in that, The method includes: During the process of a comparison operator performing a face comparison on a pair of face images, obtaining a face behavior video of the comparison operator; Obtaining face comparison result data of the comparison operator for the pair of face images; Based on the face behavior video and a gaze estimation model, determining a gaze heat map of the comparison operator during the process of performing a face comparison on the pair of face images, wherein the value of each pixel point in the gaze heat map is used to reflect the frequency at which the corresponding pixel point in each face image included in the pair of face images is focused on by the comparison operator; If the gaze heat map meets a preset entropy condition, then using the pair of face images as a sample pair of face images; Based on the sample pair of face images and the gaze heat map, training an attention network in a face feature extraction model; Based on the sample pair of face images and the corresponding face comparison result data, training the face feature extraction model.

2. The method according to claim 1, characterized in that, The determining, based on the face behavior video and the gaze estimation model, a gaze heat map of the comparison operator during the process of performing a face comparison on the pair of face images includes: For each frame image in the face behavior video, obtaining the left eye image, the right eye image, and the face image in the frame image; Inputting the left eye image, the right eye image, and the face image into the gaze estimation model to obtain gaze direction information corresponding to the frame image; Based on the gaze direction information corresponding to each frame image in the face behavior video, generating a gaze heat map of the comparison operator during the process of performing a face comparison on the pair of face images.

3. The method according to claim 1, characterized in that, The if the gaze heat map meets a preset entropy condition, then using the pair of face images as a sample pair of face images includes: Calculating the entropy of the gaze heat map; If the entropy of the gaze heat map is less than a preset threshold, then using the pair of face images as a sample pair of face images.

4. The method according to any one of claims 1-3, characterized in that, Before the obtaining, during the process of a comparison operator performing a face comparison on a pair of face images, a face behavior video of the comparison operator, the method further includes: Obtaining a pair of face images composed of two face images whose corresponding face feature data meet a preset similarity condition.

5. The method according to claim 4, wherein The obtaining a pair of face images composed of two face images whose corresponding face feature data meet a preset similarity condition includes: Obtaining a face image to be compared captured by a monitoring device; Performing face feature extraction on the face image to be compared to obtain face feature data of the face image to be compared; If there is target reference face feature data in the reference face feature data of a face feature database that meets a preset similarity condition with the face feature data of the face image to be compared, then obtaining a target reference face image corresponding to the target reference face feature data, and using the face image to be compared and the target reference face image as a pair of face images.

6. An apparatus for training a face feature extraction model, characterized in that, The device includes: An acquisition module, configured to acquire the face behavior video of the comparison person during the process of the comparison person performing face comparison on a pair of face images; acquire the face comparison result data of the comparison person for the pair of face images; determine the gaze heat map of the comparison person during the process of performing face comparison on the pair of face images based on the face behavior video and a gaze estimation model, wherein the value of each pixel point in the gaze heat map is used to reflect the frequency at which the corresponding pixel point in each face image included in the pair of face images is concerned by the comparison person; if the gaze heat map meets a preset entropy condition, then use the pair of face images as a sample pair of face images; A training module, configured to train the attention network in a face feature extraction model based on the sample pair of face images and the gaze heat map; train the face feature extraction model based on the sample pair of face images and the corresponding face comparison result data.

7. A system for training a face comparison model, characterized in that, The system includes a management device, a training device, and an image acquisition device, wherein: The image acquisition device is configured to acquire the face behavior video of the comparison person during the process of the comparison person performing face comparison on a pair of face images; The management device is configured to acquire the face comparison result data of the comparison person for the pair of face images; determine the gaze heat map of the comparison person during the process of performing face comparison on the pair of face images based on the face behavior video and a gaze estimation model, wherein the value of each pixel point in the gaze heat map is used to reflect the frequency at which the corresponding pixel point in each face image included in the pair of face images is concerned by the comparison person; if the gaze heat map meets a preset entropy condition, then use the pair of face images as a sample pair of face images; send the acquired sample pair of face images, the corresponding face comparison result data, and the gaze heat map of the comparison person during the face comparison corresponding to the face comparison result data to the training device; The training device is configured to train the attention network in a face feature extraction model based on the sample pair of face images and the gaze heat map, and train the face feature extraction model based on the sample pair of face images and the corresponding face comparison result data.

Images