A high-precision face matching method and system based on deep learning driving

Abstract

The application relates to a high-precision face matching method and system based on deep learning driving, sample data is collected and preprocessed, sample data pairs are expanded by a sample expansion model until preset conditions are met, the processed sample data pairs are normalized according to rules, the processed sample data pairs are input into a matching model for training, and the trained matching model is used for face matching; one or more image acquisition modules are used to acquire to-be-matched images, a preprocessing module is used to normalize the to-be-matched images according to the rules, a face feature extraction, matching and identification are performed on the face feature extraction, matching and identification by the face matching module, and a feedback module is used to feed back matching results. The application can orderly and efficiently expand a data set, reduce the training difficulty of a matching model, effectively train the recognition of synthetic photos and videos, improve the recognition accuracy and robustness, better prevent fraud attacks, be more suitable for security, access control and financial fields, have low computing resource demand, high real-time performance and guaranteed accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image or video recognition or understanding, and in particular to a high-precision face matching method and system based on deep learning driving. BACKGROUND

[0002] Face recognition is a technology that identifies identity based on facial feature information. It usually collects images or video streams containing faces for face detection and tracking to achieve face recognition. With the rapid development of information technology and the increasing demand for security in society, face recognition technology, as a convenient and efficient biometric identification method, has been widely used in security monitoring, access control, financial transactions and other fields.

[0003] Early face recognition systems mainly rely on manually designed feature extraction methods such as Local Binary Pattern (LBP), Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA). Although these methods can achieve certain results under certain conditions, their performance significantly decreases when facing complex environmental changes such as light changes, posture differences and occlusions. At the same time, the countermeasures for face recognition are constantly changing, and the original face recognition methods and systems are difficult to meet the needs of practical applications. In recent years, the development of deep learning technology, especially Convolutional Neural Networks (CNN), has provided a new way to solve these problems. Methods based on deep learning can automatically learn more rich and discriminative feature representations from large amounts of data, greatly improving the accuracy and robustness of face recognition. VGGNet, as a classic CNN architecture, has a simple and effective structure design, and has performed well in image classification, object detection and other computer vision tasks, and has been widely used in face recognition.

[0004] Although face recognition systems based on deep learning have made significant progress, they still face some challenges, including:

[0005] (1) Strong data dependence; the training of high-quality models requires the support of large-scale labeled data sets, and the cost of data acquisition and labeling is high, especially in cases involving privacy protection, data collection becomes more difficult;

[0006] (2) High demand for computing resources; deep learning models usually require a large amount of computing resources for training and inference, and high-precision models often contain millions or even hundreds of millions of parameters, which puts high requirements on hardware devices, limiting their application in resource-constrained environments;

[0007] (3) The balance between real-time and accuracy is difficult; to achieve real-time, many application scenarios need to complete the face matching process within a limited time, which may lead to a decrease in recognition accuracy. How to ensure high accuracy while improving the real-time response capability of the system is a problem to be solved;

[0008] (4) The impact on security; traditional face recognition systems are vulnerable to spoofing attacks, such as using photos or videos to impersonate legitimate users. In addition to integrating liveness detection technology to enhance the security of the system, it is particularly important to better identify corresponding synthetic photos, videos, etc. SUMMARY

[0009] The present application solves the problems existing in the prior art and provides a high-precision face matching method and system based on deep learning driving.

[0010] The technical scheme adopted by the present application is a high-precision face matching method based on deep learning driving, which comprises the following steps:

[0011] S1, collect sample data and preprocess to obtain sample data pairs;

[0012] S2, establish a sample expansion model to expand the sample data pairs until the preset conditions are met;

[0013] S3, normalize the sample data pairs processed in S2 according to the established rules;

[0014] S4, input the sample data pairs processed in S3 into the matching model for training;

[0015] S5, use the trained matching model for face matching.

[0016] Preferably, in S1, collect sample data, clean invalid data; assign a unique identification code to the cleaned sample data based on user information;

[0017] Establish a mixed positive sample data pair and a mixed negative sample data pair.

[0018] Preferably, the cleaned sample data is randomly matched, and the matching threshold values a and b are set, a> b> 0;

[0019] The sample data pair with a matching similarity result greater than the threshold value a and a unique identification code different from each other is a fixed negative sample data pair;

[0020] The sample data pair with a matching similarity result less than the threshold value b and a unique identification code the same is a fixed positive sample data pair;

[0021] Randomly extract a plurality of sample data pairs with a matching similarity less than threshold α and greater than threshold β as random negative sample data pairs and random positive sample data pairs;

[0022] The fixed positive sample data pairs and the random positive sample data pairs are used as mixed positive sample data pairs, and the fixed negative sample data pairs and the random negative sample data pairs are used as mixed negative sample data pairs.

[0023] Preferably, the mixed negative sample data pairs further include screening negative sample data pairs established based on associated user information after extraction of the user information.

[0024] Preferably, the sample expansion model includes a generator and a discriminator connected in sequence, and the generator includes a feature extraction module and a synthetic feature generation module connected in sequence.

[0025] Preferably, the feature extraction module includes a plurality of convolutional blocks and a plurality of fully connected layers connected in sequence, and any convolutional block outputs corresponding features and results;

[0026] The synthetic feature generation module includes a plurality of synthetic blocks connected in sequence, any synthetic block inputs the results of the previous stage, corresponding features output by the corresponding convolutional block, and random noise, and performs up-sampling.

[0027] Preferably, the sample data pairs processed by S2 include the same number of mixed positive sample data pairs and mixed negative sample data pairs.

[0028] Preferably, in S3, the normalized sample data pairs include sequentially performing uniform sample size, performing center cropping, uniform sample format, and normalization processing on the sample data.

[0029] Preferably, in S4, the matching model includes a VGG16, a similarity calculation module, and a binary classification module connected in sequence.

[0030] A high-precision face matching system based on deep learning driving, the system comprises:

[0031] One or more image acquisition modules for acquiring images to be matched;

[0032] A preprocessing module for normalizing the images to be matched according to the constructed rules;

[0033] A face matching module using the high-precision face matching method based on deep learning driving for feature extraction and matching recognition of the face;

[0034] A feedback module corresponding to the image acquisition module for feeding back the matching results of the face matching module.

[0035] The present application relates to a kind of high-precision face matching method and system based on deep learning driven, sample data is collected and preprocessed, after obtaining sample data pair, sample expansion model is established, sample data pair is expanded until meeting preset condition;With the rule of construction, the sample data pair after processing is normalized, the sample data pair after processing is input into matching model for training, and the matching model after training is used for face matching;System with one or more image acquisition modules collects image to be matched, with pre-processing module, the image to be matched is normalized according to the rule of construction, with the face matching module of the method, the face is extracted, matched and identified, and finally the matching result of face matching module is fed back by feedback module.

[0036] The beneficial effects of the present application are:

[0037] (1) the sample expansion model is established by expanding the sample acquisition method, which solves the high demand for data and high dependence on labeling of high-quality model training, can expand the data set in order and efficiently, thereby reducing the training difficulty of matching model, and can effectively train the recognition of synthetic photos and videos;

[0038] (2) through multi-level feature learning and large-scale data training, the method and system can effectively deal with light change, posture difference and occlusion problem, improve recognition accuracy and robustness, and can better prevent fraud attacks on the basis of integrated live detection, more suitable for security, access control and financial fields;

[0039] (3) the sample expansion model and matching model are separated, and the classic VGGNet model is used for actual matching training, testing and application, which solves the problems of low recognition accuracy, poor robustness and insufficient security in the prior art, has low training difficulty, low computing resource demand, strong real-time performance and guaranteed accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 The method flowchart of the present application is shown in the figure;

[0041] Figure 2 The structural schematic diagram of the sample expansion model of the present application is shown in the figure;

[0042] Figure 3 The structural schematic diagram of the matching model of the present application is shown in the figure;

[0043] Figure 4 The system structure diagram of the present application is shown in the figure. DETAILED DESCRIPTION

[0044] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0045] As shown in Figure 1 The present application relates to a high-precision face matching method based on deep learning driving, which comprises the following steps:

[0046] S1, collecting sample data and preprocessing to obtain sample data pairs;

[0047] S2, establishing a sample expansion model to expand the sample data pairs until the preset conditions are met;

[0048] S3, normalizing the sample data pairs processed by S2 with the constructed rules;

[0049] S4, inputting the sample data pairs processed by S3 into a matching model for training;

[0050] S5, using the trained matching model for face matching.

[0051] The method content will be described below in combination with the specific embodiments.

[0052] S1, collecting sample data and preprocessing to obtain sample data pairs;

[0053] In this embodiment, the data set used includes the FIW (Families in the Wild) data set, the KinFaceW-I data set and the KinFaceW-II data set; in particular, the FIW data set contains a large amount of data, which is grouped by person and grouped by family, and considering the problem of misrecognition of close relatives in face recognition, the existence of these sample data can help the present method to better train the easily mistaken faces;

[0054] In the implementation process, for the collected sample data, it is necessary to clean the invalid data. Here, the sample data is mainly image data, so some sample data with obvious unidentifiable features, damaged images or basic information are screened out.

[0055] Based on the user information, a unique identification code is assigned to the cleaned sample data, generally using the user ID, and the main purpose is to use it as a label for comparison. In actual application, the coding of the unique identification code follows a preset rule, such as assigning associated unique identification codes to members of the same family group.

[0056] The mixed positive sample data pairs and the mixed negative sample data pairs are established.

[0057] In the present application, the mixed positive sample data pairs and the mixed negative sample data pairs can effectively expand the amount of basic sample data, the positive sample data pair refers to two image samples of the same face (self image), and the negative sample data pair refers to two image samples of different faces (non-self image);

[0058] In the implementation process of the present method, the mixed negative sample data pair includes a fixed negative sample data pair, a random negative sample data pair and a screened negative sample data pair;

[0059] The mixed positive sample data pair includes a fixed positive sample data pair, a random positive sample data pair and a positive sample data pair expanded by a sample expansion model;

[0060] Finally, the number of the mixed positive sample data pairs and the mixed negative sample data pairs is equal.

[0061] S1.1, fixing of the fixed positive sample data pair and the fixed negative sample data pair;

[0062] Random matching is performed on the cleaned sample data, and the matching refers to machine matching of the cleaned sample data, such as feature extraction by VGGNet and calculation of the included angle of the feature vectors. In the specific implementation of the present application, double matching thresholds, α and β, are set, and α>β>0;

[0063] If there is a sample data pair with a matching similarity result greater than the threshold α and different unique identification codes, it is indicated that the two faces are very similar, and the sample data pair needs to be trained as a negative sample and the parameters of the matching model need to be adjusted, so the sample data pair is taken as a fixed negative sample data pair;

[0064] If there is a sample data pair with a matching similarity result less than the threshold β and the same unique identification code, it is indicated that it may be a face recognition error of the same individual under different environmental parameters or objective factors, and the sample data pair needs to be trained as a positive sample and the parameters of the matching model need to be adjusted, so the sample data pair is taken as a fixed positive sample data pair.

[0065] S1.2, acquisition of the random negative sample data pair and the random positive sample data pair;

[0066] Random matching is performed on the cleaned sample data, and after the fixed positive sample data pair and the fixed negative sample data pair are removed, a plurality of sample data pairs with a matching similarity result less than the threshold α and greater than the threshold β are randomly extracted from the sample pool, which are the random negative sample data pair and the random positive sample data pair;

[0067] S1.3, acquisition of the screened negative sample data pair;

[0068] The mixed negative sample data pair further comprises screening negative sample data pairs extracted from user information and based on screening negative sample data pairs established by associated users, that is, the aforementioned "allocating an associated unique identification code to members of the same family group", and combining face data of different members randomly extracted from the same family group into screening negative sample data pairs.

[0069] After the above processing is completed, the number of negative sample data pairs may be too large, and further expansion of training data is also required, so active sample data expansion is required.

[0070] S2, a sample expansion model is established to expand the sample data pair until a preset condition is met;

[0071] In the implementation process of the present application, the sample expansion is performed based on the generative adversarial network as the backbone network; as shown in the figure, Figure 2 Specifically, the sample expansion model comprises a generator G and a discriminator D connected in sequence, the generator comprises a feature extraction module and a synthetic feature generation module connected in sequence;

[0072] The feature extraction module comprises a plurality of convolutional blocks and a plurality of fully connected layers connected in sequence, such as 3 convolutional blocks and 2 fully connected layers, each convolutional block comprises a convolutional layer, a normalization layer and an activation function, and after the sample data passes through the feature extraction module, each convolutional block outputs corresponding features g1, g2, g3 and a result g4;

[0073] The synthetic feature generation module comprises a plurality of synthetic blocks connected in sequence, here 3, each synthetic block comprises a splicing layer and an up-sampling layer, each synthetic block inputs the result of the previous stage, the corresponding feature output by the convolutional block, and random noise, the first synthetic block inputs g4, the feature g3 and the random noise z, and outputs P1, the second synthetic block inputs g3', the feature g2 and the random noise z, and outputs P2, the third synthetic block inputs g2', the feature g1 and the random noise z, and outputs P3, and finally the generated sample data is obtained through the corresponding activation function;

[0074] The sample data input into the feature extraction module and the sample data generated by the synthetic feature generation module are paired, input into the discriminator for discrimination, and the parameter adjustment of the sample expansion model is realized by setting the loss function, the loss function comprises a loss g_loss of the generator G and a loss d_loss of the discriminator D, the d_loss comprises a real_loss that makes the output probability of the discriminator close to the true value and a fake_loss that makes the output probability of the discriminator close to the false value, which is easily understood by those skilled in the art, and those skilled in the art can set it according to actual needs.

[0075] Due to the particularity of the generative adversarial network, the increase of the selectable random positive sample data pairs in the expanded sample pool makes the processed sample data pairs include the same number of mixed positive sample data pairs and mixed negative sample data pairs.

[0076] S3, normalizing the sample data pairs processed by S2 according to the constructed rule;

[0077] In order to ensure that the model receives standardized input, the method further normalizes the sample data pairs in the specific implementation process, including sequentially performing uniform sample size (Resize), performing center cropping (CenterCrop), uniform sample format (ToTensor), and normalization processing (Normalize) on the sample data.

[0078] Specifically, in the Resize step, the size of all images is scaled to a short side length of 255 pixel values, and the original aspect ratio is guaranteed unchanged; in the CenterCrop, the image obtained after Resize is cropped to a size of 224 × 224 pixels; in the ToTensor, the image is converted into a tensor for easy transmission into the model for training; and in the Normalize processing, the pixel values of the image are normalized, i.e., the original image pixel values of 0~255 will be mapped to the interval [0, 1].

[0079] S4, inputting the sample data pairs processed by S3 into a matching model for training;

[0080] As shown in Figure 3 , the matching model includes a VGG16, a similarity calculation module and a binary classification module connected in sequence.

[0081] S4.1, when selecting a specific backbone network of VGGNet, the application uses three VGG neural networks with different depths to verify the results of the experiment for comparison, including VGG11 (8 convolutional layers + 3 fully connected layers), VGG13 (10 convolutional layers + 3 fully connected layers) and VGG16 (13 convolutional layers + 3 fully connected layers), wherein the convolution kernel size is fixed at 3 × 3, the convolution step is fixed at 1, and the padding operation padding is fixed at 1. After the image passes through the convolutional layers and fully connected layers in the VGG model, a 2622-dimensional feature vector is finally outputted, which is slightly different from the standard VGG model. The main difference is that the pre-trained VGG model originally performs a classification task on a total of 2.62 million images with 2622 categories, and the output is a 2622-dimensional probability, so it can be regarded as having extracted the features of the face image.

[0082] S4.2, the input sample data obtains two 2622-dimensional image features after passing through the backbone network of the VGGNet, which are denoted as feature A and feature B respectively, and the similarity calculation module calculates the difference between feature A and feature B using the cosine similarity method as the input of the classification model; in order to explore the influence of different similarity calculation methods on the identity verification task, Euclidean distance and Manhattan distance are also used to calculate the difference between the images;

[0083] S4.3, a binary classification module is used for prediction and classification, and a 5-layer linear fully connected network is used in the specific implementation process of the application, the input dimension is 2622, the input vector and output vector dimensions of each layer are [2622, 1024], [1024, 512], [512, 256], [256, 128], [128, 2] respectively, the former is the input dimension, and the latter is the mapping dimension, the output obtained by each full connection layer is input into the Relu activation function, and the final two-dimensional vector represents the probability that the input picture is the target picture, the maximum value of one dimension of the two-dimensional vector is taken as the prediction result, if the maximum value is the first dimension, it means that the prediction result is 0, if the maximum value is the second dimension, it means that the prediction result is 1;

[0084] S4.4, finally, the prediction result of the model is obtained according to the two-dimensional vector output by the binary classification model, the result is 0, indicating that the input image pair is illegal, and the result is 1, indicating that the input image pair is legal, the prediction result and the actual label are used to train the model by using the cross-entropy loss function, the optimization function, and the optimal parameters of the model are obtained.

[0085] The batch_size of all models in the experiment is set to 64, the epoch is set to 150, and the learning rate is set to 0.05; after the actual experiment, for the specific selection of the VGGNet backbone network, two loss functions, absolute value loss and square loss, are used to compare with the cross-entropy function, and the evaluation index adopts the accuracy Accuracy, precision Precision, recall Recall and F1 score F1Score, and the comparison results are shown in Table 1;

[0086] Table 1 Comparison of loss of different VGGNet backbone networks

[0087]

[0088] As shown in Table 1, the cross-entropy loss performs better than the absolute value loss and the square loss on the three models; when using square loss and absolute value loss, the model effect improves limitedly with the increase of the number of VGG model network layers, only about 5%; finally, the method adopts VGG16 as the backbone network.

[0089] S5, using the trained matching model for face matching.

[0090] The application also relates to a high-precision face matching system based on deep learning driving. Figure 4 As shown in the figure, the system comprises:

[0091] One or more image acquisition modules, such as a camera, a video camera, etc., are used to acquire images to be matched, and in the actual image acquisition process, the interference of the environment should be reduced as much as possible.

[0092] A preprocessing module is used to normalize the images to be matched according to the constructed rules, including but not limited to turning the face obtained from the acquired image to be right, removing edge interference information, obtaining an ROI region, etc.

[0093] A face matching module is used to extract features and match and identify the face by using the high-precision face matching method based on deep learning driving.

[0094] In the specific implementation process, a computer readable storage medium is involved, and the high-precision face matching program based on deep learning driving is stored on the computer readable storage medium, and the program is executed by a processor to realize the high-precision face matching method based on deep learning driving.

[0095] In the specific implementation process, a computer device is also involved, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, and the processor executes the program to realize the high-precision face matching method based on deep learning driving.

[0096] A feedback module is set corresponding to the image acquisition module, and is used to feed back the matching result of the face matching module, and is generally a playing device or integrated in the acquisition device.

[0097] Those skilled in the art should understand that the embodiments of the application can be provided as a method, a system or a computer program product. Therefore, the application can be in the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0098] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more of the flowchart and / or block diagrams. Figure 1 one or more of the flowchart and / or block diagrams.

[0099] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flowchart and / or block diagrams block or blocks. Figure 1 one or more of the flowchart and / or block diagrams. Figure 1 one or more of the flowchart and / or block diagrams.

[0100] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart and / or block diagrams block or blocks. Figure 1 one or more of the flowchart and / or block diagrams. Figure 1 one or more of the flowchart and / or block diagrams.

[0101] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments described and shown, and it is therefore intended that the application cover any and all variations of the preferred embodiments which fall within the scope of the present application. Accordingly, the appended claims are intended to cover all such modifications and variations as falling within the scope of the application.

[0102] It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application cover the modifications and variations of this application provided they come within the scope of the appended claims and their equivalents.

Claims

1. A high-precision face matching method based on deep learning, characterized in that: The method includes the following steps: S1. Clean up invalid data; assign a unique identification code to the cleaned sample data based on user information, assign a unique identification code associated with members of the same family group, randomly match the cleaned sample data, and set matching thresholds α and β, where α > β > 0. Positive sample data pairs consist of two image samples of the same face, while negative sample data pairs consist of two image samples of different faces. The sample data pairs with a similarity result greater than the threshold α and different unique identification codes are designated as fixed negative sample data pairs; Sample data pairs with a similarity result less than the threshold β and the same unique identification code are designated as fixed positive sample data pairs; Randomly select several pairs of sample data with matching similarity results less than the threshold α and greater than the threshold β, which are then divided into random negative sample data pairs and random positive sample data pairs. A fixed positive sample data pair and a random positive sample data pair are used as a mixed positive sample data pair; a fixed negative sample data pair and a random negative sample data pair are used as a mixed negative sample data pair; the mixed negative sample data pair also includes facial data of different members randomly selected from the same family group after extracting user information, and combined to form a screened negative sample data pair; S2. Establish a sample augmentation model to augment the sample data pairs until the preset conditions are met; the mixed positive sample data pairs include fixed positive sample data pairs, random positive sample data pairs, and positive sample data pairs augmented by the sample augmentation model. The sample augmentation model includes a generator and a discriminator connected in sequence. The generator includes a feature extraction module and a synthetic feature generation module connected in sequence. The feature extraction module includes several convolutional blocks and several fully connected layers connected in sequence. Each convolutional block outputs the corresponding feature and result. The synthetic feature generation module includes several synthetic blocks connected in sequence. Each synthetic block takes the result of the previous stage as input, and the corresponding convolutional block outputs the corresponding feature and random noise, and performs upsampling. S3. Standardize the sample data pairs processed by S2 using the constructed rules; S4. Train the input matching model using the sample data processed in S3; S5. Use the trained matching model for face matching.

2. The high-precision face matching method based on deep learning according to claim 1, characterized in that: The sample data pairs processed by S2 include the same number of mixed positive sample data pairs and mixed negative sample data pairs.

3. The high-precision face matching method based on deep learning according to claim 1, characterized in that: In S3, normalizing sample data pairs includes sequentially performing uniform sample size, center clipping, uniform sample format, and normalization processing on the sample data.

4. The high-precision face matching method based on deep learning according to claim 1, characterized in that: In S4, the matching model includes a sequentially connected VGG16, a similarity calculation module, and a binary classification module.

5. A high-precision face matching system based on deep learning, characterized in that: The system includes: One or more image acquisition modules are used to acquire images to be matched; The preprocessing module is used to normalize the images to be matched according to the established rules; The face matching module employs the high-precision face matching method based on deep learning as described in any one of claims 1 to 4, for extracting features and matching and recognizing faces. The feedback module, corresponding to the image acquisition module settings, is used to provide feedback on the matching results of the face matching module.

Images