A real-time face detection and recognition method

Abstract

The present application relates to a kind of real-time face detection identification method, comprising the following steps: S1: constructing face detection identification system;S2: training face detection model;S3: constructing query dataset;S4: all face images in query dataset are input into the face detection model trained to obtain the feature map corresponding to each face image;S5: the corresponding fusion feature matrix of each feature map is calculated, and the fusion feature matrix corresponding to all face images is spliced to obtain a query matrix Q;S6: the face image p to be detected is input into the face detection model trained to obtain the corresponding feature map, then the feature map corresponding to p is input into face recognition model to obtain the corresponding fusion feature matrix Np, matrix P is obtained by multiplying Np with Q, the maximum value in P is compared with threshold value, and then the judgment result is output.The present application can effectively and quickly recognize face, with small amount of calculation, fast calculation speed, and also multiple cameras can be simultaneously operated for detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mechanical dynamics, and particularly relates to a real-time face detection and recognition method. BACKGROUND

[0002] Face recognition has a long history, and before deep learning, traditional methods were used to extract features, perform PCA dimension reduction, and then send them to the discriminator for judgment. The detection speed and accuracy of the face detection system using traditional methods are not high, and the robustness of the system is not strong.

[0003] In recent years, the rapid development of deep learning has also brought about rapid development of face recognition. Most face recognition systems are mainly constructed by convolutional neural networks. In order to increase the accuracy of face recognition, a larger scale convolutional neural network is needed. The increase in the number of layers and the scale of the neural network will require loading more parameters and a large amount of matrix operations, making it difficult to ensure fast operation of the face detection and recognition system while improving the accuracy of face recognition and unable to realize real-time face detection of multiple cameras in the case of a large amount of data in the face database. SUMMARY

[0004] In view of the above problems existing in the prior art, the technical problems to be solved by the present application are: the current face recognition system based on python language uses a single camera and has a small face detection scale and slow detection speed.

[0005] To solve the above technical problems, the present application adopts the following technical scheme: a real-time face detection and recognition method, comprising the following steps:

[0006] S1: constructing a face detection and recognition system, the face detection and recognition system comprising a face detection model and a face recognition model;

[0007] The face detection model is an improved Retina network model, and the feature extraction and fusion model in the Retina network model is changed to a lightweight network model ShuffleNet_V2;

[0008] The face recognition model is a ResNet network model;

[0009] S2: training the face detection model to obtain a trained face detection model;

[0010] S3: constructing a query data set, the query data set comprising m face images, and each image being labeled with the name of the person in the image;

[0011] S4: inputting each face image in the query data set into the trained face detection model to obtain a feature map corresponding to each face image;

[0012] S5: input each feature map obtained in S4 into the face recognition model to obtain a fusion feature matrix N corresponding to each feature map, N is a 1 x a matrix, and a query matrix Q is obtained by splicing the fusion feature matrices corresponding to the m face images, Q is an a x m matrix;

[0013] S6: for a face image to be detected p, input p into the trained face detection model to obtain a corresponding feature map, and then input the feature map corresponding to p into the face recognition model to obtain a corresponding fusion feature matrix Np, Np is a 1 x a' matrix;

[0014] multiply Np and Q to obtain a matrix P, P is a 1 x m matrix, and the maximum value in P is compared with a threshold value, if the maximum value is greater than or equal to the threshold value, the name labeled on the face image corresponding to the maximum value is output, otherwise, a stranger is output.

[0015] As a preferred, the process of training the face detection model in S2 to obtain the trained face detection model is as follows:

[0016] S21: select MS-Celeb-1M as a data set, and use 95% of the data in the data set as a training set and 5% of the data as a detection set;

[0017] S22: initialize the face detection model;

[0018] S23: input the data in the training set into the initialized face detection model:

[0019] S24: train the face detection model by using a stochastic gradient descent method, the momentum is 0.9, the weight decay is 0.0005, the batch size is 8x4, the learning rate starts from 0.01, and increases to 0.1 after 6 rounds of network updates, then is divided by 10 at the 38th and 50th rounds, the whole training process is performed for a total of 70 rounds to obtain a suboptimal face detection model, and ends;

[0020] S25: input the detection set into the suboptimal face detection model, and calculate the loss by using formula (1):

[0021] ; (1)

[0022] wherein, represents the Sigmoid Focal Loss value of each positive and negative sample, represents the L1 Loss value of all positive samples, represents the number of positive samples, i represents all positive and negative samples, and j represents all positive samples;

[0023] The Retina network model processes all training samples in the training set to obtain the value of each training sample When IoU >= 0.5, the training sample is a positive sample, when IoU < 0.4, the training sample is a negative sample, and the samples corresponding to the remaining IoU values are discarded.

[0024] S26: When the LOSS reaches the preset value, a trained face detection model is obtained, otherwise, the training continues in S24.

[0025] Preferably, the process of obtaining the feature map corresponding to each face image in S4 is as follows:

[0026] S41: Each face image is scaled into three feature maps with different sizes, the scaling size is an integer multiple of 2, and the size of each feature map is half of the size of the previous feature map.

[0027] S42: The smallest feature map is subjected to feature extraction through SPP in ShuffleNet_V2 to obtain a feature matrix A1; the remaining two feature maps are subjected to feature extraction through CBL in ShuffleNet_V2 to obtain two feature matrices; the two feature maps obtained after CBL are subjected to Concat function calculation and up-sampling to obtain a feature matrix A2.

[0028] S43: The sizes of A1 and A2 are unified, and A1 and A2 with the same size are subjected to Concat function calculation to obtain a feature matrix A3.

[0029] S44: A3 is subjected to deep separable convolution operation to obtain the feature map corresponding to each face image.

[0030] Preferably, the specific process of obtaining the fusion feature matrix N corresponding to each feature map in S5 is as follows:

[0031] Each feature map is input into a face recognition model, and the face recognition model frames the face part on each feature map with a rectangular frame and extracts the image feature matrix of the face part.

[0032] Compared with the prior art, the present application has at least the following advantages:

[0033] 1. Access SQLite3 (can support Windows / Linux / Unix and other mainstream operating system databases) to store data and deploy on mobile devices.

[0034] 2. The vector in the original feature matching algorithm is changed from vector multiplication to matrix multiplication, and the specific operation is as follows: a 20000*512 numpy matrix is declared, all features are put into the database, and then the database is traversed, and each row of the matrix stores the features of a face. After traversal, a large matrix is obtained.

[0035] 3. The python mutilprocess library is called to realize the multi-path operation of the camera.

[0036] 4. By improving the Retina network, i.e. replacing the lightweight backbone network part, the balance effect of detection speed and detection accuracy is achieved; access SQLite3 (can support Windows / Linux / Unix and other mainstream operating system databases) to store data and deploy on mobile devices; through the improvement of the face feature matching calculation method, the face feature matching speed is greatly improved; the python mutilprocess library is called to realize the multi-path operation of the camera. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 The flowchart of the method of the application. DETAILED DESCRIPTION

[0038] Reference Figure 1 A real-time face detection and recognition method, comprising the following steps:

[0039] S1: constructing a face detection and recognition system, the face detection and recognition system comprising a face detection model and a face recognition model;

[0040] The face detection model is an improved Retina network model, and the Retina network is a prior art. The feature extraction and fusion model in the Retina network model is changed to a lightweight network model ShuffleNet_V2, and the ShuffleNet_V2 is a prior art.

[0041] The face recognition model is a ResNet network model, and the ResNet network model is a known trained model, which is a prior art.

[0042] S2: training the face detection model to obtain a trained face detection model;

[0043] The process of training the face detection model in S2 to obtain a trained face detection model is as follows:

[0044] S21: Select MS-Celeb-1M as a data set, and 95% of the data in the data set as a training set, and 5% of the data as a detection set; select the data set as MS-Celeb-1M, and select 100K from 1M celebrities according to their popularity. Then, search for about 100 pictures for each of the 100K people using a search engine. There are 100K*100=10M pictures in total. In this paper, 100 pictures of each of the 100 people are selected, and the data set is uniformly labeled. The labeled information mainly includes the center coordinates, length and width of the target frame.

[0045] S22: Initialize the face detection model;

[0046] S23: Input the data in the training set into the initialized face detection model:

[0047] S24: Train the face detection model using the stochastic gradient descent method, with a momentum of 0.9, a weight decay of 0.0005, a batch size of 8x4, a learning rate of 0.01, and an increase to 0.1 after 6 network updates. Then, divide by 10 at the 38th and 50th rounds. The whole training process is performed for 70 rounds to obtain a suboptimal face detection model, and ends;

[0048] S25: Input the detection set into the suboptimal face detection model, and calculate the loss using formula (1):

[0049] (1)

[0050] wherein, represents the Sigmoid Focal Loss value of each positive and negative sample, represents the L1 Loss value of all positive samples, represents the number of positive samples, i represents all positive and negative samples, and j represents all positive samples.

[0051] The Retina network model processes all training samples in the training set to obtain the value of each training sample. When IoU>=0.5, the training sample is a positive sample, and when IoU<0.4, the training sample is a negative sample. The samples with other IoU values are discarded.

[0052] S26: When the LOSS reaches a preset value, a trained face detection model is obtained, otherwise return to S24 to continue training.

[0053] S3: Construct a query data set, which is composed of m face images, and each image is labeled with the name of the person in the image;

[0054] S4: input each face image in the query dataset into the trained face detection model to obtain a feature map corresponding to each face image;

[0055] The process of obtaining a feature map corresponding to each face image in S4 is as follows:

[0056] S41: scale each face image into three feature maps of different sizes, the scaling size is an integer multiple of 2, and the size of each feature map is half of the size of the previous feature map;

[0057] S42: perform feature extraction on the smallest feature map in ShuffleNet_V2 through SPP to obtain a feature matrix A1, SPP is a spatial pyramid pooling module, which is prior art. The purpose of this step is to effectively avoid image distortion caused by image region cropping and scaling operations; solve the problem of repeated feature extraction of convolutional neural network on image correlation, greatly improve the speed of generating candidate frame, and save the calculation cost; the remaining two feature maps are subjected to feature extraction through CBL in ShuffleNet_V2 to obtain two feature matrices, CBL is Convs (convolution) + BN (batch normalization) + Leaky relu (activation function), which is prior art; the two feature maps obtained after CBL are calculated through Concat function and up-sampling to obtain a feature matrix A2, Concat function and up-sampling are prior art, Concat function is used to splice tensors and connect two matrices; up-sampling is to up-sample the image to a higher resolution (resolution), which is a technical means to sample a low-resolution image into a high-resolution image;

[0058] S43: unify the size of A1 and A2, and calculate A1 and A2 of the same size through Concat function to obtain a feature matrix A3;

[0059] S44: perform deep separable convolution operation on A3 to obtain a feature map corresponding to each face image, the deep separable convolution operation is prior art.

[0060] S5: input each feature map obtained in S4 into a face recognition model to obtain a fusion feature matrix N corresponding to each feature map, N is a 1×a matrix, and a query matrix Q is obtained by splicing the fusion feature matrices corresponding to m face images, Q is an a×m matrix;

[0061] The specific process of obtaining a fusion feature matrix N corresponding to each feature map in S5 is as follows:

[0062] The face recognition model frames the face part on each feature map with a rectangular box and extracts the image matrix of the face part, wherein (box[0], box[1]) is the pixel coordinate of the upper left corner of the rectangular box, and (box[2], box[3]) is the pixel coordinate of the lower right corner of the rectangular box;

[0063] The image matrix of the face part is respectively widened by 30 pixels in the horizontal direction and the vertical direction of the image matrix, and then the fusion feature matrix N is obtained.

[0064] S6: For a face image to be detected p, the p is input into the trained face detection model to obtain a corresponding feature map, and then the feature map corresponding to p is input into the face recognition model to obtain a corresponding fusion feature matrix Np, wherein Np is a 1*a' matrix;

[0065] Np is multiplied by Q to obtain a matrix P, wherein P is a 1*m matrix, and the maximum value in P is compared with a threshold value, and if the maximum value is greater than or equal to the threshold value, the name marked on the face image corresponding to the maximum value is taken as the output, otherwise a stranger is output.

[0066] In the comparison, the face feature vector extracted in the original feature matching algorithm is multiplied by the known face feature vector, and the multiplication of the extracted fusion feature matrix Np corresponding to the face image to be detected and the query matrix Q of the known face is changed. The specific operation can be: declare a 20000*512 numpy matrix (here, the number of known stored faces in the database is 20,000), put all face features into the database, and then traverse the face storage database to store the features of one face in each row of the matrix. After traversal, a large matrix is obtained. The maximum similarity calculated by this method is more accurate.

[0067] Model evaluation

[0068] After the face detection model is trained, the test set pictures are input into the face detection model for model evaluation, and the evaluation standard is that the number of correctly recognized faces in the test set / the total number of samples in the test set is greater than or equal to 99.9%.

[0069] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the purpose and scope of the present application, and all should be covered in the scope of the claims of the present application.

Claims

1. A real-time face detection and recognition method, characterized in that: Includes the following steps: S1: Construct a face detection and recognition system, which includes a face detection model and a face recognition model; The face detection model is an improved Retina network model, which changes the feature extraction and fusion model in the Retina network model to the lightweight network model ShuffleNet_V2. The face recognition model is a ResNet network model; S2: Train the face detection model to obtain a well-trained face detection model; S3: Construct a query dataset consisting of m face images, each labeled with the name of the person in the image; S4: Input each face image in the query dataset into the trained face detection model to obtain the feature map corresponding to each face image; S5: Input each feature map obtained in S4 into the face recognition model to obtain the fusion feature matrix N corresponding to each feature map. N is a 1×a matrix. Concatenate the fusion feature matrices corresponding to m face images to obtain a query matrix Q, where Q is an a×m matrix. S6: For a face image p to be detected, input p into the trained face detection model to obtain the corresponding feature map, and then input the feature map corresponding to p into the face recognition model to obtain the corresponding fusion feature matrix Np, where Np is a 1×a' matrix; Multiply Np and Q to obtain matrix P, which is a 1×m matrix. Take the maximum value in P and compare it with the threshold. If the maximum value is greater than or equal to the threshold, the name of the face image corresponding to the maximum value is output; otherwise, output "stranger".

2. The real-time face detection and recognition method as described in claim 1, characterized in that: The process of training the face detection model in S2 to obtain the trained face detection model is as follows: S21: Select MS-Celeb-1M as the dataset, use 95% of the data in the dataset as the training set and 5% of the data as the detection set; S22: Initialize the face detection model; S23: Use the data from the training set as input to the initialized face detection model: S24: The face detection model is trained using stochastic gradient descent with a momentum of 0.9, a weight decay of 0.0005, a batch size of 8x4, and a learning rate starting from 0.

01. After the network updates for 6 rounds, the learning rate is increased to 0.1, and then divided by 10 in the 38th and 50th rounds. The entire training process is carried out for a total of 70 rounds to obtain a suboptimal face detection model, and then the process ends. S25: Use the detection set as input to the suboptimal face detection model and calculate the loss using formula (1): ； （1） in, This indicates that the Sigmoid Focal Loss value is calculated for each positive and negative sample. This indicates that the L1 Loss values ​​of all positive samples are calculated. Let represent the number of positive samples, i represent all positive and negative samples, and j represent all positive samples. The Retina network model processes all training samples in the training set to obtain the results for each training sample. The training sample is considered positive when IoU >= 0.5, negative when IoU < 0.4, and the samples corresponding to other IoU values ​​are discarded. S26: When the LOSS reaches the preset value, the trained face detection model is obtained; otherwise, return to S24 to continue training.

3. The real-time face detection and recognition method as described in claim 2, characterized in that: The process of obtaining the feature map corresponding to each face image in S4 is as follows: S41: Scale each face image into 3 feature maps of different sizes, with the scaling size being an integer multiple of 2 and the size of each feature map being half the size of the previous feature map. S42: Extract features from the smallest feature map using SPP in ShuffleNet_V2 to obtain feature matrix A1; extract features from the remaining two feature maps using CBL in ShuffleNet_V2 to obtain two feature matrices; calculate the features from the two feature maps obtained after CBL using the Concat function and upsample them to obtain feature matrix A2; S43: Unify the size of A1 and A2, and use the Concat function to calculate the characteristic matrix A3 from A1 and A2 of the same size; S44: Perform depthwise separable convolution on A3 to obtain the feature map corresponding to each face image.

4. The real-time face detection and recognition method as described in claim 3, characterized in that: The specific process of obtaining the fused feature matrix N corresponding to each feature map in S5 is as follows: Each feature map is input into the face recognition model, which then uses a rectangle to outline the face portion of each feature map and extracts the image feature matrix of the face portion.

Images