A training method, a face recognition method and a device for a convolutional neural network

Abstract

The present invention discloses a training method for a convolutional neural network, a face recognition method and a device. By setting a first training sample set including a first data set and a second data set to train the convolutional neural network, the convolutional neural network can be applied to the face recognition method, and an initial face image frame can be extracted from a face detection image. In the face recognition method, the part above the eyes in the initial face image frame is intercepted as a target face detection image, and the predicted identity information corresponding to the target face detection image is determined according to the target face detection image. Then, the target face detection image is compared with the face reference image corresponding to the predicted identity information to perform face recognition.

Description

Technical Field

[0001] The present invention relates to the field of face recognition technology, and in particular to a convolutional neural network training method, a face recognition method and a device. Background Art

[0002] In general, to better recognize facial images, more and more recognition processes require the use of Convolutional Neural Networks (CNNs). To ensure accurate recognition results, CNNs need to be repeatedly trained so that they can be used for face recognition. Summary of the Invention

[0003] Embodiments of the present invention provide a convolutional neural network training method, a face recognition method, and a device for performing face recognition.

[0004] An embodiment of the present invention provides a convolutional neural network training method, comprising:

[0005] Inputting a first training sample set into the convolutional neural network, extracting an initial face image frame from each face image sample in the first training sample set, and determining a first training loss value corresponding to the initial face image frame using a set first loss function;

[0006] The first training sample set includes a first data set and a second data set; the first data set includes a plurality of facial image samples of unobstructed faces, and the second data set includes a plurality of facial image samples of at least partially obstructed faces;

[0007] In response to the first training loss value satisfying a first loss value threshold, intercepting a portion above the eyes in the initial facial image frame in a direction pointing from the eyes to the forehead as a target facial image;

[0008] Determining, based on the target facial image, predicted identity information and an identity prediction value corresponding to the target facial image, and determining a second training loss value corresponding to the identity prediction value through a set second loss function;

[0009] The network parameters of the convolutional neural network are adjusted according to the second training loss value until the training cutoff condition is met, thereby obtaining a trained convolutional neural network.

[0010] In some examples, when the first training loss value satisfies a first loss value threshold, intercepting a portion above the eyes in the initial facial image frame as the target facial image in a direction pointing from the eyes to the forehead specifically includes:

[0011] When the first training loss value satisfies the first loss value threshold, determine the feature vector of the face in the initial face image frame;

[0012] According to the left pupil feature vector and the right pupil feature vector in the feature vector of the face in the initial face image frame, correct the initial face image frame;

[0013] In the direction from the eyes to the forehead, intercept the part above the eyes in the corrected initial face image frame as the target face image.

[0014] In some examples, the correcting the initial face image frame according to the left pupil feature vector and the right pupil feature vector in the feature vector of the face in the initial face image frame specifically includes:

[0015] According to the left pupil feature vector and the right pupil feature vector in the feature vector of the face in the initial face image frame, determine the straight line where the left pupil and the right pupil in the initial face image frame are located;

[0016] Detect whether the straight line is parallel to the upper and lower opposite sides of the initial face image frame;

[0017] If so, directly use the initial face image frame as the corrected initial face image frame;

[0018] If not, use an image processing library to rotate the initial face image frame so that the straight line is parallel to the upper and lower opposite sides as the corrected initial face image frame.

[0019] In some examples, the determining the predicted identity information and the identity prediction value corresponding to the target face image according to the target face image, and determining the second training loss value corresponding to the identity prediction value through a set second loss function specifically includes:

[0020] According to the target face image, use the trained ResNet50 network to determine the feature vector of the face in the target face image;

[0021] According to the feature vector of the face in the target face image and the feature vectors of the faces corresponding to each identity information in the pre-stored identity information library, determine the predicted identity information and the identity prediction value corresponding to the target face image;

[0022] According to the determined identity prediction value corresponding to the target face image, determine the second training loss value through a set second loss function.

[0023] In some examples, the training method of the ResNet50 network includes:

[0024] Determine the feature vectors of the human faces in each human face image sample in the second training sample set; wherein, the second training sample set includes multiple human face image samples of unoccluded human faces;

[0025] According to the left pupil feature vector and the right pupil feature vector in the feature vector of the human face in each human face image sample in the second training sample set, correct each human face image sample in the second training sample set;

[0026] In the direction from the eyes pointing to the forehead, intercept the part above the eyes in the corrected human face image sample in the second training sample set as the training human face image;

[0027] Input the training human face image into the ResNet50 network to obtain the feature vector of the training human face image;

[0028] According to the feature vector of the training human face image and the feature vectors of the human faces corresponding to each identity information in the pre-stored identity information library, determine the predicted identity information and the identity prediction value corresponding to the training human face image;

[0029] According to the determined identity prediction value corresponding to the training human face image, determine the third training loss value corresponding to the training human face image through the set second loss function;

[0030] Adjust the network parameters of the ResNet50 network according to the third training loss value until the training cut-off condition is met, and obtain the trained ResNet50 network.

[0031] The face recognition method provided by the embodiments of the present invention includes:

[0032] Obtain a face detection image;

[0033] Use a convolutional neural network to extract an initial human face image frame from the face detection image;

[0034] In the direction from the eyes pointing to the forehead, intercept the part above the eyes in the initial human face image frame extracted from the face detection image as the target face detection image;

[0035] According to the target face detection image, determine the predicted identity information corresponding to the target face detection image;

[0036] Compare the target face detection image with the face reference image corresponding to the predicted identity information to perform face recognition.

[0037] In some examples, the training method of the convolutional neural network includes:

[0038] Input the first training sample set into the convolutional neural network, extract the initial face image frames in each face image sample within the first training sample set, and determine the first training loss value corresponding to the initial face image frame through a set first loss function;

[0039] Among them, the first training sample set includes a first data set and a second data set; the first data set includes multiple face image samples of unoccluded faces, and the second data set includes multiple face image samples of at least partially occluded faces;

[0040] In response to the first training loss value satisfying the first loss value threshold, in the direction from the eyes to the forehead, intercept the part above the eyes in the initial face image frame as the target face image;

[0041] According to the target face image, determine the predicted identity information and identity prediction value corresponding to the target face image, and determine the second training loss value corresponding to the identity prediction value through a set second loss function;

[0042] Adjust the network parameters of the convolutional neural network according to the second training loss value until the training termination condition is met, and obtain the trained convolutional neural network.

[0043] In some examples, the comparing the target face detection image with the face reference image corresponding to the predicted identity information for face recognition specifically includes:

[0044] According to the target face detection image and the face reference image corresponding to the predicted identity information, use the trained ResNet50 network to determine the feature vector of the face in the target face detection image and the feature vector of the face in the face reference image;

[0045] Determine the similarity according to the feature vector of the face in the target face detection image and the feature vector of the face in the face reference image;

[0046] When the similarity satisfies the similarity threshold, determine the predicted identity information as the identity information of the target face detection image.

[0047] In some examples, the intercepting the part above the eyes in the initial face image frame extracted from the face detection image as the target face detection image in the direction from the eyes to the forehead specifically includes:

[0048] Determine the feature vector of the face in the initial face image frame;

[0049] Correct the initial face image frame according to the left pupil feature vector and the right pupil feature vector in the feature vector of the face in the initial face image frame;

[0050] In the direction from the eyes pointing to the forehead, intercept the part above the eyes in the corrected initial face image frame as the target face detection image.

[0051] In some examples, the correcting the initial face image frame according to the left pupil feature vector and the right pupil feature vector in the feature vector of the face in the initial face image frame specifically includes:

[0052] Determine the straight line where the left pupil and the right pupil in the initial face image frame are located according to the left pupil feature vector and the right pupil feature vector in the feature vector of the face in the initial face image frame;

[0053] Detect whether the straight line is parallel to the upper and lower opposite sides of the initial face image frame;

[0054] If so, directly use the initial face image frame as the corrected initial face image frame;

[0055] If not, rotate the initial face image frame using an image processing library so that the straight line is parallel to the upper and lower opposite sides as the corrected initial face image frame.

[0056] A face recognition device provided by an embodiment of the present invention includes:

[0057] An image acquisition unit configured to acquire a face detection image;

[0058] An image frame extraction unit configured to extract an initial face image frame from the face detection image using a convolutional neural network;

[0059] An interception unit configured to intercept, in the direction from the eyes pointing to the forehead, the part above the eyes in the initial face image frame extracted from the face detection image as the target face detection image;

[0060] A prediction unit configured to determine the predicted identity information corresponding to the target face detection image according to the target face detection image;

[0061] A comparison unit configured to compare the target face detection image with the face reference image corresponding to the predicted identity information for face recognition.

[0062] A computer non-transitory readable storage medium provided by an embodiment of the present invention, on which a computer program is stored, and when the program is executed by a processor, the steps of the above face recognition method are implemented.

[0063] A computer device provided by an embodiment of the present invention includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the above-mentioned face recognition method are implemented. Description of the Drawings

[0064] Figure 1 It is a flowchart of some training methods in an embodiment of the present invention;

[0065] Figure 2 It is a schematic diagram of some human faces in an embodiment of the present invention;

[0066] Figure 3 It is a schematic diagram of an initial face image frame in an embodiment of the present invention;

[0067] Figure 4 It is a schematic diagram of an image after correcting the initial face image frame in an embodiment of the present invention;

[0068] Figure 5 It is a schematic diagram of an image after cropping the initial face image frame in an embodiment of the present invention;

[0069] Figure 6 It is a schematic diagram of the structure of the ResNet50 network in an embodiment of the present invention;

[0070] Figure 7 It is a flowchart of some other training methods in an embodiment of the present invention;

[0071] Figure 8 It is a flowchart of some face recognition methods in an embodiment of the present invention;

[0072] Figure 9 It is a schematic diagram of the structure of some face recognition devices in an embodiment of the present invention. Detailed Embodiments

[0073] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. And without conflict, the embodiments and features in the embodiments of the present invention can be combined with each other. All other embodiments obtained by those of ordinary skill in the art based on the described embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0074] Unless otherwise defined, the technical terms or scientific terms used in the present invention shall have the ordinary meanings understood by those of ordinary skill in the art to which the present invention pertains. The terms "first", "second" and similar terms used in the present invention do not denote any order, quantity or importance, but are only used to distinguish different components. The terms such as "comprising" or "including" mean that the elements or objects appearing before the term cover the elements or objects listed after the term and their equivalents, without excluding other elements or objects. The terms such as "connected" or "coupled" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0075] It should be noted that the sizes and shapes of the figures in the drawings do not reflect the true scale, and the purpose is only to schematically illustrate the content of the present invention. And the same or similar reference numerals throughout denote the same or similar elements or elements having the same or similar functions.

[0076] Face recognition is a technology for identity recognition based on human facial features. With the development of deep learning, the accuracy of face recognition is getting higher and higher. For example, when punching in at work in a company, face recognition can be used for punching in. Generally, face recognition methods are based on a complete face to recognize a face. However, since a mask blocks part of the facial information, the recognition rate is relatively low when using the usual face recognition method to recognize a face wearing a mask.

[0077] Generally, in order to better recognize a face image, more and more recognition processes need to use a convolutional neural network (CNN). And in order to ensure the accuracy of the recognition result, it is necessary to repeatedly train the CNN so as to use the trained CNN for face recognition.

[0078] An embodiment of the present invention provides a training method for a convolutional neural network, as Figure 1 shown, which may include the following steps:

[0079] S110. Input a first training sample set into the convolutional neural network, extract the initial face image frames in each face image sample in the first training sample set, and determine the first training loss value corresponding to the initial face image frames through a set first loss function; wherein, the first training sample set includes a first data set and a second data set; the first data set includes multiple face image samples of unoccluded faces, and the second data set includes multiple face image samples of at least partially occluded faces; at least part of the face in the face image sample of the occluded face is blocked by a mask;

[0080] S120. In response to the first training loss value satisfying the first loss value threshold, in the direction from the eyes towards the forehead, intercept the part above the eyes in the initial face image frame as the target face image;

[0081] S130. According to the target face image, determine the predicted identity information and identity prediction value corresponding to the target face image, and determine the second training loss value corresponding to the identity prediction value through a set second loss function;

[0082] S140. Adjust the network parameters of the convolutional neural network according to the second training loss value until the training termination condition is met, and obtain the trained convolutional neural network.

[0083] In the training method of the convolutional neural network provided in the embodiments of the present invention, by setting a first training sample set including a first data set and a second data set, since the first data set includes multiple face image samples of unoccluded faces, the second data set includes multiple face image samples of at least partially occluded faces, and at least part of the faces in the face image samples of occluded faces are occluded by masks. After inputting the first training sample set into the convolutional neural network, the initial face image frames in each face image sample in the first training sample set are extracted, and the first training loss value corresponding to the initial face image frame is determined through a set first loss function. Since the first training sample set contains face images occluded by masks and the part below the nose is occluded by the masks, when the first training loss value satisfies the first loss value threshold, in the direction from the eyes towards the forehead, intercept the part above the eyes in the initial face image frame as the target face image. Therefore, by using the face image samples of unoccluded faces and the face image samples of occluded faces as a sample set to input into the convolutional neural network for training, and intercepting the part above the eyes in the initial face image frame as the target face image during the training process, the target face image with parts such as eyes, eyebrows, and forehead can be retained. Then, according to the target face image, determine the predicted identity information and identity prediction value corresponding to the target face image, and determine the second training loss value corresponding to the identity prediction value through a set second loss function. Then, adjust the network parameters of the convolutional neural network according to the second training loss value until the training termination condition is met, and obtain the trained convolutional neural network to realize the training of the convolutional neural network.

[0084] In some examples, in specific implementation, the multiple face image samples of unoccluded faces included in the first data set can be set as normal face images without masks. Exemplarily, the first data set can be the Wider Face data set. It should be noted that the Wider Face data set can be substantially the same as that in the prior art and will not be elaborated here.

[0085] In some examples, in specific implementation, the multiple face image samples in the second dataset that at least partially occlude the face can be set as face images with masks or face images occluded by other objects. Exemplarily, the second dataset can be the MAFA (A Dataset of MAsked FAces) dataset. It should be noted that the MAFA dataset can be basically the same as that in the prior art and will not be elaborated here.

[0086] In some examples, in specific implementation, the first loss function can be a cross-entropy loss function or a contrastive loss function, which is not limited herein.

[0087] In some examples, in specific implementation, the second loss function can be a cross-entropy loss function or a contrastive loss function, which is not limited herein.

[0088] In some examples, in specific implementation, as Figure 2 and Figure 3 shown, an initial face image frame K1 can be marked from the Figure 2 shown image by using a convolutional neural network, and the initial face image frame K1 can be extracted to form the Figure 3 shown initial face image frame K2.

[0089] In some examples, in specific implementation, when the first training loss value satisfies the first loss value threshold, in the direction from the eyes to the forehead, the part above the eyes in the initial face image frame is intercepted as the target face image, which specifically may include:

[0090] When the first training loss value satisfies the first loss value threshold, determining the feature vector of the face in the initial face image frame;

[0091] According to the left pupil feature vector and the right pupil feature vector in the feature vector of the face in the initial face image frame, correcting the initial face image frame;

[0092] In the direction from the eyes to the forehead, intercepting the part above the eyes in the corrected initial face image frame as the target face image.

[0093] Exemplarily, since for face images with masks and face images without masks, parts such as the eyes, eyebrows, and forehead are not occluded by the mask, the initial face image frame can be corrected through the left pupil feature vector and the right pupil feature vector in the eyes, and then the part above the eyes in the corrected initial face image frame is intercepted as the target face image.

[0094] Exemplarily, correcting the initial face image frame according to the left pupil feature vector and the right pupil feature vector in the feature vector of the face in the initial face image frame may specifically include:

[0095] Determine the straight line where the left pupil and the right pupil in the initial face image frame are located according to the left pupil feature vector and the right pupil feature vector in the feature vector of the face in the initial face image frame;

[0096] Detect whether the straight line is parallel to the upper and lower opposite sides of the initial face image frame;

[0097] If so, directly use the initial face image frame as the corrected initial face image frame;

[0098] If not, use an image processing library to rotate the initial face image frame so that the straight line is parallel to the upper and lower opposite sides as the corrected initial face image frame.

[0099] Exemplarily, the face in the initial face image frame can be corrected by the feature vectors of the left pupil and the right pupil in the eyes. Combining Figure 3 and Figure 4 , the straight line L0 where the left pupil and the right pupil in the initial face image frame are located can be determined through the feature vectors of the left pupil and the right pupil in the eyes in the initial face image frame, and it is detected whether the straight line L0 is parallel to the upper and lower opposite sides of the initial face image frame. When the straight line L0 is parallel to the upper and lower opposite sides of the initial face image frame, the initial face image frame can be directly used as the corrected initial face image frame. When the straight line L0 is not parallel to the upper and lower opposite sides of the initial face image frame, an image processing library can be used to rotate the initial face image frame so that the straight line is parallel to the upper and lower opposite sides, and the rotated initial face image frame is used as the corrected initial face image frame. For example, Figure 3 As shown in Figure 4 , the straight line L0 that was not originally parallel to the upper and lower opposite sides of the initial face image frame is rotated to be parallel to the upper and lower opposite sides of the initial face image frame by rotation.

[0100] Exemplarily, the image processing library can be, for example, OpenCV. Of course, the image processing library can also be set to other forms in the prior art, which is not limited herein.

[0101] In some examples, in specific implementation, according to the target face image, determine the predicted identity information and the identity prediction value corresponding to the target face image, and determine the second training loss value corresponding to the identity prediction value through a set second loss function, which can specifically include:

[0102] According to the target face image, use the trained ResNet50 network to determine the feature vector of the face in the target face image;

[0103] According to the feature vector of the face in the target face image and the feature vectors of the faces corresponding to each identity information in the pre-stored identity information library, determine the predicted identity information and the identity prediction value corresponding to the target face image;

[0104] According to the identity prediction value corresponding to the determined target face image, determine the second training loss value through the set second loss function.

[0105] Exemplarily, the ResNet50 network structure is as Figure 6 shown, and may have a plurality of residual units 610 and a fully connected layer 620. Among them, a face image can be input into the first residual unit 610, and then these residual units 610 are used for feature calculation. Finally, the fully connected layer 620 fully connects to generate a feature vector and outputs the generated feature vector. Exemplarily, the feature vector of the face in the target face image determined by using the trained ResNet50 network can be a 512-dimensional feature vector. This can reduce the storage space of the face feature vector and improve the calculation speed of feature matching. Of course, in practical applications, the specific dimension of the feature vector can also be determined according to the requirements of practical applications, which is not limited herein.

[0106] Exemplarily, as Figure 7 shown, the training method of the ResNet50 network may include the following steps:

[0107] S210. Determine the feature vectors of the faces in each face image sample in the second training sample set; wherein, the second training sample set includes a plurality of face image samples of unoccluded faces; exemplarily, the second training sample set can be the MS-Celeb-1M dataset. For example, use the method in the prior art to determine the feature vectors of the faces in each face image sample in the second training sample set.

[0108] S220. Correct each face image sample in the second training sample set according to the left pupil feature vector and the right pupil feature vector in the feature vector of the face in each face image sample in the second training sample set; exemplarily, according to the left pupil feature vector and the right pupil feature vector in the feature vector of the face in each face image sample in the second training sample set, determine whether the left pupil and the right pupil in the same face image sample are on the same horizontal line; wherein, the horizontal line is parallel to the upper edge or the lower edge of the initial face image frame; if not, then use an image processing library to rotate the left pupil and the right pupil in the same face image sample to the horizontal line.

[0109] S230. In the direction from the eyes to the forehead, intercept the part above the eyes in the corrected face image sample in the second training sample set as the training face image; the specific process of this step can refer to the working process in step S120: in the direction from the eyes to the forehead, intercept the part above the eyes in the initial face image frame as the target face image, which is not limited herein.

[0110] S240. Input the training face images into the ResNet50 network to obtain the feature vectors of the training face images. Exemplarily, the ResNet50 network structure is as Figure 6 shown.

[0111] S250. According to the feature vectors of the training face images and the feature vectors of the faces corresponding to each identity information in the pre-stored identity information library, determine the predicted identity information and the identity prediction value corresponding to the training face images.

[0112] S260. According to the identity prediction value corresponding to the determined training face images, determine the third training loss value corresponding to the training face images through a set second loss function.

[0113] S270. Adjust the network parameters of the ResNet50 network according to the third training loss value until the training termination condition is met, and obtain the trained ResNet50 network.

[0114] The following combines specific embodiments to elaborate on the present invention in detail. It should be noted that this embodiment is for better explaining the present invention, but does not limit the present invention.

[0115] The training method of the convolutional neural network provided by the embodiment of the present invention may include the following steps:

[0116] (1) Use the Wider Face dataset and the MAFA dataset to form a first training sample set, input the first training sample set into the convolutional neural network to be trained, extract the initial face image frames in each face image sample in the first training sample set, and determine the first training loss value corresponding to the initial face image frames through a set first loss function.

[0117] Exemplarily, for each face image sample, 5 different-scale face feature vectors in the face image sample can be obtained through the convolutional neural network to be trained. Then, target retrieval and positioning can be performed according to the 5 different-scale face feature vectors in the face image sample to determine the initial face image frame where the face in the face image sample is located. The initial face image frame refers to the square area formed by the face after cropping the face from the face image sample. For example, as Figure 2 and Figure 3 shown, the convolutional neural network can mark the initial face image frame K1 from the Figure 2 shown image, and extract the initial face image frame K1 to form the Figure 3 shown initial face image frame K2.

[0118] And the first training loss value corresponding to the initial face image frame can be determined through the first loss function.

[0119] It should be noted that the required scale of the face image samples input into the convolutional neural network to be trained is set to 640*640 pixels. When the size of the face image samples in the first training sample set is different from the required scale of the face image samples input into the convolutional neural network to be trained, the size of the face image samples in the first training sample set can be changed so that the size of the face image samples in the changed first training sample set is 640*640 pixels. Thus, the size of the face image samples in the first training sample set meets the requirements of the convolutional neural network to be trained.

[0120] (2) For each initial face image frame, when the first training loss value meets the first loss value threshold, it indicates that the initial face image frame meets the requirements, and then the feature vector of the face in the initial face image frame is determined.

[0121] It should be noted that the first loss value threshold can be designed and determined according to the requirements of actual applications, and is not limited here.

[0122] (3) For the feature vector of the face in each initial face image frame, according to the left pupil feature vector and the right pupil feature vector in the feature vector of the face in the initial face image frame, the straight line where the left pupil and the right pupil in the initial face image frame are located is determined.

[0123] Exemplarily, as Figure 3 shown, the straight line L0 where the left pupil and the right pupil in the initial face image frame are located can be determined through the feature vectors of the left pupil and the right pupil in the eyes in the initial face image frame. Figure 3 The straight line L0 where the shown left pupil and right pupil are located is not parallel to the upper and lower opposite sides of the initial face image frame.

[0124] (4) Detect whether the straight line L0 is parallel to the upper and lower opposite sides of the initial face image frame. If so, perform step (5); if not, perform step (6).

[0125] (5) Directly use the initial face image frame as the corrected initial face image frame.

[0126] (6) Use an image processing library (such as OpenCV) to rotate the initial face image frame so that the straight line is parallel to the upper and lower opposite sides, and use the rotated initial face image frame as the corrected initial face image frame.

[0127] For example, Figure 3 As Figure 4 shown, the straight line L0 that was originally not parallel to the upper and lower opposite sides of the initial face image frame is rotated to be parallel to the upper and lower opposite sides of the initial face image frame by rotation.

[0128] (7) For example Figure 4 and Figure 5 As shown, in the direction from the eyes towards the forehead, the part above the eyes in the corrected initial face image frame is intercepted as the target face image. Exemplarily, the target face image may include the regions where parts such as the eyes, eyebrows, and forehead are located.

[0129] (8) Input the target face image into the trained ResNet50 network to determine the feature vector of the face in the target face image.

[0130] Exemplarily, as Figure 6 shown, input the target face image into the trained ResNet50 network, perform feature calculations through the residual unit 610 in sequence, and finally generate a 512 - dimensional feature vector of the face in the target face image through full connection by the fully - connected layer 620.

[0131] (9) For each target face image, determine the predicted identity information and the identity prediction value corresponding to the target face image according to the feature vector of the face in the target face image and the feature vectors of the faces corresponding to each identity information in the pre - stored identity information library.

[0132] Exemplarily, the feature vector of the face in the target face image and the feature vectors of the faces corresponding to each identity information in the pre - stored identity information library can be compared to predict the identity information corresponding to the target face image, that is, obtain the predicted identity information corresponding to the target face image. And when the identity information corresponding to the target face image is predicted, the identity prediction value corresponding to the predicted identity information can also be determined.

[0133] It should be noted that the identity information (for example, identity ID) of multiple persons and the face feature vectors corresponding to each person are pre - obtained to form an identity information library, and the identity information library is stored.

[0134] (10) According to the identity prediction value corresponding to the determined target face image, determine the second training loss value through the set second loss function.

[0135] (11) Adjust the network parameters of the convolutional neural network according to the second training loss value.

[0136] Repeat the above steps (1) to (11) until the training termination condition is met, and obtain the trained convolutional neural network.

[0137] An embodiment of the present invention also provides a face recognition method. As Figure 8 shown, it may include the following steps:

[0138] S310. Obtain a face detection image;

[0139] S320. Extract an initial face image frame from the face detection image using a convolutional neural network; wherein, the convolutional neural network is trained by the above training method.

[0140] S330. In the direction from the eyes towards the forehead, intercept the part above the eyes in the initial face image frame extracted from the face detection image as the target face detection image.

[0141] S340. Determine the predicted identity information corresponding to the target face detection image according to the target face detection image.

[0142] S350. Compare the target face detection image with the face reference image corresponding to the predicted identity information for face recognition.

[0143] In the face recognition method provided by the embodiments of the present invention, by applying the convolutional neural network trained above to the face recognition method, an initial face image frame can be extracted from the face detection image. Then, the part above the eyes in the initial face image frame is intercepted as the target face detection image to determine the predicted identity information corresponding to the target face detection image according to the target face detection image. Then, the target face detection image is compared with the face reference image corresponding to the predicted identity information for face recognition. Thus, the face recognition method provided by the present invention can be applied to attendance check-in, or the face recognition method provided by the present invention can also be applied to traffic detection.

[0144] In some examples, in specific implementation, comparing the target face detection image with the face reference image corresponding to the predicted identity information for face recognition may specifically include:

[0145] According to the target face detection image and the face reference image corresponding to the predicted identity information, use the trained ResNet50 network to determine the feature vector of the face in the target face detection image and the feature vector of the face in the face reference image.

[0146] Determine the similarity according to the feature vector of the face in the target face detection image and the feature vector of the face in the face reference image.

[0147] When the similarity meets the similarity threshold, determine the predicted identity information as the identity information of the target face detection image.

[0148] In some examples, in specific implementation, in the direction from the eyes towards the forehead, intercepting the part above the eyes in the initial face image frame extracted from the face detection image as the target face detection image may specifically include:

[0149] Determine the feature vector of the face in the initial face image frame.

[0150] According to the left pupil feature vector and the right pupil feature vector in the feature vector of the face in the initial face image frame, correct the initial face image frame;

[0151] In the direction from the eyes towards the forehead, intercept the part above the eyes in the corrected initial face image frame as the target face detection image.

[0152] In some examples, in specific implementation, according to the left pupil feature vector and the right pupil feature vector in the feature vector of the face in the initial face image frame, correcting the initial face image frame may specifically include:

[0153] According to the left pupil feature vector and the right pupil feature vector in the feature vector of the face in the initial face image frame, determine the straight line where the left pupil and the right pupil in the initial face image frame are located;

[0154] Detect whether the straight line is parallel to the upper and lower opposite sides of the initial face image frame;

[0155] If so, directly use the initial face image frame as the corrected initial face image frame;

[0156] If not, use an image processing library to rotate the initial face image frame so that the straight line is parallel to the upper and lower opposite sides as the corrected initial face image frame.

[0157] The following combines specific embodiments to elaborate on the present invention in detail. It should be noted that this embodiment is for better explaining the present invention, but does not limit the present invention.

[0158] The face recognition method provided by the embodiment of the present invention may include the following steps:

[0159] (1) Obtain a face detection image. Exemplarily, a face detection image is obtained by using a camera, or a face detection image can also be obtained by using an image transmission device. Of course, in practical applications, it can be designed according to the actual application requirements, and no limitation is made here.

[0160] Exemplarily, the face detection image can be collected by a camera when a user punches the clock for attendance. Or, the face detection image can also be one or more images determined from a video stream.

[0161] (2) After adjusting the size of the face detection image to the size required by the above-mentioned trained convolutional neural network, input it into the above-mentioned trained convolutional neural network to extract an initial face image frame from the face detection image. Among them, the initial face image frame refers to the square area formed by the face after cropping the face from the face detection image. For example, Figure 3 The initial face image frame K2 shown.

[0162] (3) Determine the feature vector of the face in the initial face image frame.

[0163] (4) For the feature vector of the face in each initial face image frame, determine the straight line where the left pupil and the right pupil in the initial face image frame are located according to the left pupil feature vector and the right pupil feature vector in the feature vector of the face in the initial face image frame.

[0164] Exemplarily, as Figure 3 shown, the straight line L0 where the left pupil and the right pupil in the initial face image frame are located can be determined through the feature vectors of the left pupil and the right pupil in the eyes in the initial face image frame. Figure 3 The straight line L0 where the shown left pupil and right pupil are located is not parallel to the upper and lower opposite sides of the initial face image frame.

[0165] (5) Detect whether the straight line L0 is parallel to the upper and lower opposite sides of the initial face image frame. If so, execute step (6); if not, execute step (7).

[0166] (6) Directly use the initial face image frame as the corrected initial face image frame.

[0167] (7) Use an image processing library (such as OpenCV) to rotate the initial face image frame so that the straight line is parallel to the upper and lower opposite sides, and use the rotated initial face image frame as the corrected initial face image frame.

[0168] For example, Figure 3 as Figure 4 shown, the straight line L0 that was not originally parallel to the upper and lower opposite sides of the initial face image frame is rotated to be parallel to the upper and lower opposite sides of the initial face image frame by rotation.

[0169] (8) For example Figure 4 as Figure 5 shown, in the direction from the eyes to the forehead, intercept the part above the eyes in the corrected initial face image frame as the target face detection image. Exemplarily, the target face detection image may include the regions where parts such as the eyes, eyebrows, and forehead are located.

[0170] (9) Input the target face detection image into the trained ResNet50 network to determine the feature vector of the face in the target face image.

[0171] Exemplarily, as Figure 6 shown, input the target face detection image into the trained ResNet50 network, perform feature calculations through the residual unit 610 in sequence, and finally use the fully connected layer 620 to fully connect and generate a 512-dimensional feature vector of the face in the target face detection image.

[0172] (10) Determine the predicted identity information corresponding to the target face detection image according to the feature vector of the face in the target face detection image and the feature vectors of the faces corresponding to the respective identity information in the pre-stored identity information database.

[0173] (11) According to the target face detection image and the face reference image corresponding to the predicted identity information, use the trained ResNet50 network to determine the feature vector of the face in the target face detection image and the feature vector of the face in the face reference image.

[0174] Exemplarily, as shown in Figure 6 Input the target face detection image and the face reference image into the trained ResNet50 network respectively, perform feature calculations through the residual unit 610 in sequence, and finally use the fully connected layer 620 to fully connect to generate a 512-dimensional feature vector of the face in the target face detection image and a 512-dimensional feature vector of the face in the face reference image.

[0175] (12) Determine the similarity according to the feature vector of the face in the target face detection image and the feature vector of the face in the face reference image.

[0176] Exemplarily, determine the cosine distance according to the feature vector of the face in the target face detection image and the feature vector of the face in the face reference image. Then determine the similarity according to the cosine distance.

[0177] (13) When the similarity meets the similarity threshold, determine the predicted identity information as the identity information of the target face detection image, thereby completing face recognition. If this method is applied to attendance punching, it can be stated that the identity of the face detection image has been recognized, so that the attendance punching is completed.

[0178] Exemplarily, the similarity threshold can be a value in [-1.0, 1.0]. When the similarity is greater than the similarity threshold, it can be confirmed that the person in the target face detection image and the person in the face reference image are the same person.

[0179] An embodiment of the present invention also provides a face recognition device, as shown in Figure 9 and may include:

[0180] An image acquisition unit 910, configured to acquire a face detection image; Exemplarily, the image acquisition unit may be a camera or an image transmission device. In this way, a face detection image can be obtained by using a camera, or a face detection image can be obtained by using an image transmission device. Of course, in practical applications, it can be designed according to the actual application requirements and is not limited herein.

[0181] An image frame extraction unit 920, configured to extract an initial face image frame from the face detection image using a convolutional neural network;

[0182] The cropping unit 930 is configured to crop the upper part of the eyes in the initial face image frame extracted from the face detection image in the direction of the eyes pointing to the forehead as the target face detection image;

[0183] The prediction unit 940 is configured to determine the predicted identity information corresponding to the target face detection image according to the target face detection image;

[0184] The comparison unit 950 is configured to compare the target face detection image with the face reference image corresponding to the predicted identity information for face recognition.

[0185] Exemplarily, the image frame extraction unit, the cropping unit, the prediction unit, and the comparison unit in the embodiments of the present invention may be implemented in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects.

[0186] Based on the same inventive concept, an embodiment of the present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above face recognition methods provided by the embodiments of the present invention are implemented. Specifically, the present invention may be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories and optical memories, etc.) containing computer-usable program codes.

[0187] Based on the same inventive concept, an embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the steps of any of the above face recognition methods provided by the embodiments of the present invention are implemented.

[0188] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these changes and modifications.

Claims

1. A training method for a convolutional neural network, characterized in that Including: Input the first training sample set into the convolutional neural network, extract the initial face image frames in each face image sample within the first training sample set, and determine the first training loss value corresponding to the initial face image frame through a set first loss function; wherein, the first training sample set includes a first data set and a second data set; the first data set includes multiple face image samples of unoccluded faces, and the second data set includes multiple face image samples of at least partially occluded faces; When the first training loss value meets the first loss value threshold, determine the feature vector of the face in the initial face image frame; According to the left pupil feature vector and the right pupil feature vector in the feature vector of the face in the initial face image frame, determine the straight line where the left pupil and the right pupil in the initial face image frame are located; detect whether the straight line is parallel to the upper and lower opposite sides of the initial face image frame; if so, directly use the initial face image frame as the corrected initial face image frame; if not, use an image processing library to rotate the initial face image frame so that the straight line is parallel to the upper and lower opposite sides as the corrected initial face image frame; In the direction from the eyes to the forehead, intercept the part above the eyes in the corrected initial face image frame as the target face image; According to the target face image, determine the predicted identity information and identity prediction value corresponding to the target face image, and determine the second training loss value corresponding to the identity prediction value through a set second loss function; Adjust the network parameters of the convolutional neural network according to the second training loss value until the training cut-off condition is met, and obtain the trained convolutional neural network.

2. The training method according to claim 1, characterized in that The step of determining the predicted identity information and identity prediction value corresponding to the target face image according to the target face image, and determining the second training loss value corresponding to the identity prediction value through a set second loss function specifically includes: According to the target face image, use the trained ResNet50 network to determine the feature vector of the face in the target face image; According to the feature vector of the face in the target face image and the feature vectors of the faces corresponding to each identity information in the pre-stored identity information library, determine the predicted identity information and identity prediction value corresponding to the target face image; According to the determined identity prediction value corresponding to the target face image, determine the second training loss value through a set second loss function.

3. The training method according to claim 2, wherein The training method of the ResNet50 network includes: Determine the feature vector of the face in each face image sample in the second training sample set; wherein, the second training sample set includes multiple face image samples of unoccluded faces; According to the left pupil feature vector and the right pupil feature vector in the feature vector of the face in each face image sample in the second training sample set, correct each face image sample in the second training sample set; In the direction from the eyes to the forehead, intercept the part above the eyes in the corrected face image sample in the second training sample set as the training face image; Input the training face image into the ResNet50 network to obtain the feature vector of the training face image; Determine the predicted identity information and identity prediction value corresponding to the training face image according to the feature vector of the training face image and the feature vectors of the faces corresponding to the respective identity information in the pre-stored identity information library; Determine the third training loss value corresponding to the training face image through the set second loss function according to the identity prediction value corresponding to the determined training face image; Adjust the network parameters of the ResNet50 network according to the third training loss value until the training cut-off condition is satisfied to obtain the trained ResNet50 network.

4. A face recognition method, characterized in that, Including: Obtain a face detection image; Use a convolutional neural network to extract an initial face image frame from the face detection image; Determine the feature vector of the face in the initial face image frame; Determine the straight line where the left pupil and the right pupil in the initial face image frame are located according to the left pupil feature vector and the right pupil feature vector in the feature vector of the face in the initial face image frame; Detect whether the straight line is parallel to the upper and lower opposite sides of the initial face image frame; If so, directly use the initial face image frame as the corrected initial face image frame; If not, use an image processing library to rotate the initial face image frame so that the straight line is parallel to the upper and lower opposite sides as the corrected initial face image frame; In the direction from the eyes to the forehead, intercept the part above the eyes in the corrected initial face image frame as the target face detection image; Determine the predicted identity information corresponding to the target face detection image according to the target face detection image; Compare the target face detection image with the face reference image corresponding to the predicted identity information for face recognition.

5. The face recognition method according to claim 4, characterized in that The training method of the convolutional neural network includes: Input the first training sample set into the convolutional neural network, extract the initial face image frames in each face image sample in the first training sample set, and determine the first training loss value corresponding to the initial face image frame through the set first loss function; Among them, the first training sample set includes a first data set and a second data set; The first data set includes multiple face image samples of unoccluded faces, and the second data set includes multiple face image samples of at least partially occluded faces; In response to the first training loss value satisfying the first loss value threshold, in the direction from the eyes to the forehead, intercept the part above the eyes in the initial face image frame as the target face image; Determine the predicted identity information and identity prediction value corresponding to the target face image according to the target face image, and determine the second training loss value corresponding to the identity prediction value through the set second loss function; Adjust the network parameters of the convolutional neural network according to the second training loss value until the training cut-off condition is satisfied to obtain the trained convolutional neural network.

6. The face recognition method according to claim 5, wherein The specific steps of comparing the target face detection image with the face reference image corresponding to the predicted identity information for face recognition include: Based on the target face detection image and the face reference image corresponding to the predicted identity information, use the trained ResNet50 network to determine the feature vector of the face in the target face detection image and the feature vector of the face in the face reference image; Determine the similarity based on the feature vector of the face in the target face detection image and the feature vector of the face in the face reference image; When the similarity meets the similarity threshold, determine the predicted identity information as the identity information of the target face detection image.

7. A face recognition device, characterized in that, Includes: An image acquisition unit configured to acquire a face detection image; An image frame extraction unit configured to extract an initial face image frame from the face detection image using a convolutional neural network; A cropping unit configured to determine the feature vector of the face in the initial face image frame; determine the straight line where the left pupil and the right pupil in the initial face image frame are located according to the left pupil feature vector and the right pupil feature vector in the feature vector of the face in the initial face image frame; detect whether the straight line is parallel to the upper and lower opposite sides of the initial face image frame; if so, directly use the initial face image frame as the corrected initial face image frame; if not, use an image processing library to rotate the initial face image frame so that the straight line is parallel to the upper and lower opposite sides as the corrected initial face image frame; in the direction from the eyes to the forehead, crop the part above the eyes in the corrected initial face image frame as the target face detection image; A prediction unit configured to determine the predicted identity information corresponding to the target face detection image according to the target face detection image; A comparison unit configured to compare the target face detection image with the face reference image corresponding to the predicted identity information for face recognition.

8. A computer non-transitory readable storage medium, on which a computer program is stored, characterized in that, When the program is executed by a processor, it implements the steps of the face recognition method according to any one of claims 4-6.

9. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the face recognition method according to any one of claims 4-6.

Images