Face recognition method, device, equipment and storage medium

Abstract

The present application provides a face recognition method, device, equipment and storage medium. First, a face image of a user to be identified is obtained, wherein the face image can be a face image of a person wearing a mask or a face image of a person not wearing a mask; then, an upper face image is extracted from the face image, and the upper face image is input into a pre-trained first convolutional neural network model to obtain a first image feature; at the same time, the face image is input into a pre-trained second convolutional neural network model to obtain a second image feature; the first image feature and the second image feature are fused to obtain a fused image feature; the fused image feature is compared with the face feature in the face library, and whether the face recognition of the user to be identified is passed is determined according to the comparison result. The face recognition method uses a fusion feature formed by a feature fusion method to improve the expression and identification capabilities of the face, thereby improving the accuracy of face recognition.

Description

Technical Field

[0001] The present application relates to the field of face recognition technology, and in particular to a face recognition method, device, equipment and storage medium. Background Art

[0002] With the development of computer technology, face recognition has been applied to more and more fields, such as attendance and access control. Face recognition usually uses machine learning methods to train face recognition models, and then uses the face recognition model to recognize faces in images.

[0003] In traditional face recognition, each user is required to upload a photo as a face base image, which is then associated with the user after feature extraction for identity comparison and authentication through face recognition in the future. However, the base images uploaded are generally pictures without masks. In some special periods (especially during the COVID-19 pandemic), users usually wear masks, and the difficulty in recognizing faces wearing masks is that the face recognition model must solve both face recognition with masks and face recognition without masks. Compared with not wearing a mask, the face wearing a mask lacks relevant information such as the mouth and nose. The use of current training models often leads to low recognition results, and may even lead to unrecognizable situations.

[0004] Application Contents

[0005] In view of this, embodiments of the present application provide a face recognition method, apparatus, device and storage medium.

[0006] In a first aspect, an embodiment of the present application provides a face recognition method, the method comprising:

[0007] Obtain a facial image of the user to be identified;

[0008] Extracting an upper face image from the face image;

[0009] Inputting the upper face image into a pre-trained first convolutional neural network model to obtain a first image feature;

[0010] Inputting the face image into a pre-trained second convolutional neural network model to obtain a second image feature;

[0011] fusing the first image feature and the second image feature to obtain a fused image feature;

[0012] The fused image features are compared with the facial features in the face database, and whether the facial recognition of the user to be identified is passed is determined based on the comparison result.

[0013] In a second aspect, an embodiment of the present application provides a face recognition device, the device comprising:

[0014] A face image acquisition module, used to acquire a face image of a user to be identified;

[0015] An upper face image extraction module, used to extract an upper face image from the face image;

[0016] A first feature acquisition module, used for inputting the upper face image into a pre-trained first convolutional neural network model to obtain a first image feature;

[0017] A second feature acquisition module, used for inputting the face image into a pre-trained second convolutional neural network model to obtain a second image feature;

[0018] a fusion feature acquisition module, used for fusing the first image feature and the second image feature to obtain a fusion image feature;

[0019] The comparison module is used to compare the fused image features with the facial features in the face database, and determine whether the facial recognition of the user to be identified is passed according to the comparison result.

[0020] In a third aspect, an embodiment of the present application provides a terminal device, comprising: a memory; one or more processors coupled to the memory; one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the face recognition method provided in the first aspect above.

[0021] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute the face recognition method provided in the first aspect above.

[0022] The face recognition method, apparatus, device and storage medium provided in the embodiments of the present application first obtain a face image of a user to be identified, wherein the face image can be a face image of a person wearing a mask or a face image of a person not wearing a mask; then extract an upper face image from the face image, and input the upper face image into a pre-trained first convolutional neural network model to obtain a first image feature; at the same time, input the face image into a pre-trained second convolutional neural network model to obtain a second image feature; fuse the first image feature and the second image feature to obtain a fused image feature; compare the fused image feature with the face feature in a face library, and determine whether the face recognition of the user to be identified is passed according to the comparison result.

[0023] The face recognition method uses the first convolutional neural network model and the second convolutional neural network model to extract the upper face features and the full face features, respectively, and then fuses the upper face features and the full face features, and then compares the fused features with the face features in the face library to determine whether the face recognition is passed. The fusion features formed by the feature fusion method improve the expression and identification capabilities of the face, thereby improving the accuracy of face recognition. In addition, this method can be used to identify users wearing masks, and users who are not wearing masks, which is very convenient to use. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0025] Figure 1 A schematic diagram of an application scenario of the face recognition method provided in an embodiment of the present application;

[0026] Figure 2 A flowchart of a face recognition method provided by an embodiment of the present application;

[0027] Figure 3 A schematic diagram of an improved MobileFaceNet network model provided for one embodiment of the present application;

[0028] Figure 4 A schematic diagram of an improved Resnet network model provided for one embodiment of the present application;

[0029] Figure 5 A structural diagram of a face recognition device provided in one embodiment of the present application;

[0030] Figure 6 A schematic diagram of the structure of a terminal device provided in one embodiment of the present application;

[0031] Figure 7 A schematic diagram of the structure of a computer-readable storage medium provided in one embodiment of the present application. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present application are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0033] In order to explain the present application in more detail, a face recognition method, apparatus, terminal device and computer storage medium provided by the present application are specifically described below in conjunction with the accompanying drawings.

[0034] Please refer to Figure 1 , Figure 1 A schematic diagram of an application scenario of the face recognition method provided by the embodiment of the present application is shown, and the application scenario includes the terminal device 100 provided by the embodiment of the present application, and the terminal device 100 can be various electronic devices with a display screen (such as the structural diagrams of 102, 104, 106 and 108), including but not limited to smart phones, computer devices, clock-in machines, attendance machines, etc., wherein the computer device can be at least one of a desktop computer, a portable computer, a laptop computer, a tablet computer, etc. When the user's face image is collected, the terminal device 100 executes the face recognition method of the present application, and the specific process is referred to the face recognition method embodiment.

[0035] Secondly, the terminal device 100 may generally refer to one of a plurality of terminal devices, and this embodiment is only illustrated by the terminal device 100. Those skilled in the art may know that the number of the above-mentioned terminal devices may be more or less. For example, the above-mentioned terminal devices may be only a few, or the above-mentioned terminal devices may be dozens or hundreds, or more. The embodiment of the present application does not limit the number and type of terminal devices. The terminal device 100 may be used to execute a face recognition method provided in the embodiment of the present application.

[0036] In an optional implementation, the application scenario includes, in addition to the terminal device 100 provided in the embodiment of the present application, a server, wherein a network is provided between the server and the terminal device. The network is used to provide a medium for a communication link between the terminal device and the server. The network may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0037] It should be understood that the number of terminal devices, networks and servers is only illustrative. Depending on the implementation needs, there may be any number of terminal devices, networks and servers. For example, the server may be a server cluster consisting of multiple servers. Among them, the terminal device interacts with the server through the network to receive or send messages, etc. The server may be a server that provides various services. The server may be used to execute the steps of a face recognition method provided in an embodiment of the present application. In addition, when the terminal device executes a face recognition method provided in an embodiment of the present application, part of the steps may be executed on the terminal device and part of the steps may be executed on the server, which is not limited here.

[0038] Based on this, a face recognition method is provided in the embodiment of the present application. Figure 2 , Figure 2 A schematic diagram of a face recognition method provided in an embodiment of the present application is shown, in which the method is applied to Figure 1 The terminal device in the example is used to illustrate, including the following steps:

[0039] Step S110, obtaining a facial image of a user to be identified.

[0040] The facial image of the user to be identified refers to the facial image of any user who needs to perform facial recognition. The facial image can be collected by using an image acquisition module built into the terminal device, or by using an independent image acquisition device.

[0041] Optionally, the facial image is an image of the user's entire face, which may be a masked whole-face image or an unmasked whole-face image, for example, a facial image of a person wearing a mask or a facial image of a person not wearing a mask.

[0042] Step S120, extracting the upper face image from the face image.

[0043] After obtaining the face image, the upper face image is extracted from the face image, wherein the upper face image includes information such as eyes, forehead, eyebrows, hair, etc.

[0044] The specific process is: align the face image to N*N (for example, 112*112) size, then crop the face along the line connecting the two points A(0,N / 2) and B(N,N / 2), take the upper half of the face, and then scale it to N*N size to obtain the upper half face image.

[0045] Step S130: input the upper face image into a pre-trained first convolutional neural network model to obtain a first image feature.

[0046] Step S140: input the face image into a pre-trained second convolutional neural network model to obtain a second image feature.

[0047] Furthermore, a specific implementation of the first convolutional neural network model and the second convolutional neural network model is provided, which is described as follows:

[0048] In an optional embodiment, the first convolutional neural network model and the second convolutional neural network model include a ResNet network model, a VGG network model, a DenseNet network model, a SqueezeNet network model, or a MobileFaceNet network model.

[0049] Among them, the ResNet network model, VGG network model, DenseNet network model, SqueezeNet network model and MobileFaceNet network model are all very useful network models in convolutional neural networks and can be used in feature extraction for face recognition.

[0050] In a preferred embodiment, please refer to Figure 3 As shown, the first convolutional neural network model is an improved MobileFaceNet network model, which includes the first inception layer, the first attention module, and the first shallow network, the first middle network, the first deep network, and the first fully connected layer connected in sequence; and the activation function of the improved MobileFaceNet network model is the FReLU activation function. Among them, the first inception layer is set between the first shallow network and the first middle network; the first attention module is set between the first middle network and the first deep network, and between the first deep network and the first fully connected layer network.

[0051] Specifically, the MobileFaceNet network model is composed of a normal convolution module, a separable convolution module, a bottleneck layer (i.e., a Bottleneck layer), and a global separable convolution. According to its function, the MobileFaceNet network model can be divided into a shallow network, a middle network, and a deep network. The feature map corresponding to the shallow network is a shallow feature map, the feature map corresponding to the middle network is a middle feature map, and the feature map corresponding to the deep network is a deep feature map. Among them, the division of the shallow network, the middle network, and the deep network is determined according to the depth of the feature map extracted, and the division method may be different in different conditions or scenarios.

[0052] Please refer to Figure 3 The improved MobileFaceNet network model includes the first inception layer, the first attention module, and the first shallow network, the first middle network, the first deep network, and the first fully connected layer connected in sequence. The first shallow network can be a common convolution module (i.e. Figure 3 convblock in ); the first middle layer network can be a separable convolutional module (i.e. Figure 3 The first DepthWise in , and the first deep network can be a separable convolutional module (i.e. Figure 3 The second DepthWise in is usually Global DepthWwise, i.e., a globally separable convolution module). The first fully connected layer is Figure 3 FC in.

[0053] The bottleneck layer can be combined with the first separable convolutional module Figure 3 The first DepthWise in the above example) together form the first middle layer network, and can also be combined with the second separable convolutional module (i.e. Figure 3 The second DepthWise in the network) together constitute the first deep network.

[0054] Secondly, in the first inception layer (i.e. Figure 3 The Inception in the first inception layer is set in the inception layer between the first shallow network and the first middle network; the first attention module (i.e. Figure 3 The HOA in ( ) is set between the first middle layer network and the first deep layer network, and between the first deep network and the first fully connected layer network.

[0055] Among them, Inception is also called GoogLeNet. It improves network performance by increasing the width of the network. In each inception module, convolution kernels of different sizes are used, which can be understood as different receptive fields, and then they are concat to enrich the information of each layer. After that, the BN algorithm is used (BN is used after conv and before relu) to accelerate the convergence speed of the network.

[0056] Optionally, Inception can be Inception v1, Inception v2, Inception v3, or Inception v4.

[0057] In addition, the number of inceptions in the first inception layer can be one or more. Preferably, the number of inceptions is 3.

[0058] The first attention module is a network structure based on the attention mechanism. Its main principle is inspired by the human visual mechanism, allowing the network to learn more features. The first attention module is used to extract important features of images or pictures.

[0059] In a preferred embodiment, the first attention module can be High-Order Attention, and the high-order attention module (i.e. Figure 3The HOA module has an adaptable receptive field and dynamic weights. HOA provides a global attention map for each pixel, which models the relationship between the pixel and all other pixels. The HOA module constructs the attention map through graph propagation to obtain high-order relevant context information. The feature map of each position is the weighted sum of the feature maps of all positions. The receptive field of the network propagated with the HOA module is more accurate.

[0060] Optionally, the first attention module can also be a self-attention module, a soft-attention module or a hard-attention module, where the self-attention module can include channel attention or spatial attention.

[0061] In addition, the FReLu activation function is a visual activation function, also known as Funnel ReLU. By using spatial conditions / 2D funnel conditions in the activation function (which only incurs negligible computational overhead and is very easy to implement), ReLU / PReLU can be extended to a visual parameter activation function with pixel-level modeling capabilities to extract the fine spatial layout of an object. FReLU can better capture irregular and detailed object layouts (spatial information), and its performance is better than activation functions such as ReLU, PReLU, and Swish.

[0062] In another preferred embodiment, Figure 4 As shown, the second convolutional neural network model is an improved Resnet network model, which includes a second inception layer, a second attention module, and a second shallow network, a second middle network, a second deep network, and a second fully connected layer connected in sequence; and the activation function of the improved Resnet network model is a FReLU activation function. Among them, the second inception layer is set in the second shallow network, and the second attention module is set between the second shallow network and the second middle network, and between the second middle network and the second deep network.

[0063] Specifically, the Resnet network model, also known as the residual neural network, refers to the idea of ​​adding residual learning to the traditional convolutional neural network, which solves the problems of gradient diffusion and precision degradation (training set) in deep networks, allowing the network to become deeper and deeper, while ensuring both accuracy and speed.

[0064] The Resnet network model usually includes a 5-layer convolution layer structure, a pooling layer, and a fully connected layer, wherein the 5-layer convolution layer structures are conv1_x, conv2_x, conv3_x, conv4_x, and conv5_x. The 5-layer convolution layer structure includes ordinary convolution modules and residual convolution modules, wherein the number of ordinary convolution modules and residual convolution modules can be multiple, and different numbers of ordinary convolution modules and residual convolution modules form different Resnet network model structures, such as Resnet18, Resnet50, and Resnet101.

[0065] Secondly, according to their functions, the Resnet network model can be divided into shallow network, middle network and deep network, where the feature map corresponding to the shallow network is the shallow feature map, the feature map corresponding to the middle network is the middle feature map, and the feature map corresponding to the deep network is the deep feature map. The division of shallow network, middle network and deep network is determined according to the depth of the feature map extracted, and the division method may be different in different conditions or scenarios.

[0066] Please refer to 4. The improved Resnet network model includes a second inception layer, a second attention module, and a second shallow network, a second middle network, a second deep network, and a second fully connected layer connected in sequence. The second shallow network can be a common convolution module (i.e. Figure 4 conv1 in ) and the residual convolution module (i.e. Figure 4 The second middle layer network can be a residual convolution module (i.e. Figure 4 layer2 in ), and the second deep network can be a residual convolution module (i.e. Figure 4 The second fully connected layer is layer3,4 in Figure 4 FC in.

[0067] Secondly, in the second inception layer (i.e. Figure 4 The Inception in the second shallow network is set between conv1 and layer1; the second attention module (i.e. Figure 4 The HOA in the network is set between the second shallow network and the second middle network, and between the second middle network and the second deep network.

[0068] Among them, Inception is also called GoogLeNet. It improves network performance by increasing the width of the network. In each inception module, convolution kernels of different sizes are used, which can be understood as different receptive fields, and then they are concat to enrich the information of each layer. After that, the BN algorithm is used (BN is used after conv and before relu) to accelerate the convergence speed of the network.

[0069] In addition, the number of inceptions in the second inception layer can be one or more. Preferably, the number of inceptions is 3.

[0070] Optionally, Inception can be Inception v1, Inception v2, Inception v3, or Inception v4.

[0071] The second attention module is a network structure based on the attention mechanism. Its main principle is inspired by the human visual mechanism, allowing the network to learn more features. The second attention module is used to extract important features of images or pictures.

[0072] In a preferred embodiment, the second attention module can be High-Order Attention. Figure 4 in the HOA).

[0073] Optionally, the second attention module can also be a self-attention module, a soft-attention module or a hard-attention module, where the self-attention module can include channel attention or spatial attention.

[0074] In addition, model training is to give an input vector and a target output value, and then input the input vector into one or more network structures or functions to obtain the actual output value, and calculate the bias based on the target output value and the actual output value, and determine whether the bias is within the allowable range; if it is within the allowable range, the training ends and the relevant parameters are fixed; if it is not within the allowable range, some parameters in the network structure or function are continuously adjusted until the bias is within the allowable range or a certain end condition is reached, the training ends and the relevant parameters are fixed, and finally the trained model can be obtained based on the fixed relevant parameters.

[0075] The training of the first convolutional neural network model and the second convolutional neural network model in this embodiment is actually: using image samples in the face image set to learn and train the first convolutional neural network model and the second convolutional neural network model, so as to obtain a pre-trained first convolutional neural network model and a pre-trained second convolutional neural network model. In this embodiment, the first convolutional neural network model adopts an improved MobileFaceNet network model, and the second convolutional neural network model adopts an improved Resnet network model, which can make the model training speed fast on the one hand, and the model accuracy high on the other hand, thereby improving the accuracy of face recognition.

[0076] Furthermore, an implementation method for training a second convolutional neural network model is provided, which is described in detail below.

[0077] In one embodiment, the pre-trained second convolutional neural network model is obtained by:

[0078] The second convolutional neural network model is trained based on the face image set without masks to obtain the initial second convolutional neural network model. Based on the mixed face image set with and without masks, the initial second convolutional neural network model is fine-tuned using the triplet loss method to obtain the pre-trained second convolutional neural network model.

[0079] Specifically, when the pre-trained second convolutional neural network model extracts features from a face image and obtains the features of the second image, it may extract features from the entire face image without a mask or from the entire face image with a mask. Therefore, in order to ensure that the second convolutional neural network model is prepared for feature extraction of both the entire face image without a mask and the entire face image with a mask, improvements are made when training the second convolutional neural network model. The specific steps are: training the second convolutional neural network model based on a set of face images without a mask to obtain an initial second convolutional neural network model. Based on a mixed set of face images with and without masks, the initial second convolutional neural network model is fine-tuned using the triplet loss method to obtain a pre-trained second convolutional neural network model.

[0080] Optionally, a triplet loss method is used to fine-tune the initial second convolutional neural network model, including: setting the anchor to a face image wearing a mask, setting the positive to a face image not wearing a mask, and setting the negative to a face image not wearing a mask or a face image wearing a mask; wherein the anchor and the positive face images are from the same person, while the negative face image and the anchor and positive face images are from different people.

[0081] Among them, the set of face images without masks includes multiple face images of people without masks; the set of mixed face images of people wearing masks and not wearing masks includes multiple face images of people without masks and multiple face images of people wearing masks.

[0082] In this embodiment, the second convolutional neural network is first trained using a set of face images without masks, and after the training is completed, an initial second convolutional neural network model is obtained. After the initial second convolutional neural network model is obtained, the initial second convolutional neural network model is fine-tuned based on a mixed set of face images with and without masks, and a triplet loss method is used to obtain the pre-trained second convolutional neural network model. By adopting this training method, on the one hand, the model can converge quickly, thereby improving the speed of model training; on the other hand, when the triplet loss method is used to fine-tune the initial second convolutional neural network model, the anchor is set to a face image with a mask, the positive is set to a face image without a mask, and the negative is set to a face image without a mask or a face image with a mask, which can improve the accuracy of the second convolutional neural network model.

[0083] Step S150: Fusing the first image feature and the second image feature to obtain a fused image feature.

[0084] After obtaining the first image feature and the second image feature, they are fused to obtain a fused image feature.

[0085] In addition, the dimensions of the first image feature and the second image feature may be the same or different. For example, the dimensions of the first image feature and the second image feature may both be 512 or 256; or the dimension of the first image feature may be 512, while the dimension of the second image feature may be 256.

[0086] Step S160, comparing the fused image features with the facial features in the face database, and determining whether the facial recognition of the user to be identified is passed based on the comparison result.

[0087] In one embodiment, the fused image features are compared with the facial features in the face library, and whether the facial recognition of the user to be identified is passed is determined based on the comparison result, including: calculating the cosine distance value between the fused image features and the facial features; when the cosine distance value is greater than a preset threshold, it is determined that the facial recognition of the user to be identified is passed.

[0088] The preset threshold is a preset value, which is usually a range value.

[0089] The face recognition method provided in the embodiment of the present application first obtains a face image of a user to be identified, wherein the face image can be a face image of a person wearing a mask or a face image of a person not wearing a mask; then, an upper face image is extracted from the face image, and the upper face image is input into a pre-trained first convolutional neural network model to obtain a first image feature; at the same time, the face image is input into a pre-trained second convolutional neural network model to obtain a second image feature; the first image feature and the second image feature are fused to obtain a fused image feature; the fused image feature is compared with the face feature in the face library, and whether the face recognition of the user to be identified is passed is determined according to the comparison result.

[0090] The face recognition method uses the first convolutional neural network model and the second convolutional neural network model to extract the upper face features and the full face features, respectively, and then fuses the upper face features and the full face features, and then compares the fused features with the face features in the face library to determine whether the face recognition is passed. The fusion features formed by the feature fusion method improve the expression and identification capabilities of the face, thereby improving the accuracy of face recognition. In addition, this method can be used to identify users wearing masks, and users who are not wearing masks, which is very convenient to use.

[0091] It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0092] A face recognition method is described in detail in the embodiments disclosed in the above application. The above method disclosed in the present application can be implemented using various forms of equipment. Therefore, the present application also discloses a face recognition device corresponding to the above method. A specific embodiment is given below for detailed description.

[0093] See also Figure 5 , is a face recognition device disclosed in an embodiment of the present application, mainly comprising:

[0094] The face image acquisition module 510 is used to acquire the face image of the user to be identified;

[0095] An upper face image extraction module 520 is used to extract an upper face image from a face image;

[0096] A first feature acquisition module 530, configured to input the upper face image into a pre-trained first convolutional neural network model to obtain a first image feature;

[0097] A second feature acquisition module 540, configured to input the face image into a pre-trained second convolutional neural network model to obtain a second image feature;

[0098] A fusion feature acquisition module 550 is used to fuse the first image feature and the second image feature to obtain a fusion image feature;

[0099] The comparison module 560 is used to compare the fused image features with the facial features in the face database, and determine whether the facial recognition of the user to be identified is passed according to the comparison result.

[0100] In one embodiment, the first convolutional neural network model and the second convolutional neural network model include a ResNet network model, a VGG network model, a DenseNet network model, a SqueezeNet network model, or a MobileFaceNet network model.

[0101] In one embodiment, the first convolutional neural network model is an improved MobileFaceNet network model, which includes a first inception layer, a first attention module, and a first shallow network, a first middle network, a first deep network, and a first fully connected layer connected in sequence; and the activation function of the improved MobileFaceNet network model is a FReLU activation function; wherein, the first inception layer is arranged between the first shallow network and the first middle network; the first attention module is arranged between the first middle network and the first deep network, and between the first deep network and the first fully connected layer network.

[0102] In one embodiment, the second convolutional neural network model is an improved Resnet network model, which includes a second inception layer, a second attention module, and a second shallow network, a second middle network, a second deep network, and a second fully connected layer connected in sequence; and the activation function of the improved Resnet network model is a FReLU activation function; wherein, the second inception layer is arranged between the second shallow network and the second middle network; the second attention module is arranged between the second middle network and the second deep network, and between the second deep network and the second fully connected layer network.

[0103] In one embodiment, the apparatus comprises:

[0104] The first training module is used to train the second convolutional neural network model based on a set of facial images of people who are not wearing masks to obtain an initial second convolutional neural network model.

[0105] The second training module is used to fine-tune the initial second convolutional neural network model based on a mixed set of face images with and without masks, and adopts a triplet loss method to obtain a pre-trained second convolutional neural network model.

[0106] In one embodiment, the second model training module is used to set the anchor to a face image of a person wearing a mask, set the positive to a face image of a person not wearing a mask, and set the negative to a face image of a person not wearing a mask or a face image of a person wearing a mask; wherein the anchor and the positive face images are from the same person, and the negative face image and the anchor and positive face images are from different people.

[0107] In one embodiment, the fusion feature acquisition module is used to calculate the cosine distance value between the fusion image feature and the face feature; when the cosine distance value is greater than a preset threshold, it is determined that the face recognition of the user to be identified is passed.

[0108] For the specific definition of the face recognition device, please refer to the definition of the method above, which will not be repeated here. Each module in the above device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the terminal device in hardware form, or stored in the memory of the terminal device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0109] Please refer to Figure 6 , Figure 6 It shows a structural block diagram of a terminal device provided in an embodiment of the present application. The terminal device 60 may be a computer device. The terminal device 60 in the present application may include one or more of the following components: a processor 62, a memory 64, and one or more applications, wherein the one or more applications may be stored in the memory 64 and configured to be executed by one or more processors 62, and the one or more applications are configured to execute the method described in the above method embodiment applied to the terminal device, and may also be configured to execute the method described in the above method embodiment applied to the face recognition method.

[0110] The processor 62 may include one or more processing cores. The processor 62 uses various interfaces and lines to connect the various parts of the entire terminal device 60, and executes various functions and processes data of the terminal device 60 by running or executing instructions, programs, code sets or instruction sets stored in the memory 64, and calling data stored in the memory 64. Optionally, the processor 62 can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor 62 can integrate one or more combinations of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 62, but may be implemented separately through a communication chip.

[0111] The memory 64 may include a random access memory (RAM) or a read-only memory (ROM). The memory 64 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 64 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data created by the terminal device 60 during use, etc.

[0112] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the terminal device to which the scheme of the present application is applied. The specific terminal device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0113] In summary, the terminal device provided in the embodiment of the present application is used to implement the corresponding face recognition method in the aforementioned method embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0114] See also Figure 7, which shows a block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable storage medium 70 stores program code, which can be called by a processor to execute the method described in the above face recognition method embodiment, or can be called by a processor to execute the method described in the above face recognition method embodiment.

[0115] The computer readable storage medium 70 can be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 70 includes a non-transitory computer-readable storage medium. The computer readable storage medium 70 has storage space for program code 72 that performs any method steps in the above method. These program codes can be read from or written to one or more computer program products. The program code 72 can be compressed, for example, in a suitable form.

[0116] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0117] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A face recognition method, characterized in that: The method comprises: Obtain a facial image of the user to be identified; Extracting an upper face image from the face image; Inputting the upper face image into a pre-trained first convolutional neural network model to obtain a first image feature; Inputting the face image into a pre-trained second convolutional neural network model to obtain a second image feature; fusing the first image feature and the second image feature to obtain a fused image feature; Comparing the fused image features with the facial features in the face database, and determining whether the facial recognition of the user to be identified is passed according to the comparison result; The first convolutional neural network model is an improved MobileFaceNet network model, which includes a first inception layer, a first attention module, and a first shallow network, a first middle network, a first deep network, and a first fully connected layer connected in sequence; and the activation function of the improved MobileFaceNet network model is a FReLU activation function; The first inception layer is arranged between the first shallow network and the first middle network; the first attention module is arranged between the first middle network and the first deep network, and between the first deep network and the first fully connected layer network; The second convolutional neural network model is an improved Resnet network model, which includes a second inception layer, a second attention module, and a second shallow network, a second middle network, a second deep network, and a second fully connected layer connected in sequence; and the activation function of the improved Resnet network model is a FReLU activation function; The second inception layer is arranged in the second shallow network, and the second attention module is arranged between the second shallow network and the second middle network, and between the second middle network and the second deep network.

2. The method according to claim 1, characterized in that The first convolutional neural network model and the second convolutional neural network model include a ResNet network model, a VGG network model, a DenseNet network model, a SqueezeNet network model or a MobileFaceNet network model.

3. The method according to any one of claims 1 to 2, characterized in that: The pre-trained second convolutional neural network model is obtained by: Training the second convolutional neural network model based on a set of facial images of people not wearing masks to obtain an initial second convolutional neural network model; Based on a mixed set of face images with and without masks, the initial second convolutional neural network model is fine-tuned using a triplet loss method to obtain the pre-trained second convolutional neural network model.

4. The method according to claim 3, characterized in that: The triplet loss method is used to fine-tune the initial second convolutional neural network model, including: Set anchor to the face image of a person wearing a mask, set positive to the face image of a person not wearing a mask, and set negative to the face image of a person not wearing a mask or a face image of a person wearing a mask; The anchor and positive face images are from the same person, while the negative face image and the anchor and positive face images are from different people.

5. The method according to any one of claims 1-2, characterized in that: The step of comparing the fused image features with the facial features in the facial database and determining whether the facial recognition of the user to be identified is passed according to the comparison result includes: Calculating the cosine distance value between the fused image feature and the facial feature; When the cosine distance value is greater than a preset threshold, it is determined that the face recognition of the user to be identified is passed.

6. A face recognition device, characterized in that: The device comprises: A face image acquisition module, used to acquire a face image of a user to be identified; An upper face image extraction module, used to extract an upper face image from the face image; A first feature acquisition module, used for inputting the upper face image into a pre-trained first convolutional neural network model to obtain a first image feature; A second feature acquisition module, used for inputting the face image into a pre-trained second convolutional neural network model to obtain a second image feature; a fusion feature acquisition module, used for fusing the first image feature and the second image feature to obtain a fusion image feature; A comparison module, used to compare the fused image features with the facial features in the face database, and determine whether the facial recognition of the user to be identified is passed according to the comparison result; The first convolutional neural network model is an improved MobileFaceNet network model, which includes a first inception layer, a first attention module, and a first shallow network, a first middle network, a first deep network, and a first fully connected layer connected in sequence; and the activation function of the improved MobileFaceNet network model is a FReLU activation function; wherein the first inception layer is arranged between the first shallow network and the first middle network; the first attention module is arranged between the first middle network and the first deep network, and between the first deep network and the first fully connected layer network; The second convolutional neural network model is an improved Resnet network model, which includes a second inception layer, a second attention module, and a second shallow network, a second middle network, a second deep network, and a second fully connected layer connected in sequence; and the activation function of the improved Resnet network model is a FReLU activation function; wherein, the second inception layer is arranged between the second shallow network and the second middle network; the second attention module is arranged between the second middle network and the second deep network, and between the second deep network and the second fully connected layer network.

7. A terminal device, characterized in that: include: Memory; One or more processors coupled to the memory; One or more applications, wherein the one or more applications are stored in a memory and configured to be executed by one or more processors, and the one or more applications are configured to execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 5.

Images