Face recognition method, device, equipment and storage medium based on dual feature comparison

Abstract

The present invention proposes a face recognition method, device, equipment, and storage medium based on dual feature comparison. The method includes: obtaining a target depth feature vector and a target shape feature vector based on a face image to be recognized extracted by MTCNN, determining a first similarity based on the target shape feature vector and a first fusion weight, and determining a second similarity based on the target depth feature vector and a second fusion weight. When the reliability of the deep features is affected by low resolution and environmental factors, multiple candidate face images can be matched based on the fusion similarity obtained by summing up; the values ​​of the first fusion weight and the second fusion weight are adjusted while maintaining the sum of the two to be 1, and the target recognition result is determined using the change trend and amplitude of the fusion similarity after updating, thereby improving the reliability and accuracy of face recognition.

Description

Technical Field

[0001] The present invention relates to the field of face recognition technology, and in particular to a face recognition method, device, equipment and storage medium based on dual feature comparison. Background Art

[0002] With the popularization of artificial intelligence technology, face recognition technology has been integrated into video surveillance, and face detection and identity recognition are performed based on video frames, and compared with pre-entered owner information to achieve intelligent security. Video surveillance in the property field has the characteristics of a large number of faces and fast movement speed, so it is necessary to quickly locate multiple faces in real time in the video frame and ensure the accuracy of face recognition.

[0003] In related technologies, in order to meet the real-time and accurate monitoring requirements, a multi-task convolutional neural network (MTCNN) and an identity recognition network (FaceNet) are usually integrated. MTCNN is used to perform face detection and generate facial target key points around the facial features. After aligning the face image based on the facial target key points, deep features are extracted through FaceNet, and face recognition is completed based on the deep features.

[0004] However, due to cost constraints, the video surveillance equipment for property management usually has low resolution of video frames, and the environments and occlusions of different monitoring areas are also different. This causes face recognition to be affected by lighting changes, low image resolution, and many obstructions. The extracted deep features have high noise, which affects the accuracy of face recognition. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes a face recognition method, device, equipment, and storage medium based on dual feature comparison, which can integrate multiple types of features for face recognition and improve the accuracy of face recognition.

[0006] In a first aspect, an embodiment of the present invention provides a face recognition method based on dual feature comparison, which is applied to an intelligent monitoring system, wherein the intelligent monitoring system is preset with a sample image library, a multi-task convolutional neural network MTCNN, an identity recognition network FaceNet and a fusion comparison module, wherein the sample image library is preset with a plurality of sample face images, and the sample face images are preset with a sample shape feature vector and a sample depth feature vector, and the method comprises:

[0007] Detecting a face image to be identified from a target video frame based on MTCNN, extracting a target depth feature vector from the face image to be identified through FaceNet and inputting the vector into the fusion comparison module;

[0008] Extracting target shape vector features based on the face image to be identified by the fusion comparison module, wherein the target shape vector features are used to indicate the shape of facial features;

[0009] Traversing the sample image library through the fusion comparison module, each time traversing a sample face image, determining a first similarity based on the target shape feature vector and the corresponding sample shape feature vector, determining a second similarity based on the target depth feature vector and the corresponding sample depth feature vector, and determining a fusion similarity based on the first similarity and the second similarity, wherein the sum of a first fusion weight of the first similarity and a second fusion weight of the second similarity is 1;

[0010] Based on the fusion similarity, multiple candidate face images are determined, and each of the fusion similarities is updated by adjusting the numerical values ​​of the first fusion weight and the second fusion weight. The target recognition result is determined based on the one with the largest numerical change among the multiple fusion similarities with the same numerical change direction as the first fusion weight, wherein the sum of the first fusion weight and the second fusion weight is maintained at 1.

[0011] According to some embodiments of the present invention, extracting target shape vector features based on the face image to be recognized by the fusion comparison module includes:

[0012] Determining a plurality of target key points in the face image to be identified, wherein the plurality of target key points are used to characterize a plurality of facial feature areas of the face image to be identified;

[0013] The target key points are arranged based on a preset order, and a target shape feature vector is constructed based on the image coordinates of all the target key points, wherein the preset order is used to indicate the arrangement order of the multiple facial feature areas, and the horizontal coordinate and the vertical coordinate of the target key point are respectively a vector element of the target shape feature vector.

[0014] According to some embodiments of the present invention, each of the sample shape feature vectors has the same number of elements, and determining a plurality of target key points in the face image to be recognized includes:

[0015] Determine a plurality of facial feature regions in the face image to be recognized based on the MTCNN, generate a plurality of candidate key points in each of the facial feature regions, and determine a first coordinate number of each of the facial feature regions, wherein the first coordinate number is used to represent the number of the candidate key points in the facial feature regions;

[0016] Obtaining a preset coordinate number set, and traversing each of the facial feature regions based on the coordinate number set, wherein the coordinate number set records a plurality of second coordinate numbers, each of the second coordinate numbers corresponds to one of the facial feature regions, and the plurality of second coordinate numbers are arranged in the coordinate number set based on the preset order;

[0017] When the first coordinate number corresponding to the traversed facial feature area is equal to the second coordinate number, determining the candidate key point as the target key point;

[0018] Alternatively, when the first coordinate number corresponding to the traversed facial feature area is not equal to the second coordinate number, the number of candidate key points in the traversed facial feature area is adjusted based on the difference between the second coordinate number and the first coordinate number, the position of each candidate key point is adjusted based on the principle of equidistant distribution, and all the adjusted candidate key points are determined as the target key points.

[0019] According to some embodiments of the present invention, determining a first similarity based on the target shape feature vector and the corresponding sample shape feature vector, determining a second similarity based on the target depth feature vector and the corresponding sample depth feature vector, and determining a fusion similarity based on the first similarity and the second similarity include:

[0020] Determine twice the sum of each of the second coordinate numbers in the coordinate number set as a first value, and obtain a preset second value, wherein the second value is the number of dimensions of the target depth feature vector;

[0021] Determine a first norm of the target shape feature vector based on the first value, and determine a second norm of the target depth feature vector based on the second value;

[0022] Based on the first norm, the target shape feature vector is normalized to obtain a standard shape feature vector, and based on the second norm, the target depth feature vector is normalized to obtain a standard depth feature vector;

[0023] The cosine similarity between the standard shape feature vector and the sample shape feature vector is determined as the first similarity, and the cosine similarity between the standard depth feature vector and the sample depth feature vector is determined as the second similarity, wherein the sample shape feature vector and the sample depth feature vector are both feature vectors obtained after being standardized based on corresponding norms.

[0024] According to some embodiments of the present invention, a plurality of candidate face images are determined based on the fusion similarity, and each of the fusion similarities is updated by adjusting the values ​​of the first fusion weight and the second fusion weight, including:

[0025] Determining the fusion similarity greater than a preset similarity threshold as a target similarity;

[0026] When the number of the target similarities is greater than 1, determining the sample face images corresponding to the respective target similarities as the candidate face images;

[0027] The first fusion weight and the second fusion weight are adjusted based on a preset step value, and the target similarity of each of the candidate face images is recalculated.

[0028] According to some embodiments of the present invention, the MTCNN includes a P-Net, an R-Net, and an O-Net that are cascaded in sequence, and adjusting the first fusion weight and the second fusion weight based on a preset step value includes:

[0029] Obtaining a candidate face region detected by the P-Net from the target video frame, and determining an image proportion of the face image to be recognized in the candidate face region;

[0030] When the image proportion is greater than or equal to a preset proportion threshold, reducing the first fusion weight based on the step value, and increasing the second fusion weight based on the step value;

[0031] Alternatively, when the image proportion is less than the proportion threshold, the first fusion weight is increased based on the step value, and the second fusion weight is reduced based on the step value.

[0032] According to some embodiments of the present invention, determining a target recognition result based on the one with the largest value change among a plurality of the fusion similarities having the same value change direction as the first fusion weight value, includes:

[0033] When the target similarities increase or decrease simultaneously after updating, determining the target face image based on the target similarity with the largest value after updating;

[0034] Alternatively, when at least one of the target similarities increases in value after updating, and at least one of the target similarities decreases in value after updating, the target facial image is determined by the target similarities with the largest value change in the same direction as the first fusion weight value change.

[0035] In a second aspect, an embodiment of the present invention provides a face recognition device based on dual feature comparison, comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the face recognition method based on dual feature comparison as described in the first aspect above.

[0036] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a face recognition device based on dual feature comparison as described in the second aspect above.

[0037] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the face recognition method based on dual feature comparison as described in the first aspect above.

[0038] The face recognition method based on dual feature comparison according to the embodiment of the present invention has at least the following beneficial effects: based on MTCNN, a face image to be recognized is detected from a target video frame, and a target depth feature vector is extracted from the face image to be recognized through the FaceNet and input into the fusion comparison module; the target shape vector feature is extracted based on the face image to be recognized through the fusion comparison module, wherein the target shape vector feature is used to indicate the shape of facial features; the sample image library is traversed through the fusion comparison module, and whenever a sample face image is traversed, a first phase is determined based on the target shape feature vector and the corresponding sample shape feature vector. similarity, determining a second similarity based on the target depth feature vector and the corresponding sample depth feature vector, determining a fusion similarity based on the first similarity and the second similarity, wherein the sum of the first fusion weight of the first similarity and the second fusion weight of the second similarity is 1; determining multiple candidate face images based on the fusion similarity, updating each of the fusion similarities by adjusting the values ​​of the first fusion weight and the second fusion weight, and determining a target recognition result based on the maximum value change of multiple fusion similarities in the same direction of value change as the first fusion weight, wherein the sum of the first fusion weight and the second fusion weight remains 1. According to the technical solution of the embodiment of the present invention, it is possible to identify multiple candidate face images by combining shape features and depth features in the case where the reliability of the depth features is affected by low resolution and environmental factors, and realize face recognition by using the change trend and amplitude of the fusion similarities of different candidate face images after adjusting the fusion weight, thereby improving the reliability and accuracy of face recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a schematic diagram of a face recognition method based on dual feature comparison provided by an embodiment of the present invention;

[0040] Figure 2 is a flow chart of a face recognition method based on dual feature comparison provided by another embodiment of the present invention;

[0041] Figure 3is a schematic diagram of target key points and facial features areas provided by another embodiment of the present invention;

[0042] Figure 4 It is a structural diagram of a face recognition device based on dual feature comparison provided by another embodiment of the present invention. DETAILED DESCRIPTION

[0043] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0044] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0045] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0046] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0047] The embodiment of the present invention provides a face recognition method, device, equipment, and storage medium based on dual feature comparison, wherein the face recognition method based on dual feature comparison includes: detecting a face image to be recognized from a target video frame based on MTCNN, extracting a target depth feature vector from the face image to be recognized through the FaceNet and inputting it into the fusion comparison module; extracting a target shape vector feature based on the face image to be recognized through the fusion comparison module, wherein the target shape vector feature is used to indicate the shape of facial features; traversing the sample image library through the fusion comparison module, and whenever traversing to a sample face image, comparing the target shape feature vector with the corresponding sample face image based on the target shape feature vector. The first similarity is determined based on the shape feature vector, the second similarity is determined based on the target depth feature vector and the corresponding sample depth feature vector, and the fusion similarity is determined based on the first similarity and the second similarity, wherein the sum of the first fusion weight of the first similarity and the second fusion weight of the second similarity is 1; multiple candidate face images are determined based on the fusion similarity, and each of the fusion similarities is updated by adjusting the values ​​of the first fusion weight and the second fusion weight, and the target recognition result is determined based on the maximum value change of the multiple fusion similarities in the same direction as the first fusion weight value change, wherein the sum of the first fusion weight and the second fusion weight is maintained as 1. According to the technical solution of the embodiment of the present invention, in the case of low resolution and environmental factors affecting the reliability of the depth feature, multiple candidate face images can be identified by combining shape features and depth features, and face recognition is realized by using the change trend and amplitude of the fusion similarity of different candidate face images after adjusting the fusion weight, thereby improving the reliability and accuracy of face recognition.

[0048] First, refer to Figure 1 , Figure 1 A schematic diagram of a face recognition method based on dual feature comparison provided in an embodiment of the present invention. The intelligent monitoring system of this embodiment is preset with a sample image library 20 and a face recognition model 10. The face recognition model 10 includes MTCNN, FaceNet and a fusion comparison module.

[0049] It should be noted that the intelligent monitoring system is also connected to multiple monitoring devices 30, and the multiple monitoring devices 30 send monitoring video streams to the intelligent monitoring system. The intelligent monitoring system obtains monitoring video frames based on each monitoring video stream, and executes the technical solutions of this embodiment respectively with each monitoring video frame as the target video frame. Those skilled in the art are familiar with how to build the monitoring device 30, and are familiar with how to obtain monitoring video frames respectively from multiple monitoring video streams, and will not be elaborated here.

[0050] It should be noted that the sample image library 20 can be a common database, such as a residential property owner face database. The sample graphic library 20 is preset with multiple sample face images. Each sample face image is pre-entered and preset with corresponding identity information. Those skilled in the art are familiar with how to enter sample face images in advance, and will not elaborate on them here.

[0051] It is worth noting that the sample face image of this embodiment is preset with a sample shape feature vector and a sample depth feature vector. The sample depth feature vector can be obtained by conventional model training, which will not be described in detail here. The sample shape feature vector can be obtained by extracting key points through MTCNN after entering the sample face image, and then arranging the coordinates of the extracted key points.

[0052] It should be noted that if Figure 1 As shown, MTCNN includes P-Net, R-Net and O-Net which are cascaded in sequence. P-Net is used to extract candidate face regions from the input image, R-Net is used to fine-tune candidate face regions and filter out erroneous regions, and O-Net is used to further fine-tune and output key point positions. The structures and principles of MTCNN and FaceNet in this embodiment are both prior art, and this embodiment does not involve improvements to MTCNN and FaceNet.

[0053] like Figure 1 As shown, the monitoring device 30 sends a monitoring video stream to the intelligent monitoring system. The intelligent monitoring system obtains the target video frame and inputs it into the face recognition model 10. The face recognition model 10 extracts the face image to be recognized and the target key points through MTCNN, and inputs the face image to be recognized into FaceNet and the fusion comparison module respectively. FaceNet extracts the depth feature based on the face image to be recognized to obtain the target depth feature vector, and inputs the target depth feature vector into the fusion comparison module. In the fusion comparison module, the target key points are sorted according to a preset order, and the corresponding coordinate points are inserted after sorting. The horizontal coordinates and vertical coordinates of multiple target key points are alternately arranged into a series to determine the target shape feature vector. The fusion comparison module traverses the sample face images from the sample database, determines the fusion similarity with each sample face image through the target shape feature vector and the target depth feature vector, determines the target recognition result according to the fusion similarity, and adds shape features when the depth feature is unreliable to improve the accuracy of face recognition.

[0054] The following is based on the Figure 1 The intelligent monitoring system shown further illustrates the technical solution of the embodiment of the present invention.

[0055] Reference Figure 2 , Figure 2A flow chart of a face recognition method based on dual feature comparison provided by an embodiment of the present invention, the face recognition method based on dual feature comparison includes but is not limited to the following steps:

[0056] S10, detect the face image to be identified from the target video frame based on MTCNN, extract the target deep feature vector from the face image to be identified through FaceNet and input it into the fusion comparison module;

[0057] S20, extracting target shape vector features based on the face image to be recognized by a fusion comparison module, wherein the target shape vector features are used to indicate the shape of facial features;

[0058] S30, traversing the sample image library through the fusion comparison module, and whenever traversing a sample face image, determining a first similarity based on the target shape feature vector and the corresponding sample shape feature vector, determining a second similarity based on the target depth feature vector and the corresponding sample depth feature vector, and determining a fusion similarity based on the first similarity and the second similarity, wherein the sum of a first fusion weight of the first similarity and a second fusion weight of the second similarity is 1;

[0059] S40, determine multiple candidate face images based on fusion similarity, update each fusion similarity by adjusting the values ​​of the first fusion weight and the second fusion weight, and determine the target recognition result based on the one with the largest value change among multiple fusion similarities with the same value change direction as the first fusion weight, wherein the sum of the first fusion weight and the second fusion weight is maintained at 1.

[0060] It should be noted that after acquiring the target video frame, MTCNN can detect multiple candidate face regions from the target video frame, each candidate face region corresponds to a user, and the technical solution of this embodiment can be executed independently once for each candidate face region, and will not be repeated later.

[0061] It should be noted that FaceNet extracts deep features based on the face image to be identified, which is a technology well known to those skilled in the art. The specific feature extraction process will not be described in detail here. After obtaining the target deep feature vector, it can be input into the fusion comparison module. This embodiment adds a fusion comparison module on the basis of MTCNN and FaceNet. The face image to be identified obtained by MTCNN is input into the fusion comparison module, and the target deep feature vector obtained by FaceNet is also input into the fusion comparison module.

[0062] It should be noted that MTCNN can extract key points, such as Figure 3After extracting multiple target key points 42, target shape vector features are constructed using the target key points. Each target shape vector feature is used to describe the outline of a facial feature area. For example, the set of target key points in the ear area can describe the outline of the ear. Face alignment is usually performed in MTCNN, that is, the face in the target video frame is aligned to an angle and position that can be compared with the sample face image through operations such as rotation and cropping. Therefore, the positions of all target key points can characterize the shape of the facial features. For example, the positions of the nose areas of different people are different, and the coordinates of the target key points corresponding to the nose areas have certain differences. However, the coordinates of the target key points in the nose area of ​​the same person are highly similar in the two images. Therefore, the set formed by the positions of the target key points can be used as the shape basis for face recognition. Based on this, after the fusion comparison module of this embodiment obtains the face image to be identified, it uses the target key points determined in the face image to be identified to construct a target shape feature vector, so that the target shape feature vector can represent the shape of the facial features. Even if the image resolution is low or some areas are blocked, the depth feature will be greatly affected, but the shape of the facial features can still be better identified. Therefore, the shape feature can provide recognition reference to a large extent as a compensation mechanism for the low reliability of the depth feature. The results are compared through the two dimensions of shape features and depth features to ensure the accuracy of the target recognition results.

[0063] It should be noted that if Figure 1 As shown, after the fusion comparison module obtains the target shape feature vector and the target depth feature vector, since the sample face image also includes the sample shape feature vector and the sample depth feature vector, the similarity of the corresponding feature vectors can be calculated. In this embodiment, the similarity between the target shape feature vector and the sample shape feature vector is used as the first similarity, and the similarity between the target depth feature vector and the sample depth feature vector is used as the second similarity. The first similarity is used to characterize the similarity between the face image to be identified and the sample face image in the face shape dimension, and the second similarity is used to characterize the similarity between the face image to be identified and the sample face image in the depth feature dimension. The first similarity and the second similarity are fused to obtain the fused similarity, and the target recognition result is determined based on the fused similarity.

[0064] It should be noted that the present embodiment pre-sets the first fusion weight and the second fusion weight, and ensures that the sum of the first fusion weight and the second fusion weight is 1 and both are positive numbers, and the importance of the first similarity and the second similarity is represented by different weight values, so as to fuse the first similarity and the second similarity into a fusion similarity. The present embodiment does not limit the first fusion weight to be equal to the second fusion weight, but can flexibly adjust the values ​​of the two, using the sum of the two to be 1, and adaptively adjust the value of the first fusion weight by the value of the second fusion weight, and represent the reliability of the depth feature by setting the value of the second fusion weight. For example, in monitoring scenes with more obstructions, the reliability of the depth feature is lower, and a second fusion weight with a smaller value can be set; for another example, in monitoring scenes with less interference from the external environment, the reliability of the depth feature is higher, and a second fusion weight with a higher value can be set to improve recognition accuracy.

[0065] Exemplarily, the first fusion weight is , the second fusion weight is , the first similarity is , the second similarity is , so the fusion similarity .

[0066] It should be noted that, since the target similarity is obtained by fusing the first similarity and the second similarity, when there are multiple target similarities, it is very likely that the first similarity corresponding to some target similarities is greater than the second similarity, and the second similarity corresponding to others is greater than the first similarity. The fused similarity obtained after summing satisfies the similarity threshold. This is because the complex environment, low resolution and occlusion problems have a certain impact on the depth feature. In fact, the second similarity corresponding to the matched target face image is low, and the second similarity of the misidentified sample face image is high. Through the technical solution of this embodiment, compensation can be made through the first similarity, so that when the second similarity is affected, the correct face image can still be screened out according to the target similarity.

[0067] It should be noted that after the first fusion weight and the second fusion weight are updated, since the first fusion weight and the second fusion weight of this embodiment maintain the sum equal to 1, the adjustment directions of the two are opposite, that is, when the first fusion weight increases, the second fusion weight decreases, and vice versa.

[0068] It should be noted that when some target similarities increase and some target similarities decrease after the update, it can be determined that the reason for matching multiple target similarities is that the resolution and external factors cause the target depth feature vector to be seriously distorted, resulting in a large deviation in the second similarity. Since the shape feature is more robust under external interference, this embodiment uses the direction of numerical change as the classification condition of the target similarity, and selects the target with the largest numerical change from the target similarities with the same numerical change direction as the first fusion weight to determine the target face image. When the first fusion weight value decreases, the target similarity with a larger proportion of the first similarity will also decrease accordingly. Therefore, the target similarity has the same numerical change direction as the first fusion weight. In a scene where the depth feature is more distorted, the higher the proportion of the first similarity, the greater the numerical change amplitude of the target similarity, the greater the reliability of the target similarity, and the more reliable the target recognition result determined in this way. The principle is similar when the first fusion weight value increases, and it will not be repeated here.

[0069] In addition, in one embodiment, step S20 specifically includes but is not limited to the following steps:

[0070] S21, determining a plurality of target key points in the face image to be identified, wherein the plurality of target key points are used to represent a plurality of facial feature regions of the face image to be identified;

[0071] S22, arrange the target key points based on a preset order, and construct a target shape feature vector based on the image coordinates of all the target key points, wherein the preset order is used to indicate the arrangement order of multiple facial feature areas, and the horizontal coordinate and vertical coordinate of the target key point are respectively a vector element of the target shape feature vector.

[0072] It is worth noting that MTCNN obtains the candidate face region through P-Net. After R-Net fine-tunes the face region, the key points surrounding the facial features located by O-Net are the target key points of this embodiment. Extracting key points based on MTCNN is a technology well known to those skilled in the art and will not be described in detail here. Figure 1 As shown, the facial features area 41 preset in this embodiment includes an eye area, a nose area, a mouth area and an ear area.

[0073] For example, Figure 3 As shown, the present embodiment presets the facial feature areas 41 including the eye area, the nose area, the mouth area and the ear area, locates the candidate frames of the above-mentioned facial feature areas 41 in the candidate face area, generates multiple target key points 42 in the candidate frames based on the corresponding contours, and the target key points 42 in the same facial feature area 41 are used to characterize the external contours of the corresponding facial features.

[0074] It should be noted that after obtaining the target key points, the face image can also be preprocessed by image alignment, normalization, etc., which will not be described in detail here. In this embodiment, the output result of O-Net is determined as the face image to be identified, and an image coordinate system is established in the face image to be identified, and the image coordinates of each target key point are output by O-Net. The above principles are all conventional technical principles of O-Net, and the detailed process is not described in detail here.

[0075] It should be noted that after obtaining the face image to be identified, the shape comparison of this embodiment is based on whether the coordinate positions of the target key points are similar. Therefore, it is necessary to ensure that multiple target key points are sorted in a fixed order to avoid recognition errors caused by comparing target key points in different positions. This embodiment pre-sets a preset order, and indicates the order of the facial features by the preset order. For example, the preset order is left ear, left eye, nose, mouth, right eye, and right ear. In each facial feature area, the leftmost point is used as the starting point, and the clockwise direction is the positive direction. Based on the above preset order, multiple target key points are sorted so that the standards of multiple facial images to be identified are the same.

[0076] It should be noted that after the target key points are sorted, the image coordinates of each target key point are substituted according to the sorted sequence, and the horizontal coordinate and the vertical coordinate are respectively a vector element, and the series composed of the coordinates is determined as the target shape feature vector, that is, the number of target key points is K (K is a positive integer), and the number of elements of the target shape feature vector is 2K.

[0077] For example, Figure 3 As shown, the sequence of target key points in the left ear area is ={ , },in, is the leftmost target key point in the left ear area, for The next target keypoint in the clockwise direction, The coordinates of are ( ), The coordinates of are ( ), the subsequent related subscripts are similar, A is the number of target key points in the left ear area. Similarly, the sequence of target key points in the left eye area is ={{ , }, B is the number of target key points in the left eye area; the sequence of target key points in the nose area is ={{ , }, C is the number of target key points in the nose area; the sequence of target key points in the mouth area is ={{ , }, D is the number of target key points in the mouth area; the sequence of target key points in the right eye area is ={{ , }, E is the number of target key points in the right eye area; the sequence of target key points in the right ear area is ={{ , }, F is the number of target key points in the right eye area. The key point sequence obtained after sorting is { }, after substituting the coordinates, the target shape feature sequence is { }.

[0078] It is worth noting that when the face is unobstructed, multiple target key points can be obtained according to the above method, and arranged into a target shape feature vector based on the multiple target key points. If one or more facial feature areas are obstructed, resulting in partial or complete loss of the facial feature areas, this embodiment presets default key points for each facial feature area in the intelligent monitoring system. If a facial feature area is missing as a whole, the default key points are used to fill it, ensuring that the target shape feature sequence can be aligned with the sample shape feature sequence; if the facial feature area is partially occluded, it will cause the coordinates of some target key points to mutate or be missing, and an approximate estimate can be made based on the left side before and after that has not undergone mutation, so as to reduce the impact of local occlusion on recognition accuracy.

[0079] For example, in the intelligent monitoring system , }, when the left ear area is blocked, use As the corresponding key point.

[0080] For example, in In the sequence, The vertical coordinate of mutates, and it can be determined that the corresponding position is blocked, resulting in the occlusion being recognized as part of the left ear. Therefore, based on and The vertical coordinates are averaged The normal ordinate is estimated.

[0081] In addition, in one embodiment, the number of elements of each sample shape feature vector is the same, and step S21 specifically includes but is not limited to the following steps:

[0082] S211, determining multiple facial feature regions in the face image to be recognized based on MTCNN, generating multiple candidate key points in each facial feature region, and determining a first coordinate number of each facial feature region, wherein the first coordinate number is used to represent the number of candidate key points in the facial feature region;

[0083] S212, obtaining a preset coordinate number set, and traversing each facial feature region based on the coordinate number set, wherein the coordinate number set records a plurality of second coordinate numbers, each second coordinate number corresponds to a facial feature region, and the plurality of second coordinate numbers are arranged in the coordinate number set based on a preset order;

[0084] S213, when the first coordinate number corresponding to the traversed facial feature area is equal to the second coordinate number, the candidate key point is determined as the target key point;

[0085] S214, when the first coordinate number corresponding to the traversed facial feature area is not equal to the second coordinate number, the number of candidate key points in the traversed facial feature area is adjusted based on the difference between the second coordinate number and the first coordinate number, the position of each candidate key point is adjusted based on the principle of equidistant distribution, and all the adjusted candidate key points are determined as target key points.

[0086] It should be noted that, referring to the description of facial feature areas in the above embodiments, MTCNN is able to determine multiple facial feature areas from the face image to be identified, and locate multiple candidate key points in each facial feature area. In this embodiment, the number of candidate key points of each facial feature area originally identified by MTCNN is the first coordinate number. For example, the number of candidate key points in the left ear area is the first coordinate number of the left ear area, and the number of candidate key points in the right eye area is the first coordinate number of the right eye area.

[0087] It should be noted that in the conventional MTCNN algorithm, candidate key points are generated based on the contours of the facial features. The contours of the same facial features of different faces are also quite different. For example, the eye sizes of different people are different. The eye area of ​​the face image to be identified is larger for people with larger eyes. Figure 3 As shown, the number of candidate key points that enclose a larger eye area is usually greater than the number of candidate key points that enclose a smaller eye area. Even for the same face, after being photographed at different angles, even if the angle adjustment can be achieved through image alignment technology, it is difficult to ensure that the size of the eye area in the sample face image and the face image to be identified is the same after the adjustment. According to the definition of the target shape feature vector based on the above embodiment, each vector element in the target shape feature vector is the horizontal coordinate or vertical coordinate of the target key point. When calculating the first similarity, if the number of vector elements of the target shape feature vector and the sample shape feature vector is different, it will inevitably lead to a lower calculated first similarity, affecting normal judgment.

[0088] It should be noted that each second coordinate number in the coordinate number set corresponds to a facial feature area. The second coordinate number is pre-set in this embodiment, and the number of candidate key points for each facial feature area is constrained by the second coordinate number. When the first number and the second number of the same facial feature area are the same, the candidate key point is directly determined as the target key point. If the first number and the second number are different, the number of key points needs to be increased or decreased to the second number. Ensure that the number of key points in the same facial feature area in different images is aligned.

[0089] It should be noted that if a fixed second number is used to generate candidate key points directly in the MTCNN algorithm, the accuracy of the candidate key points may be affected because the second number is too large or too small. Therefore, this embodiment does not control the number of candidate key points generated in MTCNN, but when the first number is not equal to the second number, multiple candidate key points are added or subtracted outside the contour of the facial features area, and all candidate key points are adjusted to be equidistantly distributed, and after adjusting the position, the corresponding coordinates are adjusted according to the distance from the contour to avoid affecting the accuracy of the initial positioning of MTCNN.

[0090] In addition, in one embodiment, step S30 specifically includes but is not limited to the following steps:

[0091] S31, determining twice the sum of each second coordinate number of the coordinate number set as a first value, and obtaining a preset second value, wherein the second value is the number of dimensions of the target depth feature vector;

[0092] S32, determining a first norm of the target shape feature vector based on the first value, and determining a second norm of the target depth feature vector based on the second value;

[0093] S33, normalizing the target shape feature vector based on the first norm to obtain a standard shape feature vector, and normalizing the target depth feature vector based on the second norm to obtain a standard depth feature vector;

[0094] S34, determining the cosine similarity between the standard shape feature vector and the sample shape feature vector as the first similarity, and determining the cosine similarity between the standard depth feature vector and the sample depth feature vector as the second similarity, wherein both the sample shape feature vector and the sample depth feature vector are feature vectors obtained after being standardized based on corresponding norms.

[0095] It should be noted that according to the definition of the above-mentioned target shape feature vector, the horizontal coordinate and vertical coordinate of each target key point are respectively a vector element. Therefore, the number of features of the target shape feature vector is twice the number of target key points, and the number of target key points is the sum of the second coordinate numbers. In this embodiment, twice the sum of the second coordinate numbers is the first value.

[0096] It should be noted that the number of dimensions of the target depth feature vector can be determined according to the FaceNet network. When the network structure is determined, the second value is known and can be preset.

[0097] It should be noted that the norm is used to measure the length or size of a vector in a vector space. This embodiment performs normalization processing on shape features and depth features based on the norm, which can improve the accuracy of feature comparison. The calculation formulas of the first norm and the second norm of this embodiment are as follows:

[0098] , ,in, is the target shape feature vector, is the first norm, is the i-th vector element of the target shape feature vector, is the second norm, is the i-th vector element of the target depth feature vector, k is the first value, and n is the second value.

[0099] Furthermore, the formula for normalization based on the first norm and the second norm is as follows:

[0100] , ,in, is the standard shape feature vector, is the standard deep feature vector.

[0101] It should be noted that after obtaining the standard shape feature vector and the standard depth feature vector, this embodiment uses cosine similarity as the representation of similarity. Those skilled in the art are familiar with how to calculate cosine similarity, and will not be described in detail here. It can be understood that the sample shape feature vector and the sample depth feature vector are pre-standardized based on the above method during the training phase, and the specific calculation process is not repeated here.

[0102] In addition, in one embodiment, step S40 specifically includes but is not limited to the following steps:

[0103] S41, determining a fusion similarity greater than a preset similarity threshold as a target similarity;

[0104] S42, when the number of target similarities is greater than 1, determining the sample face images corresponding to each target similarity as candidate face images;

[0105] S43, adjusting the first fusion weight and the second fusion weight based on a preset step value, and recalculating the target similarity of each candidate face image.

[0106] It should be noted that the fusion comparison module of this embodiment calculates a fusion similarity for each sample face image in the sample image library. Therefore, the number of fusion similarities is relatively large. This embodiment compares the fusion similarities based on the similarity threshold, and determines the similarity greater than the similarity threshold as the target similarity. The specific value of the similarity threshold can be set according to actual needs.

[0107] It should be noted that when there is only one target similarity, the sample face image corresponding to the target similarity can be directly used as the target face image, and no further details will be given here.

[0108] It should be noted that when there are multiple target similarities, it is possible that in the case of low resolution or external occlusion, two sample face images of different identities will be identified based on one face image to be identified. Since this embodiment fuses shape features and depth features, the first similarity and the second similarity are usually different. Therefore, the first fusion weight and the second fusion weight can be adjusted to update the target similarity, thereby achieving adaptive correction of face recognition.

[0109] It should be noted that when the first fusion weight and the second fusion weight are adjusted based on the step value, the sum of the first fusion weight and the second fusion weight is still kept at 1, so the adjustment directions of the first fusion weight and the second fusion weight are opposite. By adjusting the first fusion weight and the second fusion weight, the weight of the depth feature can be adjusted, and the weight of the depth feature can be reduced in the case of low resolution or complex external environment, and more reliance is placed on the first similarity of the shape feature, thereby improving the accuracy of face recognition.

[0110] It should be noted that after adjusting the first fusion weight and the second fusion weight, the target similarity is recalculated for each candidate face image. Since the number of candidate face images is not too large, even if recalculation is performed, it will not cause too much computational burden, and the accuracy of face recognition can be improved at a lower computational cost.

[0111] In addition, in one embodiment, the MTCNN includes a P-Net, an R-Net, and an O-Net that are cascaded in sequence, and step S43 specifically includes but is not limited to the following steps:

[0112] S431, obtaining a candidate face region detected by P-Net from the target video frame, and determining the image proportion of the face image to be recognized in the candidate face region;

[0113] S432, when the image proportion is greater than or equal to a preset proportion threshold, reducing the first fusion weight based on the step value, and increasing the second fusion weight based on the step value;

[0114] S433: When the image proportion is less than the proportion threshold, the first fusion weight is increased based on the step value, and the second fusion weight is decreased based on the step value.

[0115] It should be noted that P-Net detects candidate face regions from target video frames as an existing technology, and the specific process will not be repeated here. After obtaining the face image to be identified output by O-Net, since R-Net and O-Net will filter out irrelevant parts in the candidate face region, such as removing background images, torso images and other regions, this embodiment saves the candidate face region after obtaining it, calculates the image proportion of the face image to be identified in the candidate face region, and determines the complexity of the external environment by the image proportion. If the image proportion is less than the proportion threshold, the external environment is relatively complex, otherwise the external environment is relatively simple. The more complex the external environment, the lower the reliability of the deep feature, and the first fusion weight is increased to improve the reference value of the shape feature to improve the accuracy of face recognition; the simpler the external environment, the stronger the reliability of the deep feature, and the second fusion weight is increased to improve the reference value of the deep feature to improve the accuracy of face recognition.

[0116] In addition, in one embodiment, step S40 specifically includes but is not limited to the following steps:

[0117] S44, when the target similarities increase or decrease simultaneously after the update, determine the target face image based on the target similarity with the largest value after the update;

[0118] S45, when at least one target similarity increases in value after updating, and at least one target similarity decreases in value after updating, the target similarity with the same value change direction as the first fusion weight is used to determine the target face image.

[0119] It should be noted that if the target similarities change in the same direction after updating, such as increasing and decreasing at the same time, it can be determined that the size relationship between the first similarity and the second similarity corresponding to each target similarity is the same, for example, the first similarity is greater than the second similarity, or the first similarity is less than the second similarity. In this case, multiple sample face images and the face image to be identified are relatively similar in face shape, such as multiple users who look similar. Step S541 of this embodiment is intended to identify the scene, that is, determining that multiple target similarities are not interfered by external factors through the same-direction change of values, but that the users look similar. After identifying the scene, the target similarity with the largest updated value can be directly selected.

[0120] Exemplarily, taking target similarity 1, target similarity 2 and target similarity 3 as examples, after the value of the first fusion weight decreases, the value of target similarity 1 increases, and the values ​​of target similarity 2 and target similarity 3 decrease. Target similarity 1 is screened out, and the value change of the updated target similarity 2 is greater than the value change of the target similarity 3. Then, the proportion of the first similarity of target similarity 2 is greater than the first similarity of target similarity 3. Therefore, after the same coefficient (first fusion weight) changes, the value change of target similarity 2 is larger. The sample face image corresponding to target similarity 2 is used as the target face image, and the target recognition result is output.

[0121] like Figure 4 As shown, Figure 4 : is a structural diagram of a face recognition device based on dual feature comparison provided by an embodiment of the present invention. The present invention also provides a face recognition device based on dual feature comparison, comprising:

[0122] The processor 401 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0123] The memory 402 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 402 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 402, and the processor 401 calls and executes the face recognition method based on dual feature comparison in the embodiment of this application;

[0124] Input / output interface 403, used to implement information input and output;

[0125] Communication interface 404, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0126] A bus 404 that transmits information between various components of the device (e.g., processor 401, memory 402, input / output interface 403, and communication interface 404);

[0127] The processor 401 , the memory 402 , the input / output interface 403 and the communication interface 404 are connected to each other in communication within the device via the bus 405 .

[0128] An embodiment of the present application also provides an electronic device, including the face recognition device based on dual feature comparison as described above.

[0129] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned face recognition method based on dual feature comparison is implemented.

[0130] As a non-transient computer-readable storage medium, the memory can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and are implemented to be located in one place, or may also be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0131] It will be appreciated by those skilled in the art that all or some of the steps and systems in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically include computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0132] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions under the shared conditions without violating the spirit of the present invention. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A face recognition method based on dual feature comparison, characterized in that: Applied to an intelligent monitoring system, the intelligent monitoring system is preset with a sample image library, a multi-task convolutional neural network MTCNN, an identity recognition network FaceNet and a fusion comparison module, the sample image library is preset with a plurality of sample face images, the sample face images are preset with a sample shape feature vector and a sample depth feature vector, the method comprises: Detecting a face image to be identified from a target video frame based on MTCNN, extracting a target depth feature vector from the face image to be identified through FaceNet and inputting the vector into the fusion comparison module; Extracting a target shape feature vector based on the face image to be identified by the fusion comparison module, wherein the target shape feature vector is used to indicate the shape of facial features; Traversing the sample image library through the fusion comparison module, each time traversing a sample face image, determining a first similarity based on the target shape feature vector and the corresponding sample shape feature vector, determining a second similarity based on the target depth feature vector and the corresponding sample depth feature vector, and determining a fusion similarity based on the first similarity and the second similarity, wherein the sum of a first fusion weight of the first similarity and a second fusion weight of the second similarity is 1; Determine a plurality of candidate face images based on the fusion similarity, update each of the fusion similarities by adjusting the values ​​of the first fusion weight and the second fusion weight, and determine a target recognition result based on the one with the largest value change among the plurality of fusion similarities having the same value change direction as the first fusion weight, wherein the sum of the first fusion weight and the second fusion weight is maintained to be 1; Extracting a target shape feature vector based on the face image to be identified by the fusion comparison module includes: Determining a plurality of target key points in the face image to be identified, wherein the plurality of target key points are used to characterize a plurality of facial feature areas of the face image to be identified; Arrange the target key points based on a preset order, and construct a target shape feature vector based on the image coordinates of all the target key points, wherein the preset order is used to indicate the arrangement order of the plurality of facial feature regions, and the horizontal coordinate and the vertical coordinate of the target key point are respectively a vector element of the target shape feature vector; The number of elements of each sample shape feature vector is the same, and multiple target key points are determined in the face image to be recognized, including: Determine a plurality of facial feature regions in the face image to be recognized based on the MTCNN, generate a plurality of candidate key points in each of the facial feature regions, and determine a first coordinate number of each of the facial feature regions, wherein the first coordinate number is used to represent the number of the candidate key points in the facial feature regions; Obtaining a preset coordinate number set, and traversing each of the facial feature regions based on the coordinate number set, wherein the coordinate number set records a plurality of second coordinate numbers, each of the second coordinate numbers corresponds to one of the facial feature regions, and the plurality of second coordinate numbers are arranged in the coordinate number set based on the preset order; When the first coordinate number corresponding to the traversed facial feature area is equal to the second coordinate number, determining the candidate key point as the target key point; Alternatively, when the first coordinate number corresponding to the traversed facial feature area is not equal to the second coordinate number, the number of candidate key points in the traversed facial feature area is adjusted based on the difference between the second coordinate number and the first coordinate number, the position of each candidate key point is adjusted based on the principle of equidistant distribution, and all the adjusted candidate key points are determined as the target key points.

2. The face recognition method based on dual feature comparison according to claim 1, characterized in that: Determining a first similarity based on the target shape feature vector and the corresponding sample shape feature vector, determining a second similarity based on the target depth feature vector and the corresponding sample depth feature vector, and determining a fusion similarity based on the first similarity and the second similarity, including: Determine twice the sum of each of the second coordinate numbers in the coordinate number set as a first value, and obtain a preset second value, wherein the second value is the number of dimensions of the target depth feature vector; Determine a first norm of the target shape feature vector based on the first value, and determine a second norm of the target depth feature vector based on the second value; Based on the first norm, the target shape feature vector is normalized to obtain a standard shape feature vector, and based on the second norm, the target depth feature vector is normalized to obtain a standard depth feature vector; The cosine similarity between the standard shape feature vector and the sample shape feature vector is determined as the first similarity, and the cosine similarity between the standard depth feature vector and the sample depth feature vector is determined as the second similarity, wherein the sample shape feature vector and the sample depth feature vector are both feature vectors obtained after being standardized based on corresponding norms.

3. The face recognition method based on dual feature comparison according to claim 1, characterized in that: Determining a plurality of candidate face images based on the fusion similarity, and updating each of the fusion similarities by adjusting the values ​​of the first fusion weight and the second fusion weight, comprises: Determining the fusion similarity greater than a preset similarity threshold as a target similarity; When the number of the target similarities is greater than 1, determining the sample face images corresponding to the respective target similarities as the candidate face images; The first fusion weight and the second fusion weight are adjusted based on a preset step value, and the target similarity of each of the candidate face images is recalculated.

4. According to the face recognition method based on dual feature comparison in claim 3, the MTCNN comprises a P-Net, an R-Net and an O-Net which are cascaded in sequence, characterized in that: Adjusting the first fusion weight and the second fusion weight based on a preset step value includes: Obtaining a candidate face region detected by the P-Net from the target video frame, and determining an image proportion of the face image to be recognized in the candidate face region; When the image proportion is greater than or equal to a preset proportion threshold, reducing the first fusion weight based on the step value, and increasing the second fusion weight based on the step value; Alternatively, when the image proportion is less than the proportion threshold, the first fusion weight is increased based on the step value, and the second fusion weight is reduced based on the step value.

5. The face recognition method based on dual feature comparison according to claim 4 is characterized in that: Determining the target recognition result based on the one with the largest value change among the plurality of fusion similarities in the same direction as the value change of the first fusion weight includes: When the target similarities increase or decrease simultaneously after updating, determining the target face image based on the target similarity with the largest value after updating; Alternatively, when at least one of the target similarities increases in value after updating, and at least one of the target similarities decreases in value after updating, the target facial image is determined by the target similarities with the largest value change in the same direction as the first fusion weight value change.

6. A face recognition device based on dual feature comparison, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the face recognition method based on dual feature comparison as described in any one of claims 1 to 5.

7. An electronic device, characterized in that: It includes the face recognition device based on dual feature comparison as described in claim 6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the face recognition method based on dual-feature comparison as described in any one of claims 1 to 5.

Images