Method for generating face image of person wearing mask, method for recognizing face image and device for recognizing face image

Abstract

The present application provides a method for generating a face image of a person wearing a mask, which comprises: obtaining n images of masks with different materials and colors, any mask image is recorded as; extracting six feature points about the upper left, lower left, upper middle, lower middle, upper right and lower right of each mask image; obtaining face images of m persons without masks, any face image without masks is recorded as P j ; Detect six feature points of each face image about the upper left point of the cheek, the lower left point of the cheek, the middle point of the nose, the middle point of the chin, the upper right point of the cheek and the lower right point of the cheek; according to the image P j The size and inclination of the face, for any mask image P i The six feature points and any face image P without a mask j The six feature points are matched to synthesize the masked face image P ij ; Adjust the brightness of the mask area in the masked face image according to the light intensity of the face image without mask. The present application also provides a method and device for identifying a face wearing a mask.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically, to a method for generating a face image of a person wearing a mask, a method for recognizing a face image, and a recognition device. Background Art

[0002] Because face recognition uses the feature information of the entire face, masks will block some important feature information of the face, which will have a great impact on the accuracy of face recognition and reduce the recognition rate.

[0003] When training a model for face recognition, it is first necessary to obtain a training data set of face images wearing masks. Common methods include artificially adding masks to the face data in the training set so that the model can better recognize faces wearing masks, or methods based on data generation data sets such as GAN. The method of generating data based on GAN first needs to collect a large amount of data on faces wearing masks in order to achieve better results in GAN training. However, there is a lack of a large amount of data on real faces wearing masks, so this method affects its applicability; while artificially adding masks to the face data in the training set is relatively simple and direct. If the authenticity of the generated masked face is not considered when directly generating the masked face image, including the matching degree of the face inclination, rotation angle and mask angle, the matching degree of the face and mask size, and the different light intensities of the face and mask, it is easy to cause the authenticity of the synthesized masked face image, thereby affecting the accuracy of the trained face recognition model. Summary of the invention

[0004] In view of the prior art, the technical problem solved by the present application is to provide a method for generating a face image wearing a mask, a recognition method and a device for generating a realistic face image wearing a mask. The method for generating a face image wearing a mask, the recognition method and the device for generating a face image wearing a mask comprehensively consider the face clarity when matching and synthesizing the face image wearing a mask and the consistency factor of the light intensity of the face and the mask image.

[0005] In order to solve the above technical problems, the present application provides a method for generating a face image of a person wearing a mask, comprising:

[0006] Get n images of masks with different materials and colors, and any mask image is denoted as P i ;

[0007] Extract six feature points of the upper left, lower left, upper middle, lower middle, upper right and lower right of the mask in each mask image;

[0008] Get m people's face images without masks, any face image without masks is denoted as P j ;

[0009] Detect six feature points of each face image, namely, the upper left point of the cheek, the lower left point of the cheek, the middle point of the nose, the middle point of the chin, the upper right point of the cheek, and the lower right point of the cheek;

[0010] According to the image P j The size and inclination of the face, for any mask image P i The six feature points and any face image P without a mask j The six feature points are matched to synthesize the masked face image P ij ;

[0011] According to the face image P without mask j Adjust the light intensity of the face image P wearing a mask ij The brightness of the mask area in the middle;

[0012] Among them, the extracted mask image P i The contour formed by the six feature points and the face image P without mask j The contour formed by the six feature points is matched.

[0013] In a possible implementation, according to the image P j The size and inclination of the face, for any mask image P i The six feature points and any face image P without a mask j The six feature points are matched to synthesize the masked face image P ij The steps include:

[0014] Calculate the face image P without mask j The tilt angle;

[0015] According to the face image P without mask j The tilt angle of the mask image P i Perspective transformation to ensure that the mask image P i Compared with the face image P without mask j have the same inclination angle;

[0016] Calculate the face image P without mask j The size A of the face area that needs a mask formed by the six feature points;

[0017] According to the size A, the tilt-corrected mask image P i The mask area formed by the six feature points is scaled;

[0018] Find the tilt-corrected and scaled mask image P i The six feature points of the face image P without a mask jThe corresponding matching points in the six feature points are fitted to form the face image P wearing a mask ij .

[0019] In a possible implementation, the face image P without or with a mask is recorded i The tilt angle is α, and the face image P without mask is connected i The middle point of the nose and the middle point of the chin are calculated, and the angle θ between the line connecting the middle point of the nose and the middle point of the chin and the X-axis direction of the image coordinate system is calculated as follows:

[0020] η=90-θ, θ=arctan((y2-y1) / (x2-x1));

[0021] Among them, (x1, y1) and (x2, y2) are the coordinates of the two feature points of the middle point of the nose and the middle point of the chin of the detected face image without mask.

[0022] In one possible implementation, find the tilt-corrected mask image P i The six feature points of the face image P without a mask j The corresponding matching points among the six feature points are: Calculate the face image P without mask j The six feature points of the mask image P after tilt correction and scaling i The matching degree S of the six feature points is as follows. When the matching degree S value is the minimum, the mask image P after tilt correction and scaling is completed. i The six feature points of the face image P without a mask j The corresponding matching points among the six feature points;

[0023]

[0024] Among them, Ω is the face image P without mask j The set of six feature points, (x k ,y k ) is any feature point in the set Ω; Ω′ is the mask image P i The set of six feature points, (x′ k ,y′ k ) is any feature point in the set Ω′; 1≤k≤6, ∑ represents the sum.

[0025] In a possible implementation, the pixel value of the pixel point belonging to the mask area in the mask-wearing face image is f1, and the mask-wearing face image P is adjusted. ij The pixel value of the pixel point belonging to the mask area is f2; according to the face image P without mask j Adjust the light intensity of the face image P wearing a mask ijThe steps to adjust the brightness of the mask area include:

[0026] Calculate the face image P without mask j The light intensity β1,

[0027] Calculate the mask image P i The light intensity β2,

[0028] Adjust the face image P wearing a mask ij The pixel value of the pixel point in the middle mask area: f2 = f1 + (β1-β2);

[0029] Among them, max(p i (R,G,B))、min(p i (R,G,B)) represents the image P i The maximum and minimum values ​​of the mean values ​​of the green channel, the red channel, and the blue channel of all pixels; max(p j (R,G,B))、min(p j (R,G,B)) image P j The maximum and minimum values ​​of the green channel mean, red channel mean, and blue channel mean of all pixels.

[0030] In a possible implementation, six feature points, namely, the upper left point of the cheek, the lower left point of the cheek, the middle point of the nose, the middle point of the chin, the upper right point of the cheek, and the lower right point of the cheek, are detected based on the SDM facial feature detection method; wherein the loss function in the training process based on the SDM facial feature detection method is: L=a1×Loss1+a2×Loss2;

[0031] Among them, Loss1 is the loss function of the original SDM face feature point detection model, t k is the kth feature point detected based on the SDM face feature detection method, x′ k is the mask image P i The kth feature point of the six feature points, a1 and a2 are weighting coefficients and a1+a2=1.

[0032] In one possible implementation, six feature points of the upper left, lower left, upper middle, lower middle, upper right and lower right of each mask image are extracted by manual annotation method, or the yolov5 deep learning model is used to detect the six feature points of each mask image.

[0033] In the method for generating a face image wearing a mask, six key points of the face of any image without a mask and six key points of the mask of any mask image are extracted respectively; then, according to the face image without a mask P j The size and inclination of the face in the image P i The six feature points and any face image P without a mask j The six feature points are matched to form the face image P wearing a mask ij ; and based on the face image P without mask j Adjust the light intensity of the face image P wearing a mask ij In this way, the inclination of the mask and the inclination of the face are kept consistent during image synthesis, so that the mask is matched to the appropriate position of the face, and the synthesized face image P with mask is ij The illumination intensity of the mask area is similar to that of the image P ij The face area of ​​the image P ij There will not be a big brightness difference between the face area and the mask area. Therefore, the method for generating a face image wearing a mask comprehensively considers the factors of the face inclination and the consistency of the light intensity between the face and the mask image, and generates a face image wearing a mask that conforms to the actual situation, that is, the synthesized image P ij The result is close to the real-life photograph of a person's face wearing a mask.

[0034] The present application also provides a method for recognizing a face wearing a mask, comprising:

[0035] Training set selection and labeling: select face images without masks and face images with masks generated by the face image generation method, and label the face images without masks and the face images with masks with representative face identity IDs;

[0036] Training the FaceNet model: Input all images in the training set into the FaceNet model, and use the feature layer of the FaceNet model to extract facial feature vectors; for images with different ID labels in the training set, calculate the Euclidean distance H1 between facial feature vectors of images with different ID labels; for images with the same ID label in the training set, calculate the Euclidean distance H2 between facial feature vectors of images with the same ID label; construct the loss function L = H2-H1 for FaceNet model training, and update the weight coefficient of the feature layer of the FaceNet model through the value of the loss function L in each training process until the value of the loss function L is minimized;

[0037] Face recognition with masks: Input the face image of the person wearing a mask to be identified into the trained FaceNet model for face ID recognition;

[0038] In one possible implementation, in the selected training set, for n persons, the number of face images without masks of different persons is the same, the number of face images with masks of different persons is the same, and the number of face images without masks and with masks of the same person is the same.

[0039] In the method for recognizing a face wearing a mask, the face wearing mask data in the training set is generated by using the above-mentioned method for generating a face image wearing a mask. When generating a face wearing a mask image, the factors of the face inclination and the consistency of the light intensity of the face and the mask image are comprehensively considered to generate a face wearing a mask image that conforms to the actual situation, that is, a synthesized image P ij The results are close to the actual results of photographing people wearing masks. Real and effective training set data is conducive to training a FaceNet model with high recognition accuracy, thereby improving the accuracy of face recognition of people wearing masks.

[0040] The present application also provides a device for recognizing a face wearing a mask, comprising:

[0041] Mask image acquisition unit: used to acquire images of n masks with different materials and colors;

[0042] Mask key point extraction unit: used to extract six feature points of the upper left, lower left, upper middle, lower middle, upper right and lower right of the mask in each mask image;

[0043] A face image acquisition unit: used to acquire face images of m persons without masks;

[0044] The facial key point extraction unit is used to detect six feature points of each face image, namely, the upper left point of the cheek, the lower left point of the cheek, the middle point of the nose, the middle point of the chin, the upper right point of the cheek and the lower right point of the cheek;

[0045] Mask face synthesis unit: used to synthesize the face according to the image P j The size and inclination of the face, and for any mask image P i The six feature points and any face image P without a mask j The six feature points are matched to form the face image P wearing a mask ij ;

[0046] Brightness adjustment unit: used to adjust the brightness of the face image P without mask j Adjust the light intensity of the face image P wearing a mask ij the brightness of the masked area; and,

[0047] Face recognition unit: used to add all synthesized face images wearing masks to the training set and perform face recognition based on the FaceNet model.

[0048] In the mask-wearing face recognition device, the mask-wearing face data in the training set is generated by using the mask-wearing face image generation method. When generating the mask-wearing face image, the factors of the face inclination and the consistency of the light intensity between the face and the mask image are comprehensively considered to generate the mask-wearing face image that conforms to the actual situation, that is, the synthesized image P ij The results are close to the actual results of photographing people wearing masks. Real and effective training set data is conducive to training a FaceNet model with high recognition accuracy, thereby improving the accuracy of face recognition of people wearing masks. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0050] Figure 1 This is a flow chart of a method for generating a face image of a person wearing a mask according to Embodiment 1 of the present application;

[0051] Figure 2 This is a graph showing the extraction results of six feature points of the mask in the mask image of Example 1 of the present application;

[0052] Figure 3 This is a detection result diagram of six features of a face image without a mask according to Example 1 of the present application;

[0053] Figure 4 According to any mask image P in the first embodiment of the present application i The six feature points and any face image P without a mask j The six feature points of the face image P wearing a mask are matched and synthesized ij Flow chart of the method;

[0054] Figure 5 This is a structural block diagram of the device for recognizing faces of people wearing masks according to the third embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element.

[0057] It should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0058] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0059] The facial image generation method, recognition method and recognition device of a person wearing a mask provided in the embodiments of the present application are now described in conjunction with the accompanying drawings.

[0060] Embodiment 1

[0061] Reference Figure 1 The method for generating a face image of a person wearing a mask provided in the first embodiment of the present application comprises the following steps:

[0062] Step S100: Obtain n images of masks with different materials and colors, and any mask image is denoted as P i .

[0063] Step S200: extracting six feature points of the upper left, lower left, upper middle, lower middle, upper right and lower right of the mask in each mask image.

[0064] Step S300: Obtain m facial images of people without masks, and any facial image without masks is denoted as P j .

[0065] Step S400: Detect six feature points of each face image, namely, the upper left point of the cheek, the lower left point of the cheek, the middle point of the nose, the middle point of the chin, the upper right point of the cheek, and the lower right point of the cheek.

[0066] Step S500: Based on the image P jThe size and inclination of the face, for any mask image P i The six feature points and any face image P without a mask j The six feature points are matched to synthesize the masked face image P ij .

[0067] Step S600: Based on the face image P without mask j Adjust the light intensity of the face image P wearing a mask ij The brightness of the masked area in the middle.

[0068] Among them, the extracted mask image P i The contour formed by the six feature points and the face image P without mask j The contour formed by the six feature points is matched. Among them, 1≤i≤n,1≤j≤m.

[0069] In step S100, a mask image can be generated by collecting masks of any material and any color, and the mask shape in the collected mask image is the mask shape when it is carried on the face and observed from the frontal perspective of the face. In step S200, the six feature points of the upper left, lower left, upper middle, lower middle, upper right and lower right of each mask image are extracted by manual annotation method, or the six feature points of each mask image are detected by yolov5 deep learning model. The six feature points of the upper left, lower left, upper middle, lower middle, upper right and lower right of the mask image are K1, K2, K3, K4, K5 and K6 respectively. The six feature points of any mask image extracted are as follows: Figure 2 The upper left feature point K1 and the upper right feature point K5 are located on the left and right sides of the upper edge of the mask in the mask image, respectively; the upper middle feature point K3 is located in the middle of the upper edge of the mask in the mask image; the lower left feature point K2 and the lower right feature point K6 are located in the middle of the left edge and the middle of the right edge of the mask in the mask image, respectively.

[0070] In step S400, six feature points of each face image without a mask are detected based on the SDM (Supvised Descent Method) face feature detection method to detect six feature points of the upper left point of the cheek, the lower left point of the cheek, the middle point of the nose, the middle point of the chin, the upper right point of the cheek and the lower right point of the cheek. The upper left point of the cheek, the lower left point of the cheek, the middle point of the nose, the middle point of the chin, the upper right point of the cheek and the lower right point of the cheek are R1, R2, R3, R4, R5 and R6 respectively. The six feature points R1, R2, R3, R4, R5 and R6 of the detected face are as follows: Figure 3 shown.

[0071] The existing SDM model for detecting facial feature points will not be described in detail here. The loss function of the original SDM model is recorded as Loss1. The improvement of the embodiment of the present application is that a new loss function Loss2 is introduced into the original SDM model to form a loss function L in the training process of the SDM (Supvised Descent Method) facial feature detection method:

[0072] L=a1×Loss1+a2×Loss2 (1)

[0073]

[0074] In formulas (1) and (2), Loss1 is the loss function of the original SDM face feature point detection model, t k is the kth feature point detected based on the SDM face feature detection method, x′ k is the mask image P i The kth feature point of the six feature points, a1 and a2 are weighting coefficients and a1+a2=1. In practical applications, a1 and a2 can be adjusted as required.

[0075] in,

[0076] In formula (3), U is the input face image, W is the weight matrix of the SDM training model, and F is the activation function of the SDM training model. F can be sigmoid or tanh, but is not limited thereto.

[0077] The objective function for detecting six feature points in the SDM facial feature detection method is argmin (L), and the function argmin is used to obtain the W value when the L value is minimum.

[0078] Reference Figure 4 In step S500, according to the image P j The size and inclination of the face, for any mask image P i The six feature points and any face image P without a mask j The six feature points are matched to synthesize the masked face image P ij The steps include:

[0079] Step S501: Calculate the face image P without mask j The tilt angle.

[0080] Step S502: Based on the face image P without mask j The tilt angle of the mask image P i Perspective transformation to ensure that the mask image P iCompared with the face image P without mask j have the same inclination angle.

[0081] Step S503: Calculate the face image P without mask j The size A of the face area that needs a mask is formed by the six feature points.

[0082] Step S504: The tilt-corrected mask image P is corrected according to the size A. i The mask area formed by the six feature points is scaled.

[0083] Step S505: Find the mask image P after tilt correction and scaling i The six feature points of the face image P without a mask j The corresponding matching points in the six feature points are fitted to form the face image P wearing a mask ij .

[0084] Usually, the origin of the image coordinate system is set at the upper left corner of the image. If the angle between the connecting line of the feature points R3 and R4 of the detected face image without a mask and the Y axis of the image coordinate system is 0 degrees, it means that the face in the face image without a mask is not tilted. If the angle between the connecting line of R3 and R4 and the Y axis of the image coordinate system is greater than 0 degrees, it means that the face in the face image without a mask is tilted. At this time, if the mask is to be matched on the face of the face image without a mask, the mask needs to be made clear accordingly, so that a face image close to the real face with a mask can be generated. Otherwise, the mask in the synthesized face image with a mask is not synthesized in the appropriate position of the face, causing distortion.

[0085] In step S501, the coordinates of the feature points R3 and R4 detected in step S400 are recorded as (x1, y1) and (x2, y2) respectively, and the face image P without mask is recorded as i The tilt angle is α, and the face image P without mask is connected i The middle point of the nose R3 and the middle point of the chin R4 are taken, and the angle θ between the line connecting the middle point of the nose and the middle point of the chin and the X-axis direction of the image coordinate system is calculated, and η is: η=90-θ.

[0086] The steps to calculate the above θ include:

[0087] Connect the two feature points R3 and R4;

[0088] The angle θ between the connecting line of R3 and R4 and the X-axis of the image coordinate system is calculated, θ=arctan((y2-y1) / (x2-x1)).

[0089] In step S505, the tilt-corrected mask image P is found iThe six feature points of the face image P without a mask j The corresponding matching points among the six feature points are: Calculate the face image P without mask j The six feature points of the mask image P after tilt correction and scaling i The matching degree S of the six feature points is as follows. When the matching degree S value is the minimum, the mask image P after tilt correction and scaling is completed. i The six feature points of the face image P without a mask j The corresponding matching points among the six feature points;

[0090]

[0091] Among them, Ω is the face image P without mask j The set of six feature points, (x k ,y k ) is any feature point in the set Ω; Ω′ is the mask image P i The set of six feature points, (x′ k ,y′ k ) is any feature point in the set Ω′; 1≤k≤6, ∑ represents the sum.

[0092] Since any face image P without mask is obtained j and mask image P i The brightness of the face image P is not necessarily the same, so it is necessary to match the synthetic face image P wearing a mask ij When the face image P without mask to be synthesized is j and mask image P i The brightness of the masked face image P must be consistent to match the synthesized face image P ij In line with the reality.

[0093] In step S600, the pixel value of the pixel point belonging to the mask area in the mask-wearing face image is f1, and the mask-wearing face image P is adjusted. ij The pixel value of the pixel point belonging to the mask area is f2; according to the face image P without mask j Adjust the light intensity of the face image P wearing a mask ij The steps to adjust the brightness of the mask area include:

[0094] Calculate the face image P without mask j The light intensity β1,

[0095] Calculate the mask image P i The light intensity β2,

[0096] Adjust the face image P wearing a mask ijThe pixel value of the pixel point in the middle mask area: f2 = f1 + (β1-β2);

[0097] Among them, max(p i (R,G,B))、min(p i (R,G,B)) represents the image P i The maximum and minimum values ​​of the mean values ​​of the green channel, the red channel and the blue channel of all pixels; max(p j (R,G,B))、min(p j (R,G,B)) image P j The maximum and minimum values ​​of the green channel mean, red channel mean, and blue channel mean of all pixels.

[0098] In the method for generating a face image wearing a mask, six key points of the face of any image without a mask and six key points of the mask of any mask image are extracted respectively; then, according to the face image without a mask P j The size and inclination of the face in the image P i The six feature points and any face image P without a mask j The six feature points are matched to form the face image P wearing a mask ij ; and based on the face image P without mask j Adjust the light intensity of the face image P wearing a mask ij In this way, the inclination of the mask and the inclination of the face are kept consistent during image synthesis, so that the mask is matched to the appropriate position of the face, and the synthesized face image P with mask is ij The illumination intensity of the mask area is similar to that of the image P ij The face area of ​​the image P ij There will not be a big brightness difference between the face area and the mask area. Therefore, the method for generating a face image wearing a mask comprehensively considers the factors of the face inclination and the consistency of the light intensity between the face and the mask image, and generates a face image wearing a mask that conforms to the actual situation, that is, the synthesized image P ij The result is close to the real-life photograph of a person's face wearing a mask.

[0099] Embodiment 2

[0100] The method for recognizing a face wearing a mask provided in the second embodiment of the present application comprises the following steps:

[0101] Training set selection and labeling: selecting face images without masks and face images with masks generated by the above-mentioned face image generation method, and labeling the face images without masks and the face images with masks with representative face identity IDs;

[0102] Training the FaceNet model:

[0103] Input all the images in the training set into the FaceNet model and use the feature layer of the FaceNet model to extract the facial feature vector;

[0104] For images with different ID labels in the training set, the Euclidean distance H1 between the facial feature vectors of images with different ID labels is calculated; for images with the same ID label in the training set, the Euclidean distance H2 between the facial feature vectors of images with the same ID label is calculated;

[0105] Construct the loss function L = H2-H1 for FaceNet model training. In each training process, update the weight coefficient of the feature layer of the FaceNet model by the value of the loss function L until the value of the loss function L is minimized.

[0106] Face recognition with masks: Input the face image of a person wearing a mask to be identified into the trained FaceNet model for face ID recognition.

[0107] Among them, in the training set, for n people, the number of face images without masks of different people is the same, the number of face images with masks of different people is the same, and the number of face images without masks and with masks of the same person is the same. This is conducive to training a FaceNet model with higher recognition accuracy.

[0108] In the above steps, the annotation of the training set is collected by manual annotation method.

[0109] In the method for recognizing a face wearing a mask, the face wearing mask data in the training set is generated by using the above-mentioned method for generating a face image wearing a mask. When generating a face wearing a mask image, the factors of the face inclination and the consistency of the light intensity of the face and the mask image are comprehensively considered to generate a face wearing a mask image that conforms to the actual situation, that is, a synthesized image P ij The results are close to the actual results of photographing people wearing masks. Real and effective training set data is conducive to training a FaceNet model with high recognition accuracy, thereby improving the accuracy of face recognition of people wearing masks.

[0110] Embodiment 3

[0111] Reference Figure 5Embodiment 3 of the present application provides a device 100 for recognizing a face wearing a mask, and the method in Embodiment 1 of the present application and the method in Embodiment 2 are applied to the device 100 for recognizing a face wearing a mask. The device 100 for recognizing a face wearing a mask includes a mask image acquisition unit 10, a mask key point extraction unit 20, a face image acquisition unit 30, a face key point extraction unit 40, a mask face synthesis unit 50, a brightness adjustment unit 60, and a face recognition unit 70.

[0112] Specifically, the mask image acquisition unit 10 is used to acquire images of n masks with different materials and colors. The mask key point extraction unit 20 is used to extract six feature points of the upper left, lower left, upper middle, lower middle, upper right and lower right of the mask in each mask image. The face image acquisition unit 30 is used to acquire face images of m people without masks. The face key point extraction unit 40 is used to detect six feature points about the upper left cheek point, the lower left cheek point, the middle point of the nose, the middle point of the chin, the upper right cheek point and the lower right cheek point of each face image. The mask face synthesis unit 50 is used to extract the six feature points of the upper left cheek point, the lower left cheek point, the middle point of the nose, the middle point of the chin, the upper right cheek point and the lower right cheek point according to the image P. j The size and inclination of the face, and for any mask image P i The six feature points and any face image P without a mask j The six feature points are matched to form the face image P wearing a mask ij The brightness adjustment unit 60 is used to adjust the brightness of the face image P without a mask. j Adjust the light intensity of the face image P wearing a mask ij The face recognition unit 70 is used to add all synthesized face images wearing masks to the training set and perform face recognition based on the FaceNet model.

[0113] The implementation principle of the mask key point extraction unit 20 is the same as step S200 of the first embodiment, the implementation principle of the face key point extraction unit 40 is the same as step S400 of the first embodiment, the implementation principle of the mask face synthesis unit 50 is the same as step S500 of the first embodiment, the implementation principle of the brightness adjustment unit 60 is the same as step S600 of the first embodiment, and the implementation principle of the face recognition unit 70 is the same as the method for recognizing a masked face in the second embodiment. No further details will be given here.

[0114] In the mask-wearing face recognition device, the mask-wearing face data in the training set is generated by using the mask-wearing face image generation method. When generating the mask-wearing face image, the factors of the face inclination and the consistency of the light intensity between the face and the mask image are comprehensively considered to generate the mask-wearing face image that conforms to the actual situation, that is, the synthesized image P ijThe results are close to the actual results of photographing people wearing masks. Real and effective training set data is conducive to training a FaceNet model with high recognition accuracy, thereby improving the accuracy of face recognition of people wearing masks.

[0115] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for generating a face image of a person wearing a mask, characterized in that: include: Get n images of masks with different materials and colors, and any mask image is denoted as P i ; Extract six feature points of the upper left, lower left, upper middle, lower middle, upper right and lower right of the mask in each mask image; Get m people's face images without masks, any face image without masks is denoted as P j ; Detect six feature points of each face image, namely, the upper left point of the cheek, the lower left point of the cheek, the middle point of the nose, the middle point of the chin, the upper right point of the cheek, and the lower right point of the cheek; According to the image P j The size and inclination of the face, for any mask image P i The six feature points and any face image P without a mask j The six feature points are matched to synthesize the masked face image P ij ; According to the face image P without mask j Adjust the light intensity of the face image P wearing a mask ij The brightness of the mask area in the middle; Among them, the extracted mask image P i The contour formed by the six feature points and the face image P without mask j The contour formed by the six feature points is matched; The pixel value of the pixel point belonging to the mask area in the mask-wearing face image is f1, and the pixel value of the mask-wearing face image P is adjusted. ij The pixel value of the pixel point belonging to the mask area is f2; according to the face image P without mask j Adjust the light intensity of the face image P wearing a mask ij The steps to adjust the brightness of the mask area include: Calculate the face image P without mask j The light intensity β1, Calculate the mask image P i The light intensity β2, Adjust the face image P wearing a mask ij The pixel value of the pixel point in the middle mask area: f2 = f1 + (β1-β2); Among them, max(p i (R,G,B))、min(p i (R,G,B)) represents the image P i The maximum and minimum values ​​of the mean values ​​of the green channel, the red channel, and the blue channel of all pixels; max(p j (R,G,B))、min(p j (R,G,B)) image P j The maximum and minimum values ​​of the green channel mean, red channel mean and blue channel mean of all pixels; Based on the SDM facial feature detection method, six feature points of the upper left point of the cheek, the lower left point of the cheek, the middle point of the nose, the middle point of the chin, the upper right point of the cheek and the lower right point of the cheek are detected; wherein the loss function in the training process based on the SDM facial feature detection method is: L = a1 × Loss1 + a2 × Loss2; Among them, Loss1 is the loss function of the original SDM face feature point detection model, t k is the kth feature point detected based on the SDM face feature detection method, x′ k is the mask image P i The kth feature point of the six feature points, a1 and a2 are weighting coefficients and a1+a2=1.

2. The method for generating a face image of a person wearing a mask as claimed in claim 1, characterized in that: According to the image P j The size and inclination of the face, for any mask image P i The six feature points and any face image P without a mask j The six feature points are matched to synthesize the masked face image P ij The steps include: Calculate the face image P without mask j The tilt angle; According to the face image P without mask j The tilt angle of the mask image P i Perspective transformation to ensure that the mask image P i Compared with the face image P without mask j have the same inclination angle; Calculate the face image P without mask j The size A of the face area that needs a mask formed by the six feature points; According to the size A, the tilt-corrected mask image P i The mask area formed by the six feature points is scaled; Find the tilt-corrected and scaled mask image P i The six feature points of the face image P without a mask j The corresponding matching points in the six feature points are fitted to form the face image P wearing a mask ij .

3. The method for generating a face image of a person wearing a mask as claimed in claim 2, characterized in that: Remember the face image P without mask i The tilt angle is α, and the face image P without mask is connected i The middle point of the nose and the middle point of the chin are calculated, and the angle θ between the line connecting the middle point of the nose and the middle point of the chin and the X-axis direction of the image coordinate system is calculated as follows: η=90-θ, θ=arctan((y2-y1) / (x2-x1)); Among them, (x1, y1) and (x2, y2) are the coordinates of the two feature points of the middle point of the nose and the middle point of the chin of the detected face image without mask.

4. The method for generating a face image of a person wearing a mask as claimed in claim 2, characterized in that: Find the tilt-corrected mask image P i The six feature points of the face image P without a mask j The corresponding matching points among the six feature points are: Calculate the face image P without mask j The six feature points of the mask image P after tilt correction and scaling i The matching degree S of the six feature points is as follows. When the matching degree S value is the minimum, the tilt correction and scaled mask image P is completed. i The six feature points of the face image P without a mask j The corresponding matching points among the six feature points; Among them, Ω is the face image P without mask j The set of six feature points, (x k ,y k ) is any feature point in the set Ω; Ω′ is the mask image P i The set of six feature points, (x′ k ,y′ k ) is any feature point in the set Ω′; 1≤k≤6, ∑ represents the sum.

5. The method for generating a face image of a person wearing a mask as claimed in claim 1, characterized in that: The six feature points of the upper left, lower left, upper middle, lower middle, upper right and lower right of each mask image are extracted by manual labeling method, or the six feature points of each mask image are detected by yolov5 deep learning model.

6. A method for recognizing a face wearing a mask, characterized in that: include: Training set selection and labeling: selecting a face image without a mask and a face image with a mask generated by the method for generating a face image with a mask according to any one of claims 1 to 5, and labeling the face image without a mask and the face image with a mask with representative face identity IDs; Training the FaceNet model: Input all images in the training set into the FaceNet model, and use the feature layer of the FaceNet model to extract facial feature vectors; for images with different ID labels in the training set, calculate the Euclidean distance H1 between facial feature vectors of images with different ID labels; for images with the same ID label in the training set, calculate the Euclidean distance H2 between facial feature vectors of images with the same ID label; construct the loss function L = H2-H1 for FaceNet model training, and update the weight coefficient of the feature layer of the FaceNet model through the value of the loss function L in each training process until the value of the loss function L is minimized; Face recognition with masks: Input the face image of a person wearing a mask to be identified into the trained FaceNet model for face ID recognition.

7. The method for recognizing a face wearing a mask as claimed in claim 6, characterized in that: In the selected training set, for n persons, the number of face images without masks of different persons is the same, the number of face images with masks of different persons is the same, and the number of face images without masks and with masks of the same person is the same.

8. A device for recognizing faces of people wearing masks, characterized in that: include: Mask image acquisition unit: used to acquire images of n masks with different materials and colors; Mask key point extraction unit: used to extract six feature points of the upper left, lower left, upper middle, lower middle, upper right and lower right of the mask in each mask image; A face image acquisition unit: used to acquire face images of m persons without masks; The facial key point extraction unit is used to detect six feature points of each face image, namely, the upper left point of the cheek, the lower left point of the cheek, the middle point of the nose, the middle point of the chin, the upper right point of the cheek and the lower right point of the cheek; Mask face synthesis unit: used to synthesize the face according to the image P j The size and inclination of the face, for any mask image P i The six feature points and any face image P without a mask j The six feature points are matched to synthesize the masked face image P ij ; Brightness adjustment unit: used to adjust the brightness of the face image P without mask j Adjust the light intensity of the face image P wearing a mask ij The brightness of the mask area in the middle; Face recognition unit: used to add all synthesized face images wearing masks to the training set and perform face recognition based on the FaceNet model; The pixel value of the pixel point belonging to the mask area in the mask-wearing face image is f1, and the pixel value of the mask-wearing face image P is adjusted. ij The pixel value of the pixel point belonging to the mask area is f2; according to the face image P without mask j Adjust the light intensity of the face image P wearing a mask ij The brightness of the mask area includes: Calculate the face image P without mask j The light intensity β1, Calculate the mask image P i The light intensity β2, Adjust the face image P wearing a mask ij The pixel value of the pixel point in the middle mask area: f2 = f1 + (β1-β2); Among them, max(p i (R,G,B))、min(p i (R,G,B)) represents the image P i The maximum and minimum values ​​of the mean values ​​of the green channel, the red channel, and the blue channel of all pixels; max(p j (R,G,B))、min(p j (R,G,B)) image P j The maximum and minimum values ​​of the green channel mean, red channel mean and blue channel mean of all pixels; Based on the SDM facial feature detection method, six feature points of the upper left point of the cheek, the lower left point of the cheek, the middle point of the nose, the middle point of the chin, the upper right point of the cheek and the lower right point of the cheek are detected; wherein the loss function in the training process based on the SDM facial feature detection method is: L = a1 × Loss1 + a2 × Loss2; Among them, Loss1 is the loss function of the original SDM face feature point detection model, t k is the kth feature point detected based on the SDM face feature detection method, x′ k is the mask image P i The kth feature point of the six feature points, a1 and a2 are weighting coefficients and a1+a2=1.

Images