Standardize the identification methods, devices, and storage media for wearing masks.

Abstract

This application discloses a method and related equipment for identifying whether a user is wearing a mask correctly. The method includes: inputting a target face image into an image recognition model to obtain an image recognition result; if the image recognition result includes evidence of a mask being worn, determining whether the image recognition result includes the coordinates of a nose key point; if the image recognition result includes the coordinates of a nose key point, determining the detection score corresponding to the nose key point; if the detection score is greater than a preset value, determining that the target user is not wearing a mask correctly. This invention can quickly and accurately identify whether a user is wearing a mask and whether the mask is worn correctly.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to a method, device, and storage medium for identifying the proper wearing of face masks. Background Technology

[0002] With the widespread application of deep learning, deep neural networks have achieved remarkable results in image classification, object detection, semantic segmentation, and other fields. Deep learning, with its advantages of speed, scalability, and end-to-end learning, has spawned many detection frameworks. Among these, deep learning-based object detection algorithms can be mainly divided into two types: one-stage object detection algorithms, represented by RCNN and candidate region-based deep learning object detection algorithms (RCNN, SPP-NET, Fast-RCNN, Faster-RCNN, etc.); and two-stage object detection algorithms, represented by YOLO and regression-based deep learning object detection algorithms (YOLO, SSD, etc.).

[0003] Most patents currently available for mask-wearing detection methods can only detect whether a person is wearing a mask, but cannot further determine whether the person is wearing the mask correctly. However, in actual application scenarios, there are often cases where people do not wear masks correctly, which renders the mask ineffective. Summary of the Invention

[0004] This application provides a method, device, and storage medium for identifying whether a user is wearing a mask correctly.

[0005] The first aspect of this application provides a method for identifying the proper wearing of face masks, which may include:

[0006] Obtain the facial image of the target user, where the target user is the user whose mask is being identified as being worn correctly;

[0007] The face image is preprocessed to obtain the target face image;

[0008] The target face image is input into an image recognition model to obtain the image recognition result;

[0009] If the image recognition result includes a mask being worn, then determine whether the image recognition result includes the coordinates of the key points of the nose;

[0010] If the image recognition result contains the coordinates of the key points of the nose, then the detection score corresponding to the key points of the nose is determined;

[0011] If the detection score is greater than a preset value, it is determined that the target user is not wearing a mask properly.

[0012] In one possible design, the image recognition model includes a feature extraction network structure, a convolutional neural network structure, a nose keypoint regression network structure, and a binary classification network structure. The feature extraction network structure is used to extract facial features and mask features from the target face image. The convolutional neural network structure is used to determine the coordinates of the region of interest in the face in the target face image. The nose keypoint regression network structure is used to determine the coordinates of the nose keypoints in the target face image. The binary classification network structure is used to determine whether the target user is wearing a mask.

[0013] In one possible design, the method further includes:

[0014] Obtain a training image dataset, which includes face images of people wearing masks correctly, face images of people wearing masks incorrectly, and face images of people not wearing masks.

[0015] Each face image in the training image dataset is preprocessed, including data augmentation and data enlargement.

[0016] Extract facial features from each face image in the preprocessed training image dataset;

[0017] Extract target features from at least one region of interest in the target face features, wherein the target face features are the face features of the target face image, and the target face image is any one of multiple face images in the training image dataset;

[0018] Determine the first loss function, the second loss function, and the third loss function;

[0019] Construct a joint loss function based on the first loss function, the second loss function, and the third loss function;

[0020] Based on the first feature, convolutional layer, fully connected layer, and joint loss function, iterative calculations are performed until a preset iteration termination condition is reached to obtain the image recognition model.

[0021] In one possible design, constructing the joint loss function based on the first loss function, the second loss function, and the third loss function includes:

[0022] The joint loss function is constructed as follows:

[0023] ;

[0024] in, Let the joint loss function be... The weights of the first loss function are... The weights of the second loss function are... The weights of the third loss function, Let the first loss function be... For the second loss function, Let be the third loss function, where:

[0025] , The intersection-union loss function is... , For the at least one face region of interest, As the gold standard, For bias terms, , The gold standard for key points of the nose Regression coordinates of key points of the nose The Euclidean distance between them For the at least one face region of interest and the gold standard The diagonal distance of the smallest closure rectangle;

[0026] , The index of the i-th image in the training image dataset. The total number of face images in the training image dataset. The predicted coordinate vector of the key points of the nose. This is the gold standard coordinate vector of the key points of the nose. The gold standard for determining whether a nose landmark exists in the i-th image is given, and , , ;

[0027] , For the first Each image is predicted to be the probability of wearing a mask, and .

[0028] In one possible design, the method further includes:

[0029] Determine whether the number of iterations has reached a preset value. If so, determine that the preset iteration termination condition is met.

[0030] or,

[0031] Determine whether the model parameters of the image recognition model have converged. If so, determine that the preset iteration termination condition is met.

[0032] In one possible design, preprocessing the face image to obtain the target face image includes:

[0033] Data augmentation is performed on the face image to obtain a first face image;

[0034] The first face image is augmented to obtain the target face image, wherein the data augmentation includes at least one combination of cropping, translation, and rotation.

[0035] In one possible design, the data augmentation of the face image to obtain the first face image includes:

[0036] The first face image is obtained by performing data augmentation on the face image using the following formula:

[0037] ;

[0038] in, The face image, The first face image.

[0039] The second aspect of this application provides a device for recognizing the proper wearing of face masks, including:

[0040] The acquisition unit is used to acquire the facial image of the target user, wherein the target user is the user to be identified as to whether he / she is wearing a mask properly;

[0041] The preprocessing unit is used to preprocess the face image to obtain the target face image;

[0042] The recognition unit is used to input the target face image into the image recognition model to obtain the image recognition result;

[0043] The judgment unit is used to determine whether the image recognition result includes the coordinates of the nose key point if the image recognition result includes a mask being worn.

[0044] The determining unit is configured to determine the detection score corresponding to the key nose point if the image recognition result contains the coordinates of the key nose point;

[0045] The determining unit is further configured to determine that the target user is not wearing a mask properly if the detection score is greater than a preset value.

[0046] In one possible design, the device further includes:

[0047] Training unit, the training unit is used for:

[0048] Obtain a training image dataset, which includes face images of people wearing masks correctly, face images of people wearing masks incorrectly, and face images of people not wearing masks.

[0049] Each face image in the training image dataset is preprocessed, including data augmentation and data enlargement.

[0050] Extract facial features from each face image in the preprocessed training image dataset;

[0051] Extract target features from at least one region of interest in the target face features, wherein the target face features are the face features of the target face image, and the target face image is any one of multiple face images in the training image dataset;

[0052] Determine the first loss function, the second loss function, and the third loss function;

[0053] Construct a joint loss function based on the first loss function, the second loss function, and the third loss function;

[0054] Based on the first feature, convolutional layer, fully connected layer, and joint loss function, iterative calculations are performed until a preset iteration termination condition is reached to obtain the image recognition model.

[0055] In one possible design, the training unit constructs a joint loss function based on the first loss function, the second loss function, and the third loss function, including:

[0056] The joint loss function is constructed as follows:

[0057] ;

[0058] in, Let the joint loss function be... The weights of the first loss function are... The weights of the second loss function are... The weights of the third loss function, Let the first loss function be... For the second loss function, Let be the third loss function, where:

[0059] , The intersection-union loss function is... , For the at least one face region of interest, As the gold standard, For bias terms, , The gold standard for key points of the nose Regression coordinates of key points of the nose The Euclidean distance between them For the at least one face region of interest and the gold standard The diagonal distance of the smallest closure rectangle;

[0060] , The index of the i-th image in the training image dataset. The total number of face images in the training image dataset. The predicted coordinate vector of the key points of the nose. This is the gold standard coordinate vector of the key points of the nose. The gold standard for determining whether a nose landmark exists in the i-th image is given, and , , ;

[0061] , For the first Each image is predicted to be the probability of wearing a mask, and .

[0062] In one possible design, the training unit is further used for:

[0063] Determine whether the number of iterations has reached a preset value. If so, determine that the preset iteration termination condition is met.

[0064] or,

[0065] Determine whether the model parameters of the image recognition model have converged. If so, determine that the preset iteration termination condition is met.

[0066] In one possible design, the preprocessing unit is specifically used for:

[0067] Data augmentation is performed on the face image to obtain a first face image;

[0068] The first face image is augmented to obtain the target face image, wherein the data augmentation includes at least one combination of cropping, translation, and rotation.

[0069] In one possible design, the preprocessing unit performs data augmentation on the face image to obtain a first face image, including:

[0070] The first face image is obtained by performing data augmentation on the face image using the following formula:

[0071] ;

[0072] in, The face image, The first face image.

[0073] A third aspect of this application provides a terminal device, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, which causes the processor to perform the operation of the method for identifying the proper wearing of a mask.

[0074] A fourth aspect of this application provides a computer-readable storage medium storing at least one executable instruction that, when executed on a computing device, causes the computing device to perform the identification method for properly wearing a mask as described in the first aspect of this application.

[0075] The fifth aspect of this application discloses a computer program product that, when run on a computer, causes the computer to execute the identification method for properly wearing a mask as described in the first aspect of this application.

[0076] The sixth aspect of this application discloses an application publishing platform for publishing computer program products, wherein when the computer program products are run on a computer, the computer executes the identification method for properly wearing masks as described in the first aspect of this application.

[0077] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0078] The image recognition model identifies the facial images of target users. This model uses a multi-task learning approach to simultaneously detect key points on the nose in the target facial image and classify whether a mask is being worn. Combining the results of both, it further judges the standardization of the user's mask wearing. It can quickly and accurately identify whether a user is wearing a mask and whether the mask is being worn correctly, thereby preventing people from not wearing masks in public places and reducing the spread of respiratory infectious diseases. Attached Figure Description

[0079] The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0080] Figure 1 A schematic diagram illustrating an embodiment of the method for identifying the proper wearing of masks provided in this application;

[0081] Figure 2 A schematic diagram illustrating the training process of the image recognition model provided in this application embodiment;

[0082] Figure 3 This is a schematic diagram of the feature extraction network provided in an embodiment of this application;

[0083] Figure 4 This is a schematic diagram of the network structure of the feature channel attention residual module provided in an embodiment of this application;

[0084] Figure 5 This is a schematic diagram of the network structure of the dimension reduction residual module provided in the embodiments of this application;

[0085] Figure 6 This is a schematic diagram of the network structure of a multi-scale module provided in an embodiment of this application;

[0086] Figure 7 This is a schematic diagram of the face region of interest network structure provided in an embodiment of this application;

[0087] Figure 8 A schematic diagram of the convolutional neural network structure provided in the embodiments of this application;

[0088] Figure 9 A schematic diagram illustrating the design of the bias term for the face region of interest regression loss function provided in this application embodiment;

[0089] Figure 10 A virtual structural diagram of the mask-wearing recognition device provided in the embodiments of this application;

[0090] Figure 11 This is a schematic diagram of the structure of the terminal device in the embodiments of this application. Detailed Implementation

[0091] To enable those skilled in the art to better understand the present application, the technical solutions of the embodiments of the present application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. All embodiments based on the present application should fall within the scope of protection of the present application.

[0092] The following describes the method for identifying properly worn masks provided in this application, using the proper mask-wearing recognition device as an example.

[0093] Please see Figure 1 , Figure 1 A flowchart illustrating the method for identifying the proper wearing of face masks provided in this application embodiment:

[0094] Step 101: Obtain the face image corresponding to the target user.

[0095] In this embodiment, when the device for recognizing whether a target user is wearing a mask correctly needs to identify whether the target user is wearing a mask correctly, it can acquire the facial image of the target user. The target user is the user whose mask is to be identified. The method of acquiring the facial image of the target user is not limited here. For example, it can be acquired by a camera set on the device for recognizing whether the mask is worn correctly. Of course, it can also be acquired by other methods, such as acquiring the facial image of the target user through an independent camera device, and then sending the facial image of the target user to the device for recognizing whether the mask is worn correctly. The specific method is not limited.

[0096] Step 102: Preprocess the face image to obtain the target face image.

[0097] In this embodiment, after obtaining the facial image corresponding to the target user, the mask-wearing recognition device can perform data augmentation on the facial image to obtain a first facial image. Then, it can perform data amplification on the first facial image to obtain the target facial image. This data amplification includes at least one combination of cropping, translation, and rotation. In other words, since the identification of whether a user is wearing a mask correctly is performed using an image recognition model, preprocessing of the facial image is required before inputting it into the model. The main purpose of preprocessing is to enhance the recognizability of relevant information, thereby improving the accuracy of convolutional neural network feature extraction and recognition steps. In this application, the preprocessing of the facial image is mainly achieved through data augmentation and data amplification. Specifically, data augmentation uses grayscale stretching, and the first facial image can be obtained by performing data augmentation using the following formula:

[0098] ;

[0099] in, For facial images, For the first face image, grayscale stretching was applied. This reduced noise interference in the region of interest and increased the anisotropy between the target and the background, thus accelerating the convergence speed of the deep convolutional neural network. Data augmentation used a combination of cropping, translation, and rotation to expand the dataset and address the various problems caused by dataset imbalance.

[0100] Step 103: Input the target face image into the image recognition model to obtain the image recognition result.

[0101] In this embodiment, after preprocessing the face image to obtain the target face image, the mask-wearing recognition device can input the target face image into a pre-trained image recognition model to obtain the image recognition result. The image recognition result includes whether a mask is worn and whether the coordinates of the nose key point are included. The image recognition model is an end-to-end convolutional neural network model for key point detection with enhanced region of interest features. It adopts a multi-task learning method to simultaneously realize the detection of the nose key point on the image to be recognized and the binary classification of whether a mask is worn. Combining the results of the two, it further judges the standardization of mask wearing. While ensuring the real-time performance of the algorithm, it also has stronger generalization and robustness, and higher detection accuracy.

[0102] It should be noted that this image recognition model includes a feature extraction network structure, a convolutional neural network structure, a nose keypoint regression network structure, and a binary classification network structure. The feature extraction network structure is used to extract the facial features (features corresponding to the eyes, nose, mouth, ears, and eyebrows) and mask features of the target face image. The convolutional neural network structure is used to determine the coordinates of the region of interest in the face in the target face image. The nose keypoint regression network structure is used to determine the coordinates of the nose keypoints in the target face image. The binary classification network structure is used to determine whether the target user is wearing a mask.

[0103] Step 104: If the image recognition result includes a mask being worn, determine whether the image recognition result includes the coordinates of the nose key point. If yes, proceed to step 105; otherwise, proceed to step 107.

[0104] In this embodiment, after obtaining the image recognition result through the image recognition model, the mask-wearing recognition device can determine whether the image recognition result includes a mask being worn. If the image result includes a mask being worn, it means that the target user is a user who is already wearing a mask. Then, it can continue to determine whether the image recognition result includes the coordinates of the nose key point. If the image recognition result includes the coordinates of the nose key point, step 105 is executed. If the image recognition result does not include the coordinates of the nose key point, step 107 is executed.

[0105] Step 105: If the image recognition result includes the coordinates of the nose key points, then determine the detection score corresponding to the nose key points.

[0106] In this embodiment, when the mask-wearing recognition device determines that the image recognition result includes the coordinates of the nose key point, it determines the detection score corresponding to the nose key point. The specific method for calculating the detection score corresponding to the nose key point is not limited here, as long as the detection score of the nose key point can be calculated. For example, the detection score of the nose key point can be calculated by matching the target face image with the score image template through image template matching.

[0107] Step 106: If the detection score is greater than the preset value, it is determined that the target user is not wearing a mask properly.

[0108] In this embodiment, the mask-wearing recognition device can be pre-set with a preset value (e.g., 0.8, or other preset values, not limited to any specific value). After obtaining the detection score corresponding to the key points of the nose, it can be determined whether the detection score is greater than the preset value. If the detection score is greater than the preset value, it is determined that the target user's nose is exposed outside the mask, indicating that the target user is not wearing the mask properly. It is understood that when it is determined that the target user is not wearing the mask properly, a "Mask not worn properly" prompt message can be output to remind the user to wear the mask properly.

[0109] Step 107: Perform other operations.

[0110] In this embodiment, when the image recognition result does not include the coordinates of the key points of the nose, it indicates that the target user is a user who is wearing a mask properly, so no operation is performed or a "wearing a mask properly" prompt message is output; when the mask classification result in the image recognition result includes not wearing a mask, a "not wearing a mask" prompt message is directly output.

[0111] In summary, it can be seen that in the embodiments provided in this application, the facial image of the target user is identified by an image recognition model. This image recognition model adopts a multi-task learning method to simultaneously detect the key points of the nose on the target facial image and classify whether a mask is being worn. Combining the results of both, it further judges the standardization of the user's mask wearing. It can quickly and accurately identify whether the user is wearing a mask and whether the mask is being worn correctly, thereby preventing people from not wearing masks in public places and reducing the spread of respiratory infectious diseases.

[0112] The following is combined with Figure 2 The training process of the image recognition model provided in the embodiments of this application is described below. Please refer to [link to relevant documentation]. Figure 2 , Figure 2 The images provided for embodiments of this application are schematic diagrams of the training process of the model, including:

[0113] Step 201: Obtain the training image dataset.

[0114] In this embodiment, the mask-wearing recognition device can acquire a training image dataset, which includes multiple facial images of people wearing masks correctly, people wearing masks incorrectly, and people not wearing masks.

[0115] Step 202: Preprocess each face image in the training image dataset.

[0116] In this embodiment, after acquiring the training image dataset, the mask-wearing recognition device can preprocess each face in the training image dataset, including data augmentation and data amplification.

[0117] It should be noted that the specific preprocessing operations have been explained in detail in step 102 above, and will not be repeated here. Additionally, the preprocessed training image dataset can be divided according to a certain ratio, such as a 1:9 ratio, with 9 parts used to train the image recognition model and 1 part used to validate the trained model. Of course, other ratios are also possible, such as 2:8; no specific limitation is imposed.

[0118] Step 203: Extract facial features from each face in the preprocessed training image dataset.

[0119] In this embodiment, the mask-wearing recognition device can extract the facial features corresponding to each preprocessed face image. Specifically, each face image can be input into a feature extraction network structure to obtain facial features, that is, to obtain the global features of each face image. The facial features include facial eyebrow features, facial respiratory organ features, facial visual organ features, facial oral organ features, facial auditory organ features, and mask features. Of course, it also includes other features, such as hair features, forehead features, etc., which are not specifically limited.

[0120] The feature extraction network structure includes a feature channel attention residual module, a dimensionality reduction residual module, and a multi-scale module. The feature channel attention residual module corrects the balance between the foreground and background regions of the target face image. The dimensionality reduction residual module reduces the dimensionality of the features in each face image and aggregates the reduced features. The multi-scale module restores the feature scale of each face image. This application proposes an improved feature extraction network structure based on the traditional convolutional neural network structure. By incorporating the feature channel attention residual module, the dimensionality reduction residual module, and the multi-scale module, it better extracts facial features, masks, and other special structures. The network structure is as follows: Figure 3 As shown, the following are combined with Figures 4 to 6 The feature channel attention residual module, dimensionality reduction residual module, and multi-scale module of the feature extraction network structure are explained below:

[0121] I. Feature Channel Attention Residual Module;

[0122] Convolution operations aggregate information across feature dimensions, and features from different channels have equal weights when the feature map is further processed. In tasks such as facial nose landmark detection and mask classification, the foreground region (face) is relatively small compared to the entire image, which is clearly unreasonable as the network depth increases. Therefore, this application adds a feature channel attention residual module to the feature extraction network structure. This module addresses the imbalance between foreground and background regions during feature extraction from facial images. Please refer to [link to relevant documentation]. Figure 4 , Figure 4 A schematic diagram of the network structure of the feature channel attention residual module provided in this application is as follows:

[0123] The feature channel attention residual module consists of two branches, with branch 401 consisting of a global pooling layer and two convolutional kernels of size [missing information]. It consists of convolutional layers with a stride of 1, assuming the input feature map... Size is First, feature map The feature maps are fed into a global pooling layer, where each 2D feature map is compressed along the spatial dimension to obtain a real number. This real number, to some extent, possesses a global receptive field. At this point, the size of the feature map becomes... Since the number of feature channels in the input and output are the same, the feature map at this time... It represents the global distribution of responses on feature channels, and even layers close to the input have a global receptive field.

[0124] Then, it first goes through a fully connected layer to reduce the number of feature channels to that of the input. The feature map is then activated by a Rectified Linear Unit (ReLU) layer, followed by a convolutional layer with the same kernel size to increase the number of channels back to the original dimension. A fully connected layer is used to perform dimensionality reduction followed by dimensionality increase on the feature map. Dimensionality reduction significantly reduces the number of parameters and enables cross-channel information combination. A subsequent non-linear activation function greatly increases non-linear features. Finally, a sigmoid gate function is used to obtain... The normalized weights between the input and output feature vectors indicate the importance of each feature channel. Multiplying this feature vector with the input feature map after two convolutions yields the output of the first branch, thus enabling feature capture from the global receptive field.

[0125] Branch 402 employs the concept of a residual structure to address the performance degradation and gradient issues caused by increasing network depth. This is achieved by processing the input feature map... It is directly connected to the recalibrated feature map, and the output of the entire module is obtained through a ReLU activation function, i.e., the output is... .

[0126] II. Dimensionality Reduction Residual Module;

[0127] Please see Figure 5 , Figure 5 This is a schematic diagram of the network structure of the dimension reduction residual module provided in the embodiments of this application. The dimension reduction residual module is used to process feature maps. Dimensionality reduction is performed, and the dimension reduction residual module is also divided into two branches, branch 501 and branch 502.

[0128] Among them, branch 501 consists of a convolution kernel size of It consists of convolutional layers 5011 with a stride of 2. Branch 502 first passes through a convolutional kernel with a size of 2. A convolutional layer with a stride of 2 (5021) is used to process the feature map. Dimensionality reduction; then through a convolution kernel of size... A convolutional layer with a stride of 1 (5022) is used to aggregate features; finally, a convolutional kernel with a size of 1 is used. A 5023 convolutional layer with a stride of 1 eliminates redundant connections, reducing parameters and computational cost.

[0129] This residual module uses convolutional layers with a stride of 2 instead of pooling layers. Since pooling layers cause the loss of feature correlations of some feature map elements, using convolutional layers with a stride of 2 instead can improve accuracy without affecting computational efficiency.

[0130] III. Multi-scale modules;

[0131] Since the size of the region of interest (ROI) and the key points of the nose differ greatly from the size of the original input image, if the same receptive field size is used, the feature extraction network structure is very likely to lose its focus on the foreground information of the target ROI after the network is deepened. Therefore, a multi-scale module is added to the feature extraction network structure to restore the feature scale step by step.

[0132] Please see Figure 6 , Figure 6 This is a schematic diagram of the network structure of the multi-scale module provided in the embodiments of this application. Unlike the traditional pyramid structure or feature layering, the multi-scale module in this application consists of two convolutional kernels with a size of [missing information]. A convolutional layer with a stride of 1 and a convolutional kernel with a size of 1 The system consists of deconvolutional layers with a stride of 2, which map high-dimensional features back to low-dimensional features. Each resolution feature map is processed by a multi-scale module, scaled by a factor of two, and then the backbone of the feature extraction is performed (e.g., ...). Figure 3The feature maps of the corresponding resolutions are superimposed on each layer of the backbone 301. Through this connection, the feature maps of each layer in the backbone 301 fuse features of different resolutions, including high-dimensional semantic information and low-dimensional texture information, thereby achieving the fusion of feature maps of different resolutions and allowing the network to balance its attention to different resolution sizes. At the same time, since this method only adds extra cross-layer connections to the original network, it adds almost no extra time and computation in practical applications.

[0133] It should be noted that since the center point of the nose is only a single pixel in the entire image to be detected, it is unreasonable to use the global feature map to regress its coordinate position (the foreground and background regions are extremely unbalanced). In this application, we first extract the feature map containing the region of interest of the face from the global features, and then simultaneously regress the region of interest of the face, detect the nose key points, and classify whether a mask is being worn.

[0134] Step 204: Extract target features from at least one region of interest of the target face.

[0135] In this embodiment, after preprocessing the training image dataset, the mask-wearing recognition device can randomly select a target facial feature for processing. This target facial feature is the facial feature of a target face image, which is any one of multiple face images in the training image dataset. Then, it extracts at least one region of interest from the target facial feature. The following describes how... Figure 7 To explain, Figure 7 This is a schematic diagram of the network structure of the region of interest (ROI) for the face provided in the embodiment of this application (i.e., the convolutional neural network structure mentioned above). The network structure uses two branches to distinguish the ROI from the global features (i.e., the target face features) and to learn the local feature maps of facial features, masks, etc. by regressing the offset of the ROI. Branch 701 distinguishes between the foreground and the background through the Softmax classification layer (the ROI of the face is the foreground by default). Branch 702 calculates the boundary offset of the real ROI of the face by using the candidate boxes of the face region given by prior knowledge to obtain accurate positioning. Finally, the feature maps output by the two parts are multiplied to obtain the local feature map of the face region.

[0136] Based on prior knowledge, this embodiment selects five candidate bounding boxes for facial regions of interest (ROIs) of different sizes. For branch 701, feature map 703 is first fed into a convolutional layer 7011 with a kernel size of 1, a stride of 1, and 10 output channels for convolution. The choice of 10 output channels corresponds to the candidate bounding boxes, as each candidate bounding box could be either a foreground or background region. Then, feature map reshaping 7012 reshapes the feature map 703, increasing its dimensionality. The data is then fed into the Softmax layer 7013 for foreground and background classification, determining whether feature map 703 is a region of interest for a face. Since it has undergone the Softmax normalization exponential function, the feature map output by branch 701 is now essentially a feature vector with values ​​close to 0 or 1, used to distinguish between foreground and background regions, and no longer contains information from the original feature map.

[0137] Branch 702 passes through only one convolutional layer with a kernel size of 1, a stride of 1, and 10 channels, preserving the information of feature map 703. The output feature maps of branches 701 and 702 are multiplied along the channel dimension of the feature maps to obtain the facial interest region feature map (feature maps judged as foreground regions are multiplied by a number close to 1 and are completely preserved, while feature maps judged as background regions are multiplied by a number close to 0 and are thus eliminated).

[0138] Feature Map Reshaping 7012 upgrades the feature map to a higher dimension. This is necessary to perform softmax on the entire feature map, thereby distinguishing between foreground and background regions. Afterward, feature map reshaping (7014) is performed to reduce the dimensionality of the feature map. This allows it to be multiplied with the output of branch 702.

[0139] Understandably, after the facial region of interest feature map is processed through a max pooling layer, it can be used to regress the offset of candidate bounding boxes and the coordinates of nose key points, as well as to classify whether a mask is being worn. Below, we will combine... Figure 8 To explain, Figure 8 The diagram shows the convolutional neural network structure, the nose keypoint regression network structure, and the binary classification network structure provided in the embodiments of this application.

[0140] Step 205: Determine the first loss function, the second loss function, and the third loss function.

[0141] In this embodiment, the loss function corresponding to the convolutional neural network structure is the first loss function, the loss function corresponding to the nose keypoint regression network structure is the second loss function, and the loss function of the binary classification network structure is the third loss function. The following will combine... Figure 8The determination of the three loss functions is explained below:

[0142] 1. First loss function:

[0143] Figure 8 Branch 801 is used to calculate candidate bounding boxes for the face region of interest. The output feature map is passed through three convolutional layers 8011 with a kernel size of 3 and a stride of 1, and then through two fully connected layers 8012. It is then directly fed into the loss function and iteratively calculated with the gold standard. The offset is divided into four values, namely, scaling and translation. Assuming the input is... , These are the coordinates of the feature map (i.e., the target features) mapped to the original image (i.e., the target face image). It is the original width and length of the candidate bounding box for the region of interest of the face, the target The conversion formula is:

[0144] ;

[0145] ;

[0146] ;

[0147] ;

[0148] in, , , , It can be estimated through linear regression:

[0149] ;

[0150] It is the gold standard. It is the input feature vector. This is the bias mapping relationship that needs to be calculated. The gold standard for the candidate bounding box regression branch of the face region of interest is the true coordinates of the face region, using (x, y, w, h), which represent the center point coordinates and width and height of the face region (the entire face region), respectively.

[0151] When determining the loss function for branch 801, the positions of the nose keypoints are combined and constrained with the coordinates of the region of interest on the face. The loss function for branch 801 (i.e., the first loss function) is defined as follows:

[0152] ;

[0153] The loss function is in cross-union ratio The loss function was improved by adding a bias term. .in, , To predict candidate bounding boxes (i.e., at least one region of interest for a face). It is the gold standard. It is used for evaluation and The distance is the most direct indicator, but considering that if we directly use... As a loss function, It cannot accurately reflect the degree of overlap between the predicted candidate box and the gold standard. Therefore, combining the relationship between the nose keypoint and the face region location, in A bias term was added to the existing one, and the bias term was defined as follows:

[0154] ;

[0155] in, The gold standard for key points of the nose (i.e., the true coordinates of the nose key points) and the regression coordinates of the nose key point regression branch 802. European distance, for and The diagonal distance of the smallest closure rectangle, such as... Figure 9 As shown.

[0156] Understandably, since the gold standard for nose landmarks lies within the face region, this bias term constrains the overlap and scale between the face region gold standard and the predicted candidate boxes, thus avoiding... The divergence problem during training. When predicting candidate boxes. Gold Standard Even when there is no overlap, the bounding box can still be given a direction of movement through the constraint of the bias term; and the first loss function It can accelerate the convergence of the loss function by minimizing the predicted and actual coordinates of the nose key points.

[0157] 2. Second loss function;

[0158] Branch 802 is used to regress the coordinates of the key points on the nose. To obtain more accurate results, convolutional operations are still needed on the feature map to further extract features before feeding it into a fully connected layer for regression calculation. Here, 5 convolutional kernels of size [missing information] are used. After further feature extraction by convolutional layer 8021 with a stride of 1, the data is fed into fully connected layer 8022, which outputs the coordinates of the key points of the nose. Understandably, the output of the nose keypoint detection is also estimated using linear regression:

[0159] ;

[0160] in, It is the gold standard. It is the input feature vector. The second loss function is defined as the mapping relationship between the features and coordinate points that needs to be calculated.

[0161] ;

[0162] in, It is the index of the input image in the training image dataset (that is, which image in the training image dataset the input image is). It is the total number of training images included in the training image dataset. This is the predicted coordinate vector of the key points of the nose. This is the gold standard coordinate vector of the key points of the nose. The gold standard for determining whether a key point of the nose exists. The definition is as follows:

[0163] ;

[0164] in, .use After the loss function, when the predicted nose keypoint coordinates are small in difference from the gold standard, the gradient will not be too large; when the difference is large, the gradient value is small enough to avoid gradient explosion, which is beneficial to model training.

[0165] 3. Third loss function:

[0166] Branch 803 is used for binary classification of whether a face region is wearing a mask. It requires three convolutional kernels of a certain size as input to the feature map. A convolutional layer 8031 ​​with a stride of 1 is used to perform convolution operations to further extract features, which are then fed into a fully connected layer 8032, and finally pass through a softmax layer 8033 to obtain the classification result. The third loss function is defined as follows:

[0167] ;

[0168] Using binary cross-entropy, where It is the index of the input image in the training image dataset. It is the total number of training image datasets. For the first The probability of each image being associated with wearing a mask is predicted to be... For the first The gold standard true label for whether a nose keypoint exists in an image is 0 for non-existence and 1 for existence. That is, when... When, the i-th image does not contain a nose keypoint, when At that time, the i-th image contains a nose key point.

[0169] Step 206: Construct a joint loss function based on the first loss function, the second loss function, and the third loss function.

[0170] In this embodiment, after defining the first loss function, the second loss function, and the third loss function, the mask-wearing recognition device can construct a joint loss function based on the first loss function, the second loss function, and the third loss function, as follows:

[0171] ;

[0172] Based on the difficulty level of the three branches, to avoid the relatively simple tasks dominating the entire training process, the weights of the loss functions for the three tasks—face region of interest regression, nose keypoint regression, and mask-wearing classification—were adjusted accordingly. , , (Of course, you can also set it according to the actual situation; there are no specific limitations.)

[0173] It should be noted that, after adopting the network structure provided in this application, the simultaneous learning of the three branch tasks can improve the generalization of the model, mainly in the following ways: First, the unique parts of each branch task are equivalent to noise for other tasks, and adding noise during the learning process can improve generalization; Second, since the local minima of each task are different, the interaction of different branches can make the shared hidden layer escape local extrema; Third, the shared features of the tasks can weaken the fitting ability to a specific task, thereby preventing the network from overfitting.

[0174] Step 207: Perform iterative calculations based on the first feature, convolutional layer, fully connected layer, and joint loss function until a preset iteration termination condition is reached to obtain the image recognition model.

[0175] In this embodiment, after constructing the joint loss function, the mask-wearing recognition device can iteratively calculate the target features using convolutional layers and fully connected layers based on this joint loss function until a preset iteration termination condition is reached, thus obtaining an image recognition model. In other words, during the iterative calculation process, it can be determined whether the number of iterations has reached a preset value; if so, the preset iteration termination condition is satisfied. Alternatively, it can be determined whether the model parameters of the image recognition model have converged; if so, the preset iteration termination condition is satisfied.

[0176] In summary, it can be seen that in the embodiments provided in this application, during the training of the image recognition model, feature extraction network structure is used to extract features from the face image, and a multi-task learning method is adopted to simultaneously detect the key points of the nose on the target face image and classify whether a mask is worn. Combining the results of both, the standardization of the user's mask wearing is further judged. This can quickly and accurately identify whether the user is wearing a mask and whether the mask is worn correctly, thereby preventing people from not wearing masks in public places and reducing the spread of respiratory infectious diseases.

[0177] The above describes this application from the perspective of identifying the correct way to wear a mask. The following describes this application from the perspective of identifying the correct way to wear a mask.

[0178] Please see Figure 10 , Figure 10 This is a virtual structural diagram of a mask-wearing recognition device provided in an embodiment of this application. The mask-wearing recognition device 1000 includes:

[0179] The acquisition unit 1001 is used to acquire the facial image of the target user, wherein the target user is the user to be identified as to whether he or she is wearing a mask properly;

[0180] Preprocessing unit 1002 is used to preprocess the face image to obtain the target face image;

[0181] The recognition unit 1003 is used to input the target face image into an image recognition model to obtain an image recognition result. The image recognition model includes a feature extraction network structure, a convolutional neural network structure, a nose keypoint regression network structure, and a binary classification network structure. The feature extraction network structure is used to extract facial features and mask features from the target face image. The convolutional neural network structure is used to determine the coordinates of the region of interest in the face in the target face image. The nose keypoint regression network structure is used to determine the coordinates of the nose keypoints in the target face image. The binary classification network structure is used to determine whether the target user is wearing a mask.

[0182] The judgment unit 1004 is used to determine whether the image recognition result includes the coordinates of the nose key point if the image recognition result includes a mask being worn.

[0183] The determining unit 1005 is used to determine the detection score corresponding to the key nose point if the image recognition result contains the coordinates of the key nose point;

[0184] The determining unit 1005 is further configured to determine that the target user is not wearing a mask properly if the detection score is greater than a preset value.

[0185] In one possible design, the device further includes:

[0186] Training unit 1006, the training unit 1006 is used for:

[0187] Obtain a training image dataset, which includes face images of people wearing masks correctly, face images of people wearing masks incorrectly, and face images of people not wearing masks.

[0188] Each face image in the training image dataset is preprocessed, including data augmentation and data enlargement.

[0189] Extract facial features from each face image in the preprocessed training image dataset;

[0190] Extract target features from at least one region of interest in the target face features, wherein the target face features are the face features of the target face image, and the target face image is any one of multiple face images in the training image dataset;

[0191] Determine the first loss function, the second loss function, and the third loss function;

[0192] Construct a joint loss function based on the first loss function, the second loss function, and the third loss function;

[0193] Based on the first feature, convolutional layer, fully connected layer, and joint loss function, iterative calculations are performed until a preset iteration termination condition is reached to obtain the image recognition model.

[0194] In one possible design, the training unit 1006 constructs a joint loss function based on the first loss function, the second loss function, and the third loss function, including:

[0195] The joint loss function is constructed as follows:

[0196] ;

[0197] in, Let the joint loss function be... The weights of the first loss function are... The weights of the second loss function are... The weights of the third loss function, Let the first loss function be... For the second loss function, Let be the third loss function, where:

[0198] , The intersection-union loss function is... , For the at least one face region of interest, As the gold standard, For bias terms, , The gold standard for key points of the nose Regression coordinates of key points of the nose The Euclidean distance between them For the at least one face region of interest and the gold standard The diagonal distance of the smallest closure rectangle;

[0199] , The index of the i-th image in the training image dataset. The total number of face images in the training image dataset. The predicted coordinate vector of the key points of the nose. This is the gold standard coordinate vector of the key points of the nose. The gold standard for determining whether a nose landmark exists in the i-th image is given, and , , ;

[0200] , For the first Each image is predicted to be the probability of wearing a mask, and .

[0201] In one possible design, the training unit 1006 is further used for:

[0202] Determine whether the number of iterations has reached a preset value. If so, determine that the preset iteration termination condition is met.

[0203] or,

[0204] Determine whether the model parameters of the image recognition model have converged. If so, determine that the preset iteration termination condition is met.

[0205] In one possible design, the preprocessing unit 1002 is specifically used for:

[0206] Data augmentation is performed on the face image to obtain a first face image;

[0207] The first face image is augmented to obtain the target face image, wherein the data augmentation includes at least one combination of cropping, translation, and rotation.

[0208] In one possible design, the preprocessing unit 1002 performs data augmentation on the face image to obtain a first face image, including:

[0209] The first face image is obtained by performing data augmentation on the face image using the following formula:

[0210] ;

[0211] in, The face image, The first face image.

[0212] See Figure 11 , Figure 11 This is a schematic diagram of the hardware structure of a terminal device 1100 provided in an embodiment of this application. The terminal device 1100 can execute the aforementioned method for identifying properly worn masks and can be applied to… Figure 1 In the application scenarios shown, and corresponding Figure 1 The standard configuration includes a mask recognition device. The terminal device 1100 can be a smartphone, personal computer, tablet PC, PAD, etc.

[0213] Specifically, such as Figure 11 As shown, the terminal device 1100 includes: at least one processor 1101, at least one network interface 1104 or other user interface 1103, memory 1105, and at least one communication bus 1102. The communication bus 1102 is used to realize the connection and communication between these components. The terminal device 1100 may optionally include a user interface 1103, including a display (e.g., touch screen, LCD, CTR, holographic imaging, or projector), a keyboard, or a clicking device (e.g., mouse, trackball, touchpad, or touch screen).

[0214] Memory 1105 may include read-only memory and random access memory, and provide instructions and data to the processor. A portion of memory 1105 may also include non-volatile random access memory (NVRAM).

[0215] In some embodiments, memory 1105 stores executable modules or data structures, or subsets thereof, or extended sets thereof:

[0216] Operating system 11051 contains various system programs, such as the framework layer, core library layer, and driver layer, which are used to implement various basic business functions and handle hardware-based tasks.

[0217] Application module 11052 contains various applications, such as desktop launcher, media player, and browser, to implement various application services.

[0218] In this embodiment of the application, all the operations performed by the mask-wearing recognition device are realized by calling the program or instructions stored in the memory 1105.

[0219] This application also provides a computer-readable storage medium storing at least one executable instruction, which, when executed on a computing device, causes the computing device to perform the identification method for properly wearing a mask as described in any of the above embodiments.

[0220] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0221] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0222] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0223] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0224] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0225] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0226] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0227] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for identifying proper mask-wearing, characterized in that, include: Obtain the facial image of the target user, where the target user is the user whose mask is being identified as being worn correctly; The face image is preprocessed to obtain the target face image; The target face image is input into an image recognition model to obtain image recognition results. These results include whether a mask is being worn and whether the coordinates of the nose keypoints are included. The image recognition model includes a feature extraction network structure, a convolutional neural network structure, a nose keypoint regression network structure, and a binary classification network structure. The feature extraction network structure extracts facial features and mask features from the target face image. The convolutional neural network structure determines the coordinates of the region of interest (ROI) in the target face image. The nose keypoint regression network structure determines the coordinates of the nose keypoints in the target face image. The binary classification network structure determines whether the target user is wearing a mask. The image recognition model iteratively calculates based on target features, convolutional layers, fully connected layers, and a joint loss function until a preset iteration termination condition is reached. The joint loss function is constructed based on a first loss function, a second loss function, and a third loss function. If the image recognition result includes a mask being worn, then determine whether the image recognition result includes the coordinates of the key points of the nose; If the image recognition result contains the coordinates of the key points of the nose, then the detection score corresponding to the key points of the nose is determined; If the detection score is greater than a preset value, it is determined that the target user is not wearing a mask properly.

2. The method according to claim 1, characterized in that, The method further includes: Obtain a training image dataset, which includes face images of people wearing masks correctly, face images of people wearing masks incorrectly, and face images of people not wearing masks. Each face image in the training image dataset is preprocessed, including data augmentation and data enlargement. Extract facial features from each face image in the preprocessed training image dataset; Extract target features from at least one region of interest in the target face features, wherein the target face features are the face features of the target face image, and the target face image is any one of multiple face images in the training image dataset.

3. The method according to claim 2, characterized in that, The step of constructing a joint loss function based on the first loss function, the second loss function, and the third loss function includes: The joint loss function is constructed as follows: ； in, Let the joint loss function be... The weights of the first loss function are... The weights of the second loss function are... The weights of the third loss function, Let the first loss function be... For the second loss function, Let be the third loss function, where: , The intersection-union loss function is... , For the at least one face region of interest, As the gold standard, For bias terms, , The gold standard for key points of the nose Regression coordinates of key points of the nose The Euclidean distance between them For the at least one face region of interest and the gold standard The diagonal distance of the smallest closure rectangle; , The index of the i-th image in the training image dataset. The total number of face images in the training image dataset. The predicted coordinate vector of the key points of the nose. This is the gold standard coordinate vector of the key points of the nose. The gold standard for determining whether a nose landmark exists in the i-th image is given, and , , ; , For the first Each image is predicted to be the probability of wearing a mask, and .

4. The method according to claim 2, characterized in that, The method further includes: Determine whether the number of iterations has reached a preset value. If so, determine that the preset iteration termination condition is met. or, Determine whether the model parameters of the image recognition model have converged. If so, determine that the preset iteration termination condition is met.

5. The method according to any one of claims 1 to 3, characterized in that, The preprocessing of the face image to obtain the target face image includes: Data augmentation is performed on the face image to obtain a first face image; The first face image is augmented to obtain the target face image, wherein the data augmentation includes at least one combination of cropping, translation, and rotation.

6. The method according to claim 5, characterized in that, The step of performing data augmentation on the face image to obtain the first face image includes: The first face image is obtained by performing data augmentation on the face image using the following formula: ； in, The face image, The first face image.

7. A device for recognizing the proper wearing of face masks, characterized in that, include: The acquisition unit is used to acquire the facial image of the target user, wherein the target user is the user to be identified as to whether he / she is wearing a mask properly; The preprocessing unit is used to preprocess the face image to obtain the target face image; A recognition unit is used to input the target face image into an image recognition model to obtain an image recognition result. The image recognition result includes whether a mask is being worn and whether the coordinates of the nose keypoint are included. The image recognition model includes a feature extraction network structure, a convolutional neural network structure, a nose keypoint regression network structure, and a binary classification network structure. The feature extraction network structure is used to extract facial features and mask features from the target face image. The convolutional neural network structure is used to determine the coordinates of the region of interest in the face in the target face image. The nose keypoint regression network structure is used to determine the coordinates of the nose keypoint in the target face image. The binary classification network structure is used to determine whether the target user is wearing a mask. The image recognition model iteratively calculates based on target features, convolutional layers, fully connected layers, and a joint loss function until a preset iteration termination condition is reached. The joint loss function is constructed based on a first loss function, a second loss function, and a third loss function. The judgment unit is used to determine whether the image recognition result includes the coordinates of the nose key point if the image recognition result includes a mask being worn. The determining unit is configured to determine the detection score corresponding to the key nose point if the image recognition result contains the coordinates of the key nose point; The determining unit is further configured to determine that the target user is not wearing a mask properly if the detection score is greater than a preset value.

8. A terminal device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation of the identification method for wearing a mask as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one executable instruction, which, when executed on a computing device, causes the computing device to perform the identification method for properly wearing a mask as described in any one of claims 1 to 6.

Images