Face mask wearing detection method and device

Abstract

The invention discloses a face mask wearing detection method and device, and relates to the technical field of computers. A specific embodiment of the method comprises the steps that a plurality of face images and annotation data of each face image are acquired, the face images comprise face images wearing masks and face images not wearing masks, and the annotation data comprise face key point positions, mask positions and categories in the face images; the category indicates that the mask is worn or not worn; determining an image feature of each face image; and training according to the image features corresponding to each face image in the plurality of face images and the annotation data to obtain a face mask wearing detection model. According to the embodiment, the face key point position and the mask position and category can be detected, so that the mask wearing specification degree can be determined according to the face key point position and the mask position, automatic detection of the mask wearing specification can be realized, and how to normatively wear the mask can be prompted.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for detecting when a person is wearing a mask on their face. Background Art

[0002] Detecting whether a person is wearing a mask plays an important role in daily life. Wearing a mask can effectively prevent the widespread spread of the virus. In related technologies, research on whether a person is wearing a mask is relatively common, but how to determine the degree of mask wearing standardization and how to provide mask wearing recommendations have received little attention. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a method and device for detecting whether a face is wearing a mask, which can detect the positions of key points on the face, the position and category of the mask, so as to determine the degree of mask wearing standards based on the positions of key points on the face and the position of the mask, thereby realizing automatic detection of mask wearing standards and providing prompts on how to wear the mask in a standardized manner.

[0004] To achieve the above object, according to a first aspect of an embodiment of the present invention, a training method for a face mask wearing detection model is provided, comprising:

[0005] Acquire multiple facial images and annotation data for each facial image, wherein the multiple facial images include facial images of people wearing masks and facial images of people not wearing masks, and the annotation data include positions of key facial points, positions of masks, and categories in the facial images, wherein the categories indicate whether a mask is worn or not worn;

[0006] Determining image features of each of the facial images;

[0007] Training is performed based on the image features and annotation data corresponding to each of the multiple facial images to obtain a facial mask wearing detection model.

[0008] Optionally, determining the image features of each of the facial images includes:

[0009] Using a residual network to obtain feature information of multiple levels of the face image;

[0010] The feature information of the multiple levels is fused using a feature pyramid network to obtain the image features.

[0011] Optionally, training is performed according to image features and annotation data corresponding to each of the multiple facial images to obtain a facial mask wearing detection model, including:

[0012] Using a fully convolutional neural network to classify the image features, and obtain predicted categories, mask positions, and facial key point positions corresponding to the image features;

[0013] Construct a loss function that includes mask classification loss, mask frame regression loss, and face key point loss;

[0014] Based on the predicted category, mask position, facial key point position and annotation data corresponding to each image feature, the loss function is iteratively trained to obtain the facial mask wearing detection model.

[0015] A second aspect of an embodiment of the present invention provides a method for detecting when a face mask is worn, comprising:

[0016] Get the face image to be detected;

[0017] Input the face image to be detected into the face mask wearing detection model to obtain the face key point position, mask position and category of the face image to be detected;

[0018] The mask wearing result of the face image to be detected is determined according to the facial key point positions, mask position and category of the face image to be detected.

[0019] Optionally, determining the mask wearing result of the face image to be detected according to the face key point positions, mask position and category of the face image to be detected includes:

[0020] When the category indicates wearing a mask, the mask wearing standard index value is determined according to the facial key point positions and the mask position; when the mask wearing standard index value is greater than a preset threshold, the mask wearing result is determined to be a result indicating that the mask is worn standardly; when the mask wearing standard index value is less than or equal to the preset threshold, the mask wearing result is determined to be a result indicating that the mask is not worn standardly.

[0021] Optionally, the facial key point positions include positions of multiple key points in the face, and the mask position includes the size and center point of the mask frame; determining the mask wearing standard index value according to the facial key point positions and the mask position includes:

[0022] Determine a preset shape area formed by the plurality of key points;

[0023] Determine whether the multiple key points are within the mask frame;

[0024] When the plurality of key points are all within the mask frame, determine whether the center point of the mask frame is within the preset shape area, and if so, determine the mask wearing standard index value to be a first preset value; if not, determine the mask wearing standard index value according to the distance between the center point of the mask frame and the preset shape area;

[0025] When at least one of the plurality of key points is not within the mask frame, determining the mask wearing standard index value to be a second preset value,

[0026] The first preset value is greater than the preset threshold, and the second preset value is less than the preset threshold.

[0027] Optionally, determining the mask wearing standard index value according to the distance between the center point of the mask frame and the preset shape area includes:

[0028] Determine the shortest distance between the center point of the mask frame and the preset shape area;

[0029] Determine the length of the diagonal of the mask frame;

[0030] The mask wearing standard index value is determined according to a preset rule, wherein the preset rule is a cosine function constructed by the shortest distance and the length of the diagonal.

[0031] Optionally, after determining the mask wearing result of the face image to be detected according to the face key point positions, mask position and category of the face image to be detected, the method further includes:

[0032] In the case where the mask wearing result indicates that the mask is not worn properly,

[0033] If the mask wearing standard index value is a second preset value, a corresponding reminder message is generated according to the key point that is not within the mask frame;

[0034] If the mask wearing standard index value is greater than the second preset value and less than the preset threshold, a reminder message instructing to wear the mask centered is generated.

[0035] A third aspect of an embodiment of the present invention provides a training device for a face mask wearing detection model, comprising:

[0036] A first acquisition module acquires a plurality of face images and annotation data of each face image, wherein the plurality of face images include face images of people wearing masks and face images of people not wearing masks, and the annotation data include positions of key face points, positions of masks, and categories in the face images, wherein the categories indicate whether a mask is worn or not worn;

[0037] A first determination module determines the image features of each of the facial images;

[0038] The training module is used to perform training based on the image features and annotation data corresponding to each of the multiple facial images to obtain a facial mask wearing detection model.

[0039] A fourth aspect of an embodiment of the present invention provides a device for detecting when a face mask is worn, comprising:

[0040] The second acquisition module acquires the face image to be detected;

[0041] The detection module inputs the face image to be detected into the face mask wearing detection model to obtain the face key point position, mask position and category of the face image to be detected;

[0042] The second determination module determines the mask wearing result of the face image to be detected according to the facial key point positions, mask position and category of the face image to be detected.

[0043] According to another aspect of an embodiment of the present invention, there is provided an electronic device, including:

[0044] one or more processors;

[0045] a storage device for storing one or more programs,

[0046] When the one or more programs are executed by the one or more processors, the one or more processors implement the training method of the face mask wearing detection model or the face mask wearing detection method provided by the present invention.

[0047] According to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the training method of the face mask wearing detection model or the face mask wearing detection method provided by the present invention is implemented.

[0048] An embodiment of the above invention has the following advantages or beneficial effects: obtaining multiple face images and annotation data of each face image, the multiple face images include face images wearing masks and face images not wearing masks, the annotation data include the position of key face points, the position of masks and the category in the face images, and the category indicates wearing masks or not wearing masks; determining the image features of each face image; training based on the image features and annotation data corresponding to each face image in the multiple face images to obtain a face mask wearing detection model. This implementation can detect the position of key face points, the position of masks and the category, so as to determine the degree of mask wearing standards based on the position of key face points and the position of masks, so as to realize automatic detection of mask wearing standards based on face images and prompt how to wear masks in a standardized manner.

[0049] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0051] Figure 1 It is a schematic diagram of the main process of a training method for a face mask wearing detection model according to an embodiment of the present invention;

[0052] Figure 2 It is a schematic diagram of the main process of a method for detecting the wearing of a face mask according to an embodiment of the present invention;

[0053] Figure 3 is a schematic diagram of the main process of another method for detecting face mask wearing according to an embodiment of the present invention;

[0054] Figure 4 It is a flow chart of a method for detecting the wearing of a face mask according to an embodiment of the present invention;

[0055] Figure 5 is a schematic diagram of main modules of a training device for a face mask wearing detection model according to an embodiment of the present invention;

[0056] Figure 6 is a schematic diagram of main modules of a device for detecting face mask wearing according to an embodiment of the present invention;

[0057] Figure 7 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0058] Figure 8 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0060] It should be noted that in the technical solution of this disclosure, the collection, collection, update, analysis, processing, use, transmission, storage and other aspects of user personal information (such as facial images, etc.) involved are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0061] Figure 1is a schematic diagram of the main process of a training method for a face mask wearing detection model according to an embodiment of the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0062] Step S101: obtaining a plurality of face images and annotation data of each face image, wherein the plurality of face images include face images of people wearing masks and face images of people not wearing masks, and the annotation data include positions of key face points, positions of masks and categories in the face images, and the categories indicate whether a mask is worn or not worn;

[0063] Step S102: determining the image features of each face image;

[0064] Step S103: training is performed based on the image features and annotation data corresponding to each face image in the multiple face images to obtain a face mask wearing detection model.

[0065] In an embodiment of the present invention, a plurality of face images are first obtained, the plurality of face images include face images of people wearing masks and face images of people not wearing masks, and the face images of people wearing masks include face images of people wearing masks in a standardized manner and face images of people wearing masks in an irregular manner. Then, for each face image, the face key point positions, mask positions and categories in each face image are marked to obtain the annotation data of each face image. Among them, the category of the face image indicates whether a mask is worn or not; the face key point positions may include the coordinates of the left eye, the right eye, the tip of the nose, the left corner of the mouth and the right corner of the mouth, and may also include the coordinates of the left nose wing and the right nose wing, that is, the face key point positions include key points indicating a face, and also include face key points covered by the mask when the mask is worn in a standardized manner, so as to evaluate whether the mask is worn in a standardized manner based on these face key points. The mask position is the position of the mask frame in the face image, and the mask frame can be represented by a rectangle, and the mask position can be the coordinates of the four vertices of the rectangle.

[0066] In an embodiment of the present invention, after obtaining multiple face images and the annotation data of each face image, the multiple face images are processed to obtain the image features of each face image, so as to use the image features for model training. Specifically, determining the image features of each face image includes:

[0067] Use residual network to obtain multiple levels of feature information of face images;

[0068] The feature pyramid network is used to fuse feature information at multiple levels to obtain image features.

[0069] In an embodiment of the present invention, the face mask wearing detection model is a deep learning model, and the residual network can be ResNet50 (a residual neural network with a depth of 50 layers). ResNet50 is used as a backbone network to extract feature information in the face image, and the feature information is feature information of multiple levels, that is, feature information of different levels of depth, including deep features with rich semantic information and shallow features with clear location information. Then, the feature information of multiple levels obtained is fused by a feature pyramid network (Feature Pyramid Network, FPN) to obtain image features, so that the image features have deep features with rich semantic information and shallow features with clear location information at the same time. Through the fusion of feature information of different levels of depth, the image features are richer, so that the detection accuracy of the face mask wearing detection model obtained by training is higher, and the precision is improved. In addition, the feature information of multiple layers and different depths has different sizes, which also enhances the robustness of the model.

[0070] In an embodiment of the present invention, after determining the image features of each face image, training is performed according to the image features and annotation data corresponding to each face image in a plurality of face images to obtain a face mask wearing detection model, which may specifically include:

[0071] Use a fully convolutional neural network to classify image features and obtain the predicted categories, mask positions, and facial key point positions corresponding to the image features;

[0072] Construct a loss function that includes mask classification loss, mask frame regression loss, and face key point loss;

[0073] Based on the predicted category, mask position, facial key point position and annotation data corresponding to each image feature, the loss function is iteratively trained to obtain a face mask wearing detection model.

[0074] In an embodiment of the present invention, a fully convolutional neural network (FCN) is used to classify the image features of each face image, obtain the predicted category, mask position and face key point position corresponding to the face image or the image feature, construct a multi-task loss function including mask classification loss, mask frame regression loss and face key point loss, and iteratively train the loss function using the predicted category, mask position and face key point position and labeled data, and continuously optimize to obtain a face mask wearing detection model.

[0075] In the embodiment of the present invention, the mask classification loss is the loss between the predicted category and the true category, the mask frame regression loss is the loss between the predicted mask frame size and the upper left corner vertex coordinates and the true mask frame size and the upper left corner coordinates, and the face key point position loss is the loss between the predicted face key point coordinates and the true face key point coordinates. The mask classification loss is represented by the cross entropy loss function, the face key point position loss and the mask frame regression loss are represented by the smoothL1 function, and the expression of the multi-task loss function L including the mask classification loss, the mask frame regression loss and the face key point position loss is shown in formula (2),

[0076]

[0077] In formula (2), represents the mask classification loss, where p i Indicates the probability that the prediction box i is a mask, Represents the true value of the predicted box. represents the mask frame regression loss, where t i =(t x ,t y ,t w ,t h ) represents the position coordinates and size of the prediction box corresponding to the mask box, (t x ,t y ) represents the vertex coordinates, t w ,t h Respectively represent the length and width of the mask frame, Indicates the position coordinates and size of the real mask frame in the labeled data. Represents the position loss of facial key points, where the parameter l i =(l x1 ,l y1 ,l x2 ,l y2 ,……l x5 ,l y5 ) represents the predicted coordinate values ​​of the five facial key points (left eye, right eye, nose tip, left corner of mouth, right corner of mouth) on the x-axis and y-axis respectively. Indicates the real coordinate values ​​of the five facial key points on the x-axis and y-axis.

[0078] In an embodiment of the present invention, the face mask wearing detection model is composed of a network including a residual network, FPN and FCN, wherein other neural networks may also be used to obtain predicted categories, mask positions and facial key point positions corresponding to image features.

[0079] like Figure 2 As shown, the embodiment of the present invention also provides a method for detecting when a face mask is worn, comprising:

[0080] Step S201: obtaining a face image to be detected;

[0081] Step S202: inputting the face image to be detected into the face mask wearing detection model to obtain the face key point positions, mask positions and categories of the face image to be detected;

[0082] Step S203: Determine the mask wearing result of the face image to be detected according to the facial key point positions, mask position and category of the face image to be detected.

[0083] In an embodiment of the present invention, when performing face mask wearing detection, first obtain a face image to be detected, which can be a face image taken in real time or a face image collected in a video. Input the face image to be detected into the face mask wearing detection model, and obtain the face key point position, mask position and category. The category indicates whether the face to be detected is wearing a mask. The face key point position can include the coordinates of the left eye, right eye, nose tip, left corner of the mouth and right corner of the mouth. The mask position includes the size of the mask frame and the coordinates of any vertex of the mask frame or the coordinates of the center point of the mask frame. The mask frame size includes the length and width of the mask frame. Thus, the mask wearing result can be determined according to the obtained face key point position, mask position and category, wherein the mask wearing result can include whether the mask is worn, whether the mask is worn in a standard manner, etc.

[0084] In an embodiment of the present invention, determining the mask wearing result of the face image to be detected according to the face key point positions, mask position and category of the face image to be detected includes:

[0085] When the category indicates wearing a mask, the mask wearing standard index value is determined according to the position of the facial key points and the mask position. When the mask wearing standard index value is greater than the preset threshold, the mask wearing result is judged to be an indication of wearing the mask standardly. When the mask wearing standard index value is less than or equal to the preset threshold, the mask wearing result is judged to be an indication of not wearing the mask standardly.

[0086] In an embodiment of the present invention, the mask wearing standard index value is used to characterize the standard degree of mask wearing, and the range of the mask wearing standard index value can be customized, for example, it can be set to a range of 0 to 1. If the category indicates that the face image to be detected is a case of wearing a mask, it is further determined whether the mask is worn in a standardized manner. The mask wearing standard index value can be determined by the position of the key points of the face and the position of the mask. If the mask wearing standard index value is greater than a preset threshold, it is determined to be standard wearing of the mask. If the mask wearing standard index value is less than or equal to the preset threshold, it is determined to be non-standard wearing of the mask, wherein the preset threshold can be customized according to the actual situation. For example, the preset threshold can be set to 0.5. If the mask wearing standard index value is greater than 0.5, it means that the mask is worn in a standardized manner. If the mask wearing standard index value is less than or equal to 0.5, it means that the mask is worn in an irregular manner, and correction suggestions need to be provided according to the irregular position.

[0087] In an embodiment of the present invention, if the category indicates that a mask is not worn, the category indication result is used as the mask wearing result, that is, if a mask is not worn, a reminder message such as "Please wear a mask" can be issued based on the mask wearing result; or the mask wearing result can be determined as not wearing the mask properly, and a reminder message such as "Please wear a mask properly" can be issued.

[0088] In the embodiment of the present invention, Figure 3 As shown, the facial key point positions include the positions of multiple key points on the face, and the mask position includes the size and center point of the mask frame; the mask wearing standard index values ​​are determined based on the facial key point positions and the mask position, including:

[0089] Step S301: determining a preset shape area composed of a plurality of key points;

[0090] Step S302: Determine whether the multiple key points are all within the mask frame; if not, execute step S303; if yes, execute step S304;

[0091] Step S303: Determine that the mask wearing standard index value is a second preset value,

[0092] Step S304: Determine whether the center point of the mask frame is within the preset shape area. If so, execute step S305; if not, execute step S306;

[0093] Step S305: Determine that the mask wearing standard index value is a first preset value;

[0094] Step S306: Determine the mask wearing standard index value according to the distance between the center point of the mask frame and the preset shape area;

[0095] Among them, the first preset value is greater than the preset threshold, and the second preset value is less than the preset threshold.

[0096] In an embodiment of the present invention, the key points of the face include the positions of multiple key points in the face, and the multiple key points can be multiple key points of the face covered by the mask when the mask is worn in a standardized manner, such as the multiple key points can include the tip of the nose, the left corner of the mouth and the right corner of the mouth, and can also include the left nose wing, the right nose wing, the left cheek, the right cheek, etc. The mask position includes the size and center point of the mask frame, and the mask frame size includes the length and width of the mask frame. Determine a preset shape area composed of multiple key points, and the preset shape area can be customized, such as a triangular area, an elliptical area, etc.

[0097] After determining the preset shape area composed of multiple key points, it is determined whether the multiple key points are all within the mask frame. If the multiple key points are all within the mask frame, it means that the multiple key points are all covered by the mask, and then it is further determined whether the center point of the mask frame is within the preset shape area. If the center point of the mask frame is within the preset shape area, the mask wearing standard index value is determined to be a first preset value, and the first preset value is greater than the preset threshold, which means that the mask wearing result is a standard mask wearing result; if the center point of the mask frame is not within the preset shape area, it is necessary to further determine the mask wearing standard index value based on the distance between the center point of the mask frame and the preset shape area. If at least one of the multiple key points is not within the mask frame, such as at least one of the nose tip, the left corner of the mouth, or the right corner of the mouth is not within the mask frame, the mask wearing standard index value is determined to be a second preset value, and the second preset value is less than the preset threshold, which means that the mask wearing result is not a standard mask wearing result. Among them, the preset threshold can be set to 0.5, the first preset value can be 1, and the second preset value can be 0, that is, the mask wearing standard index value is within the range between the first preset value and the second preset value.

[0098] In an embodiment of the present invention, determining a mask wearing standard index value according to the distance between the center point of the mask frame and the preset shape area includes:

[0099] Determine the shortest distance between the center point of the mask frame and the preset shape area;

[0100] Determine the length of the diagonal of the mask frame;

[0101] The mask wearing standard index value is determined according to a preset rule, wherein the preset rule is a cosine function constructed by the shortest distance and the length of the diagonal.

[0102] In an embodiment of the present invention, when multiple key points are within the mask frame and the center point of the mask frame is not within the preset shape area, the shortest distance between the center point of the mask frame and the preset shape area is calculated, and the length of the diagonal of the mask frame is calculated, wherein the shortest distance is greater than 0 and less than half of the length of the diagonal; then the mask wearing standard index value is determined according to the cosine function constructed by the shortest distance and the length of the diagonal. Wherein, the cosine function is shown in formula (1),

[0103]

[0104] Among them, l AC is the length of the mask frame diagonal, l min It is the shortest distance between the center point of the mask frame and the preset shape area.

[0105] In the embodiment of the present invention, the greater the shortest distance, the smaller the mask wearing standard index value, the shortest distance 0 increases to half the length of the diagonal, the mask wearing standard index value decreases from 1 to 0, and the shortest distance and the mask wearing standard index value form a convex curve relationship. Therefore, the preset rule can also be a normal distribution function of the shortest distance and the mask wearing standard index value process.

[0106] In an embodiment of the present invention, after determining the mask wearing result of the face image to be detected according to the face key point positions, mask position and category of the face image to be detected, the method further includes:

[0107] In the case where the mask wearing result indicates that the mask is not worn properly,

[0108] If the mask wearing standard index value is a second preset value, a corresponding reminder message is generated according to the key points that are not within the mask frame;

[0109] If the mask wearing standard index value is greater than the second preset value and less than the preset threshold, a reminder message instructing to wear the mask centered is generated.

[0110] In an embodiment of the present invention, if the mask wearing result indicates that the mask is not worn in a standardized manner, that is, the mask wearing standard index value is less than a preset threshold, the position where the mask is not worn in a standardized manner is further detected to make correction suggestions based on the position where the mask is not worn in a standardized manner. If the mask wearing standard index value is a second preset value, that is, at least one key point is not in the mask frame, the key points that are not in the mask frame are determined, and corresponding reminder information is generated based on the key points that are not in the mask frame, that is, the reminder information prompts the key points or corresponding face positions that are not covered by the mask, such as the nose tip is not in the mask frame, then a reminder message "Please wear the mask upward to cover the nose" can be generated, such as the left corner of the mouth and / or the right corner of the mouth is not in the mask frame, then a reminder message "Please wear the mask downward to cover the chin" can be generated. If the mask wearing standard index value is less than the preset threshold, but is not the second preset threshold, it means that multiple key points are in the mask frame, but the center point of the mask frame is not in the preset shape area, that is, the mask covers all key points but the mask position is not centered, then a reminder message "Please wear the mask reasonably centered to cover the mouth and nose" can be generated.

[0111] Figure 4 A schematic flow chart of a method for detecting the wearing of a face mask according to an embodiment of the present invention is shown. The method comprises:

[0112] Step S401: obtaining a face image to be detected;

[0113] Step S402: inputting the face image to be detected into the face mask wearing detection model to obtain the face key point position, mask position and category; wherein the face key point position includes the coordinates of the left eye, right eye, nose tip, left corner of the mouth and right corner of the mouth, and the mask position includes the mask frame size and center point;

[0114] Step S403: If the category indicates that a mask is not worn, a reminder message "Please wear a mask" is generated;

[0115] Step S404: if the category indicates wearing a mask, determining a preset shape area consisting of a plurality of key points, wherein the plurality of key points include a nose tip, a left corner of the mouth, and a right corner of the mouth, and the preset shape area is a triangular area;

[0116] Step S405: Determine whether the multiple key points are all within the mask frame; if not, execute step S406; if yes, execute step S409;

[0117] Step S406: Determine that the mask wearing standard index value is 0;

[0118] Step S407: determining that the mask wearing result is not properly worn, and determining key points that are not within the mask frame;

[0119] Step S408: Generate corresponding reminder information according to the key points that are not within the mask frame;

[0120] Step S409: Determine whether the center point of the mask frame is within the preset shape area; if so, execute step S410; if not, execute step S411;

[0121] Step S410: determining that the mask wearing standard index value is 1, and determining that the mask wearing result is a mask wearing standard;

[0122] Step S411: determining the shortest distance between the center point of the mask frame and the preset shape area, and determining the length of the diagonal line of the mask frame;

[0123] Step S412: determining the mask wearing standard index value according to a preset rule; wherein the preset rule is a cosine function constructed by the shortest distance and the length of the diagonal, as shown in formula (1);

[0124] Step S413: Determine whether the mask wearing standard index value is greater than 0.5, if so, execute step S414, if not, execute step S415;

[0125] Step S414: determining that the mask wearing result is a result indicating a mask wearing standard;

[0126] Step S415: Determine if the mask wearing result indicates that the mask is not worn properly, and generate a reminder message "Please wear the mask properly and center it to cover your mouth and nose."

[0127] The training method of the face mask wearing detection model provided by the embodiment of the present invention extracts multi-level features of the face image through the residual network, fuses the features using the feature pyramid network to enhance the features, constructs the loss function using the convolutional neural network, and uses the image features and annotation data of each face image in multiple face images to train the face mask wearing detection model, so that the face key points and mask positions and categories in the face image can be detected by using the model. The face mask wearing detection method of the embodiment of the present invention uses the face mask wearing detection model to detect the face key point position and mask position and category in the face image to be detected, and then determines whether the mask is worn according to the category, determines the standard degree of mask wearing according to the face key point position and mask position, and determines the position of non-standard wearing for the case of non-standard wearing of the mask, so as to send a reminder message according to the position of non-standard wearing, so as to provide correction suggestions for mask wearing. The method can judge the standard degree of mask wearing according to the face key point position and mask position, and provide correction suggestions, so as to realize automatic monitoring and prompting of the mask wearing situation of the crowd in public places, so as to remind people to wear masks in a standardized manner more efficiently and intelligently, thereby reducing the risk of virus transmission.

[0128] In a third aspect of the embodiment of the present invention, Figure 5As shown, a training device 500 for a face mask wearing detection model is provided, comprising:

[0129] A first acquisition module 501 acquires a plurality of face images and annotation data of each face image, wherein the plurality of face images include face images of people wearing masks and face images of people not wearing masks, and the annotation data include positions of key face points, positions of masks, and categories in the face images, and the categories indicate whether a mask is worn or not worn;

[0130] A first determination module 502 determines the image features of each face image;

[0131] The training module 503 performs training based on the image features and annotation data corresponding to each face image in multiple face images to obtain a face mask wearing detection model.

[0132] In the embodiment of the present invention, the first determination module 502 is further used to: use a residual network to obtain multiple levels of feature information of the face image; use a feature pyramid network to perform feature fusion on the multiple levels of feature information to obtain image features.

[0133] In an embodiment of the present invention, the training module 503 is further used to: classify image features using a fully convolutional neural network to obtain predicted categories, mask positions, and facial key point positions corresponding to the image features; construct a loss function including mask classification loss, mask frame regression loss, and facial key point loss; based on the predicted categories, mask positions, and facial key point positions corresponding to each image feature and the labeled data, iteratively train the loss function to obtain a face mask wearing detection model.

[0134] In a fourth aspect of the embodiments of the present invention, Figure 6 As shown, a device 600 for detecting the wearing of a face mask is provided, comprising:

[0135] The second acquisition module 601 acquires a face image to be detected;

[0136] The detection module 602 inputs the face image to be detected into the face mask wearing detection model to obtain the face key point position, mask position and category of the face image to be detected;

[0137] The second determination module 603 determines the mask wearing result of the face image to be detected according to the facial key point positions, mask position and category of the face image to be detected.

[0138] In an embodiment of the present invention, the second determination module 603 is further used to: when the category indicates wearing a mask, determine the mask wearing standard index value according to the position of the key points of the face and the position of the mask; when the mask wearing standard index value is greater than a preset threshold, determine the mask wearing result as a result indicating that the mask is worn standardly; when the mask wearing standard index value is less than or equal to the preset threshold, determine the mask wearing result as a result indicating that the mask is not worn standardly.

[0139] In the embodiment of the present invention, the facial key point positions include the positions of multiple key points in the face, and the mask position includes the size and center point of the mask frame; the second determination module 603 is further used to:

[0140] Determine a preset shape area formed by a plurality of key points;

[0141] Determine whether multiple key points are within the mask frame;

[0142] When multiple key points are within the mask frame, determine whether the center point of the mask frame is within the preset shape area. If so, determine the mask wearing standard index value as a first preset value; if not, determine the mask wearing standard index value according to the distance between the center point of the mask frame and the preset shape area;

[0143] When at least one of the multiple key points is not within the mask frame, determining the mask wearing standard index value to be a second preset value,

[0144] Among them, the first preset value is greater than the preset threshold, and the second preset value is less than the preset threshold.

[0145] In an embodiment of the present invention, the second determination module 603 is further used to: determine the shortest distance between the center point of the mask frame and the preset shape area; determine the length of the diagonal of the mask frame; determine the mask wearing standard index value according to preset rules, wherein the preset rule is a cosine function constructed by the shortest distance and the length of the diagonal.

[0146] In the embodiment of the present invention, the second determination module 603 is further used to: after determining the mask wearing result of the face image to be detected according to the face key point positions, mask positions and categories of the face image to be detected,

[0147] In the case where the mask wearing result indicates that the mask is not worn properly,

[0148] If the mask wearing standard index value is a second preset value, a corresponding reminder message is generated according to the key points that are not within the mask frame;

[0149] If the mask wearing standard index value is greater than the second preset value and less than the preset threshold, a reminder message instructing to wear the mask centered is generated.

[0150] According to another aspect of an embodiment of the present invention, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the training method of the face mask wearing detection model or the face mask wearing detection method provided by the present invention.

[0151] According to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the training method of the face mask wearing detection model or the face mask wearing detection method provided by the present invention is implemented.

[0152] Figure 7 An exemplary system architecture 700 of a training method for a face mask wearing detection model or a training device for a face mask wearing detection model, a face mask wearing detection method or a face mask wearing detection method to which an embodiment of the present invention may be applied is shown.

[0153] like Figure 7 As shown, system architecture 700 may include terminal devices 701, 702, 703, network 704 and server 705. Network 704 is used to provide a medium for communication links between terminal devices 701, 702, 703 and server 705. Network 704 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0154] Users can use terminal devices 701, 702, and 703 to interact with server 705 through network 704 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 701, 702, and 703, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0155] The terminal devices 701 , 702 , and 703 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0156] The server 705 may be a server that provides various services, such as a backend management server (only for example) that supports shopping websites browsed by users using the terminal devices 701, 702, and 703. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - only for example) to the terminal device.

[0157] It should be noted that the training method of the face mask wearing detection model or the face mask wearing detection method provided in the embodiments of the present invention are generally executed by the server 705. Accordingly, the training device of the face mask wearing detection model or the face mask wearing detection device are generally arranged in the server 705.

[0158] It should be understood that Figure 7 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0159] Reference below Figure 8 , which shows a schematic diagram of the structure of a computer system 800 of a terminal device suitable for implementing an embodiment of the present invention. Figure 8 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0160] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the system 800 are also stored. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0161] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage section 808 as needed.

[0162] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the above-mentioned functions defined in the system of the present invention are executed.

[0163] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0164] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0165] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes a second acquisition module, a detection module, and a second determination module. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the second acquisition module may also be described as a "module for acquiring a face image to be detected".

[0166] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: obtaining multiple face images and annotation data of each face image, the multiple face images include face images wearing masks and face images not wearing masks, the annotation data include the position of key face points, the position of masks and the category in the face images, and the category indicates wearing a mask or not wearing a mask; determining the image features of each face image; training according to the image features and annotation data corresponding to each face image in the multiple face images to obtain a face mask wearing detection model.

[0167] According to the technical solution of the embodiment of the present invention, the training method of the face mask wearing detection model extracts multi-level features of the face image through the residual network, fuses the features using the feature pyramid network to enhance the features, constructs the loss function using the convolutional neural network, and trains the face mask wearing detection model using the image features and annotation data of each face image in multiple face images, so that the face key point position, mask position and category in the face image can be detected using the model. The face mask wearing detection method of the embodiment of the present invention uses the face mask wearing detection model to detect the face key point position, mask position and category in the face image to be detected, and then determines whether the mask is worn according to the category, determines the standard degree of mask wearing according to the face key point position and the mask position, and determines the position of the non-standard wearing of the mask in the case of non-standard wearing of the mask, so as to send a reminder message according to the non-standard wearing position, so as to provide correction suggestions for mask wearing. This method can judge the degree of mask wearing standards based on the position of key points on the face and the position of the mask, and provide correction suggestions, so as to realize automatic monitoring and prompting of mask wearing conditions of people in public places, so as to remind people to wear masks correctly in a more efficient and intelligent way, thereby reducing the risk of virus transmission.

[0168] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A training method for a face mask wearing detection model, characterized in that: include: Acquire multiple facial images and annotation data for each facial image, wherein the multiple facial images include facial images of people wearing masks and facial images of people not wearing masks, and the annotation data include positions of key facial points, positions of masks, and categories in the facial images, wherein the categories indicate whether a mask is worn or not worn; Determining image features of each of the facial images; Training is performed based on the image features and annotation data corresponding to each of the multiple facial images to obtain a facial mask wearing detection model.

2. The training method according to claim 1, characterized in that: Determining the image features of each of the facial images, including: Using a residual network to obtain feature information of multiple levels of the face image; The feature information of the multiple levels is fused using a feature pyramid network to obtain the image features.

3. The training method according to claim 1, characterized in that: Training is performed based on the image features and annotation data corresponding to each of the multiple face images to obtain a face mask wearing detection model, including: Using a fully convolutional neural network to classify the image features, and obtain predicted categories, mask positions, and facial key point positions corresponding to the image features; Construct a loss function that includes mask classification loss, mask frame regression loss, and face key point loss; Based on the predicted category, mask position, facial key point position and annotation data corresponding to each image feature, the loss function is iteratively trained to obtain the facial mask wearing detection model.

4. A method for detecting when a person is wearing a mask on their face, characterized in that: include: Get the face image to be detected; Input the face image to be detected into the face mask wearing detection model to obtain the face key point position, mask position and category of the face image to be detected; The mask wearing result of the face image to be detected is determined according to the facial key point positions, mask position and category of the face image to be detected.

5. The method according to claim 4, characterized in that Determining the mask wearing result of the face image to be detected according to the face key point positions, mask position and category of the face image to be detected, including: When the category indicates wearing a mask, the mask wearing standard index value is determined according to the facial key point positions and the mask position; when the mask wearing standard index value is greater than a preset threshold, the mask wearing result is determined to be a result indicating that the mask is worn standardly; when the mask wearing standard index value is less than or equal to the preset threshold, the mask wearing result is determined to be a result indicating that the mask is not worn standardly.

6. The method according to claim 5, characterized in that The facial key point positions include the positions of multiple key points in the face, and the mask position includes the size and center point of the mask frame; Determining a mask wearing standard index value according to the facial key point positions and the mask position includes: Determine a preset shape area formed by the plurality of key points; Determine whether the multiple key points are all within the mask frame; When the plurality of key points are all within the mask frame, determine whether the center point of the mask frame is within the preset shape area, and if so, determine the mask wearing standard index value to be a first preset value; if not, determine the mask wearing standard index value according to the distance between the center point of the mask frame and the preset shape area; When at least one of the plurality of key points is not within the mask frame, determining the mask wearing standard index value to be a second preset value, The first preset value is greater than the preset threshold, and the second preset value is less than the preset threshold.

7. The method according to claim 6, characterized in that Determining the mask wearing standard index value according to the distance between the center point of the mask frame and the preset shape area includes: Determine the shortest distance between the center point of the mask frame and the preset shape area; Determine the length of the diagonal of the mask frame; The mask wearing standard index value is determined according to a preset rule, wherein the preset rule is a cosine function constructed by the shortest distance and the length of the diagonal.

8. The method according to claim 5, characterized in that After determining the mask wearing result of the face image to be detected according to the face key point positions, mask position and category of the face image to be detected, the method further includes: In the case where the mask wearing result indicates that the mask is not worn properly, If the mask wearing standard index value is a second preset value, a corresponding reminder message is generated according to the key point that is not within the mask frame; If the mask wearing standard index value is greater than the second preset value and less than the preset threshold, a reminder message instructing to wear the mask centered is generated.

9. A training device for a face mask wearing detection model, characterized in that: include: A first acquisition module acquires a plurality of face images and annotation data of each face image, wherein the plurality of face images include face images of people wearing masks and face images of people not wearing masks, and the annotation data include positions of key face points, positions of masks, and categories in the face images, wherein the categories indicate whether a mask is worn or not worn; A first determination module determines the image features of each of the facial images; The training module is used to perform training based on the image features and annotation data corresponding to each of the multiple facial images to obtain a facial mask wearing detection model.

10. A device for detecting the wearing of a mask on a face, characterized in that: include: The second acquisition module acquires the face image to be detected; The detection module inputs the face image to be detected into the face mask wearing detection model to obtain the face key point position, mask position and category of the face image to be detected; The second determination module determines the mask wearing result of the face image to be detected according to the facial key point positions, mask position and category of the face image to be detected.

11. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

12. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images