Mouth occlusion judgment method, device, computer-readable storage medium and equipment

Through convolutional neural network, the mouth area image is classified in binary and combined with the information processing of face key points, the problem of poor generalization and misjudgment of mouth occlusion in the existing technology is solved, and the mouth occlusion with high accuracy and fast speed is achieved judge.

CN114255490BActive Publication Date: 2024-11-29BEIJING TECHSHINO TECHNOLOGY CO LTD +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202010952464.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-09-11
Publication Date
2024-11-29
Estimated Expiration
2040-09-11

AI Technical Summary

Technical Problem

The prior art has poor general use in the judgment of mouth occlusion and is easily affected by mouth opening images, resulting in low accuracy, especially the problem of misjudgment of mouth occlusion is difficult to solve.

Method used

Convolutional neural network is used to classify the mouth area images in binary, calculate the mouth opening degree through the face key points and divide the mouth opening and non-mouth images. Combining the upper and lower lip distances and the left and right eye spacing to calculate the mouth opening degree, rotation and normalization After the mouth area image is imaged, convolution operation and full connection layer classification are performed to determine whether the mouth is blocked.

Benefits of technology

It improves the accuracy and robustness of mouth occlusion judgments, reduces the computational volume, achieves fast and efficient classification, and can control the depth and width of the convolutional neural network, ensuring high accuracy and fast speed.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114255490B_ABST
    Figure CN114255490B_ABST
Patent Text Reader

Abstract

The present invention discloses a mouth occlusion judgment method, device, computer-readable storage medium and equipment, which belong to the field of biometrics. It includes: obtaining a face image and locating the key points of the face; calculating the degree of mouth opening according to the key points of the face, and dividing the face image into an open-mouth image or a non-open-mouth image according to the degree of mouth opening; for a non-open-mouth image, obtaining a mouth area image from the non-open-mouth image according to the mouth key points in the face key points; inputting the mouth area image into a pre-trained convolutional neural network to determine whether the mouth is occluded. The present invention solves the problem of misjudgment of mouth occlusion in an open-mouth image, and has high accuracy, good generalization, strong robustness and fast speed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of biometrics, and in particular to a method, device, computer-readable storage medium and equipment for judging mouth occlusion. Background Art

[0002] The rise of deep learning has brought great development to biometric recognition, especially in the research of face recognition and image understanding. The biological characteristics of organisms can not only distinguish biological individuals, but also judge the status of biological individuals. When recognizing faces, we must first obtain clear and complete facial images in sufficient light, and then process the facial images. In the face recognition task, incomplete or occluded facial images will lead to reduced accuracy, especially the occlusion of facial features such as the mouth, which has a greater impact on the performance of the algorithm. In addition, the occlusion of the mouth has a great impact on the accuracy of liveness detection based on feature point movements. Therefore, the judgment of mouth occlusion has important research value.

[0003] The current method for detecting mouth occlusion is to obtain a face image and extract its feature vector (such as HOG feature vector, etc.), and then input the feature vector into a pre-trained support vector machine or other classifier for classification, thereby determining whether the mouth is occluded. However, the feature vector is an artificially designed feature, which cannot well represent whether the mouth is occluded or not, and the artificially designed feature cannot adapt to various mouth occlusion situations, and its versatility is poor.

[0004] In addition, in the judgment of mouth occlusion, there is another difficulty in judging the open mouth image. Since the mouth is open, the presence or absence of teeth in the image, the presence or absence of tongue coating, etc. will affect the mouth occlusion judgment method of the existing technology and reduce the accuracy of the judgment. Summary of the invention

[0005] In order to solve the technical problems that the existing mouth occlusion judgment method has poor versatility and is easily affected by open mouth images, the present invention provides a mouth occlusion judgment method, device, computer-readable storage medium and equipment, which solve the problem of misjudgment of mouth occlusion by open mouth images, and have high accuracy, good generalization, strong robustness and fast speed.

[0006] The present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a method for determining mouth occlusion, the method comprising:

[0008] Obtain a face image and locate key points of the face;

[0009] Calculating the mouth opening degree according to the facial key points, and dividing the facial image into an open-mouth image or a non-open-mouth image according to the mouth opening degree;

[0010] For a non-open mouth image, a mouth area image is obtained from the non-open mouth image according to the mouth key points in the facial key points;

[0011] The mouth area image is input into a pre-trained convolutional neural network to determine whether the mouth is occluded.

[0012] Further, the calculating the mouth opening degree according to the facial key points and dividing the facial image into an open-mouth image or a non-open-mouth image according to the mouth opening degree includes:

[0013] Calculate the vertical distance between the upper lip key point and the lower lip key point in the facial key points to obtain the upper and lower lip distance h;

[0014] The left-right eye distance w is calculated based on the left-eye key point and the right-eye key point in the face key points;

[0015] The mouth opening degree h / w is calculated according to the distance h between the upper and lower lips and the distance w between the left and right eyes;

[0016] If the mouth opening degree h / w reaches the set mouth opening threshold, the face image is an open-mouth image, otherwise the face image is a non-open-mouth image.

[0017] Furthermore, the method of acquiring the mouth region image from the non-mouth-opening image according to the mouth key points in the face key points includes:

[0018] Calculating the tilt angle of the mouth according to the mouth key points in the facial key points, and rotating and aligning the facial image according to the tilt angle;

[0019] The mouth region is cut out from the aligned face image according to the mouth key points among the face key points and normalized to obtain a mouth region image.

[0020] Furthermore, the inputting of the mouth region image into a pre-trained convolutional neural network to determine whether the mouth is blocked includes:

[0021] Performing a first convolution operation, a first BN operation, a first ReLU operation, and a first pooling operation on the mouth area image to obtain a first feature map;

[0022] Expanding the edge of the first feature map by a number of pixels, and performing a second convolution operation, a second BN operation, a second ReLU operation, and a second pooling operation to obtain a second feature map;

[0023] Expanding the edge of the second feature map by a number of pixels, and performing a third convolution operation, a third BN operation, a third ReLU operation, and a third pooling operation to obtain a third feature map;

[0024] Performing a first full connection operation, a fourth ReLU operation, a first dropout operation, a second full connection operation, a fifth ReLU operation, and a second dropout operation on the third feature map to obtain a fourth feature map;

[0025] Performing a third full connection operation with an output dimension of 2 on the fourth feature map, and using Softmax to perform classification, to obtain the probability that the mouth area image is a mouth and the probability that it is not a mouth;

[0026] If the probability that the mouth area image is the mouth exceeds the set occlusion threshold, the mouth is judged to be unoccluded, otherwise the mouth is judged to be occluded.

[0027] Furthermore, the convolutional neural network is trained by the following method:

[0028] Constructing a training set, wherein the training set includes positive samples and negative samples, wherein the positive samples are unobstructed mouth images, and the negative samples are mouth images obstructed by non-mouth images;

[0029] Performing data expansion on the training set, wherein the data expansion includes translating, rotating, modifying contrast, and / or adding noise to positive samples and negative samples;

[0030] The convolutional neural network is trained using the data augmented training set.

[0031] Furthermore, the non-mouth image includes a non-skin color image and a skin color image outside the mouth area; the positive sample and the negative sample both include a number of slightly open mouth images, and the mouth opening degree of the slightly open mouth images does not reach the mouth opening degree of the open mouth image.

[0032] In a second aspect, the present invention provides a mouth occlusion determination device, the device comprising:

[0033] An image acquisition module is used to acquire a face image and locate key points of the face;

[0034] A mouth opening detection module, used for calculating the mouth opening degree according to the key points of the human face, and classifying the human face image into an open-mouth image or a non-open-mouth image according to the mouth opening degree;

[0035] An image extraction module, for obtaining a mouth area image from a non-open mouth image according to a mouth key point in a human face key point;

[0036] The second judgment module is used to input the mouth area image into a pre-trained convolutional neural network to determine whether the mouth is blocked.

[0037] Furthermore, the mouth opening detection module includes:

[0038] The upper and lower lip distance calculation unit is used to calculate the vertical distance between the upper lip key point and the lower lip key point in the face key points to obtain the upper and lower lip distance h;

[0039] A left-right eye distance calculation unit is used to calculate the left-right eye distance w according to the left eye key point and the right eye key point in the face key points;

[0040] A mouth opening degree calculation unit, used for calculating the mouth opening degree h / w according to the upper and lower lip distance h and the left and right eye distance w;

[0041] The first judgment unit is used to determine that if the mouth opening degree h / w reaches a set mouth opening threshold, the face image is an open-mouth image, otherwise the face image is a non-open-mouth image.

[0042] Furthermore, the image extraction module includes:

[0043] A rotation unit, used for calculating the tilt angle of the mouth according to the mouth key points in the face key points, and rotating and aligning the face image according to the tilt angle;

[0044] The normalization unit is used to cut out the mouth area from the aligned face image according to the mouth key points in the face key points and perform a normalization operation to obtain a mouth area image.

[0045] Furthermore, the second judgment module includes:

[0046] A first operation unit, configured to perform a first convolution operation, a first BN operation, a first ReLU operation, and a first pooling operation on the mouth region image to obtain a first feature map;

[0047] A second operation unit is used to expand the edge of the first feature map by a number of pixels, and perform a second convolution operation, a second BN operation, a second ReLU operation, and a second pooling operation to obtain a second feature map;

[0048] A third operation unit, used to expand the edge of the second feature map by a number of pixels, and perform a third convolution operation, a third BN operation, a third ReLU operation and a third pooling operation to obtain a third feature map;

[0049] a fourth operation unit, configured to perform a first full connection operation, a fourth ReLU operation, a first dropout operation, a second full connection operation, a fifth ReLU operation, and a second dropout operation on the third feature map to obtain a fourth feature map;

[0050] A classification unit, configured to perform a third fully connected operation with an output dimension of 2 on the fourth feature map, and classify the image using Softmax to obtain a probability that the mouth region image is a mouth and a probability that the mouth region image is not a mouth;

[0051] The second judgment unit is used to judge that the mouth is not blocked if the probability that the mouth area image is the mouth exceeds a set blocking threshold, otherwise, judge that the mouth is blocked.

[0052] Furthermore, the convolutional neural network is trained by the following modules:

[0053] A training set construction module, used to construct a training set, wherein the training set includes positive samples and negative samples, wherein the positive samples are unobstructed mouth images, and the negative samples are mouth images obstructed by non-mouth images;

[0054] A data expansion module, used for performing data expansion on the training set, wherein the data expansion includes translating, rotating, modifying contrast and / or adding noise to positive samples and negative samples;

[0055] The training module is used to train the convolutional neural network using the training set after data expansion.

[0056] Furthermore, the non-mouth image includes a non-skin color image and a skin color image outside the mouth area; the positive sample and the negative sample both include a number of slightly open mouth images, and the mouth opening degree of the slightly open mouth images does not reach the mouth opening degree of the open mouth image.

[0057] In a third aspect, the present invention provides a computer-readable storage medium for mouth occlusion judgment, comprising a memory for storing processor-executable instructions, wherein the instructions, when executed by the processor, implement the steps of the mouth occlusion judgment method described in the first aspect.

[0058] In a fourth aspect, the present invention provides a device for judging mouth occlusion, comprising at least one processor and a memory storing computer executable instructions, wherein the processor implements the steps of the method for judging mouth occlusion described in the first aspect when executing the instructions.

[0059] The present invention has the following beneficial effects:

[0060] The present invention solves the problem of misjudgment of mouth occlusion in images with open mouth, and has a high judgment accuracy. It also uses a convolutional neural network to perform binary classification of whether the mouth area image is occluded, with high accuracy, good generalization, and strong robustness. Compared with using the entire face image for classification, the calculation amount is small and the speed is fast. It is also easier to control the depth and width of the convolutional neural network, so that the classification has a higher accuracy and a faster speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 is a flow chart of the mouth occlusion determination method of the present invention;

[0062] Figure 2 This is a flow chart of S200 in the mouth occlusion determination method of the present invention;

[0063] Figure 3 is a flow chart of S300 in the mouth occlusion determination method of the present invention;

[0064] Figure 4 This is a flow chart of S400 in the mouth occlusion determination method of the present invention;

[0065] Figure 5 Schematic diagram of the training process of the convolutional neural network in the present invention;

[0066] Figure 6 A schematic diagram of an example of facial key points;

[0067] Figure 7 This is a schematic diagram of the mouth key points among the key points of the face;

[0068] Figure 8 Schematic diagram of mouth area image acquisition and normalization;

[0069] Fig. 9 Schematic diagram of the structure of the convolutional neural network in the present invention;

[0070] Fig.10 is a schematic diagram of a mouth occlusion judgment device of the present invention;

[0071] Fig.11 is a schematic diagram of a mouth opening detection module in the mouth occlusion judgment device of the present invention;

[0072] Fig.12 is a schematic diagram of an image extraction module in the mouth occlusion judgment device of the present invention;

[0073] Fig.13 is a schematic diagram of a second judgment module in the mouth occlusion judgment device of the present invention;

[0074] Fig.14 Schematic diagram of the training module of the convolutional neural network in the present invention. DETAILED DESCRIPTION

[0075] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0076] Embodiment 1:

[0077] The embodiment of the present invention provides a method for judging mouth occlusion. Figure 1 As shown, the method includes:

[0078] S100: Acquire a face image and locate key points of the face.

[0079] This step does not limit the method of obtaining facial images. Exemplarily, facial images can be collected through a facial lens. When collecting, a single image can be collected, or a video can be collected and a single frame or multiple frames of images can be obtained from the video.

[0080] After obtaining the face image, face detection and key point location need to be performed. For example, the face image can be detected using an SSD face detector, and then the key point location can be performed using the TDCNN method to obtain the coordinates of the face key points. The face detector is not limited to SSD, and CRAFT, ADABOOST, etc. can also be used. Similarly, the face key point location method is not limited to using TDCNN, and SDM, etc. can also be used. It only needs to obtain the position of the face key points.

[0081] Different facial key point localization methods can obtain different numbers of facial key points, for example, 68 (such as Figure 6 As shown in the figure), 106, 1000, etc., all of these key points include the key points of the mouth area (hereinafter referred to as the mouth key points) and the key points of the eye area. There are multiple mouth key points, including the left and right corners of the mouth key points and the upper and lower lips key points, etc., and the upper and lower lips key points can have multiple points. There are also multiple key points in the eye area, including the left and right corners of the eyes key points and the upper and lower eyelid key points, etc., and the upper and lower eyelid key points can also have multiple points. Some key point positioning methods can also locate key points such as the pupil center.

[0082] S200: Calculating the mouth opening degree according to the key points of the human face, and classifying the human face image into an open-mouth image or a non-open-mouth image according to the mouth opening degree.

[0083] When the mouth is opened to a certain extent, the open mouth has lost the shape characteristics of a closed mouth and is easily judged as an occluded mouth image. In addition, the teeth and tongue coating in the open mouth will also affect the occlusion judgment of the mouth.

[0084] In order to prevent the image of the mouth being opened from being misidentified, the present invention can remove the image of the mouth being opened before determining whether the mouth is blocked. The above-mentioned key points of the face have been located, and the key points of the face include multiple key points of the mouth. Therefore, the degree of mouth opening can be calculated according to the key points of the face. The image with a mouth opening too large is classified as an open-mouth image, and the image with a small mouth opening (slightly open mouth) and the closed mouth image are classified as non-open-mouth images.

[0085] For the open-mouth image, since the mouth is wide open, it can be directly judged that the mouth is not blocked. Although the open-mouth image is not blocked, such images with wide open mouths are not conducive to the application of face recognition nor to the retention of images. Therefore, the face image can be directly judged as not meeting the requirements and rejected for use.

[0086] S300: For a non-open mouth image, a mouth region image is obtained from the non-open mouth image according to mouth key points in the facial key points.

[0087] The mouth area image obtained in this step is used for subsequent input into the convolutional neural network (CNN) for classification. The mouth area image is smaller than the entire face image. Using only the mouth area image can save calculation and time consumption. In addition, the face image is more complex than the mouth area image. Using only the mouth area image can make CNN easier to train and require less training data. It is easier to control the depth and width of the network, so that the classification has a higher accuracy and faster speed.

[0088] S400: Input the mouth area image into a pre-trained convolutional neural network to determine whether the mouth is blocked.

[0089] The convolutional neural network is a binary classification network that has been pre-trained with a large number of samples (samples with the mouth occluded and samples with the mouth not occluded). By inputting the mouth area image into the network, it can be determined whether the mouth is occluded.

[0090] Compared with the manually designed features (HOG features, etc.) of the prior art, the convolutional neural network does not require manually designed features, but automatically learns and extracts corresponding features. Its features have a good representation effect on mouth occlusion, high classification accuracy, good generalization, and strong robustness.

[0091] The present invention first calculates the degree of mouth opening through facial key points, and divides the facial image into an open-mouth image (an image with a wide mouth) or a non-open-mouth image (an image with a slightly open mouth or an image with a closed mouth) according to the degree of mouth opening. For the open-mouth image, it is directly judged that the mouth is not blocked or the facial image does not meet the requirements and is rejected for use. For the non-open-mouth image, a binary classification convolutional neural network is continued to be used for classification to determine whether the mouth is blocked.

[0092] Compared with the prior art, the present invention solves the problem of misjudgment of mouth occlusion in images with open mouth, and has a high judgment accuracy; and uses a convolutional neural network to perform binary classification of mouth area images as to whether they are occluded, with high accuracy, good generalization, and strong robustness; compared with using the entire face image for classification, the present invention has less computational effort and faster speed, and can also more easily control the depth and width of the convolutional neural network, so that the classification has a higher accuracy and a faster speed.

[0093] In the present invention, the mouth opening degree can be calculated by various indicators, which are illustrated by two specific examples below.

[0094] Example 1:

[0095] like Figure 6 As shown, this example directly uses the distance h between the upper and lower lips as the degree of mouth opening. The larger the distance, the greater the degree of mouth opening. When the distance h is greater than the set threshold, it can be considered that the face image is divided into an open-mouth image.

[0096] Example 2:

[0097] Since the mouth sizes of different users are different, and when collecting face images, the sizes of face images collected at different distances are different due to the imaging rule of cameras and other collection equipment that the near side is larger and the far side is smaller. This will lead to deviations in directly using h as the mouth opening degree, resulting in errors in classifying face images as open-mouth images or non-open-mouth images.

[0098] To solve the above problem, this example uses the ratio h / w of the distance between the upper and lower lips h and the distance w between the left and right eyes as the degree of mouth opening. Since the distance w between the left and right eyes of the same user is unchanged in practice, h / w can accurately describe the degree of mouth opening.

[0099] Based on this example, Figure 2 As shown, the aforementioned S200 includes:

[0100] S210: Calculate the vertical distance between the upper lip key point and the lower lip key point in the facial key points to obtain the upper and lower lip distance h.

[0101] When the number of upper lip key points and lower lip key points are both 1, the vertical distance between these two key points is used as the upper and lower lip distance h. When the number of upper lip key points and lower lip key points are both multiple, the vertical distance between the top key point of the upper lip key points and the bottom key point of the lower lip key points is used as the upper and lower lip distance h, such as Figure 6 shown.

[0102] S220: Calculate the left-right eye distance w according to the left eye key point and the right eye key point in the face key points.

[0103] The key points of the face include the key points of the left eye and the right eye. The distance w between the left and right eyes can be calculated based on these key points. The distance w between the left and right eyes can preferably be the distance between the left and right pupils, such as Figure 6 shown.

[0104] S230: Calculate the mouth opening degree h / w according to the distance h between the upper and lower lips and the distance w between the left and right eyes.

[0105] S240: If the mouth opening degree h / w reaches the set mouth opening threshold, the face image is an open-mouth image, otherwise the face image is a non-open-mouth image.

[0106] The mouth opening threshold can be determined according to needs and statistical experiments. After multiple statistical experiments, the mouth opening threshold in the present invention can be set to 1 / 5. Face images with a mouth opening degree h / w greater than or equal to 1 / 5 are judged as open-mouth images, and the others are regarded as non-open-mouth images (slightly open-mouth images and closed-mouth images).

[0107] For images with open mouths, it can be directly judged that there is no occlusion or the image does not meet the requirements and is rejected. Because the experimental results show that for images with mouth occlusion, the h / w ratio is less than 1 / 5, so images with a mouth opening degree greater than or equal to 1 / 5 cannot be occluded images, and there is no need to perform the next step of binary classification. This can effectively solve the problem of misjudgment in the classification of such images.

[0108] In addition, images with h / w greater than or equal to 1 / 5 are not conducive to the application of face recognition nor to the retention of images, so they can be directly rejected. For images with h / w less than 1 / 5, the above-mentioned neural network binary classification is required.

[0109] As an improvement of the embodiment of the present invention, Figure 3 As shown, the aforementioned S300 includes:

[0110] S310: Calculate the tilt angle of the mouth according to the mouth key points in the facial key points, and rotate and align the facial image according to the tilt angle.

[0111] In this step, the tilt angle of the mouth is preferably calculated by the left and right corner key points. Assuming that the coordinates of the left corner key point are (x1, y1), and the coordinates of the right corner key point are (x2, y2), the tilt angle of the mouth is the angle a between the line connecting the left and right corner key points and the horizontal direction. The calculation method of a is as follows:

[0112]

[0113] Rotating the face image by an angle a can align the face image and eliminate the tilt angle of the mouth.

[0114] S320: Cut out the mouth region from the aligned face image according to the mouth key point among the face key points and perform a normalization operation to obtain a mouth region image.

[0115] When capturing the mouth area, one of the specific implementation methods is as follows Figure 7 , 8 As shown in the figure: Find the mouth edge points among the located face key points, that is, the four key points at the far left, far right, top and bottom, expand these four key points outward by 10 pixels respectively, and get four new edge key points. Take these four edge key points as the edge to get the mouth area, which is a rectangle (not a square). Then normalize and scale the mouth area to an image with a width of 80 pixels and a height of 60 pixels, which is the mouth area image.

[0116] The normalized mouth area image of the present invention is a rectangle, and the normalization method is more conducive to the judgment of mouth occlusion.

[0117] In the present invention, the structure of the convolutional neural network is as follows: Fig. 9 As shown in Figure 2, the specific processing process of the convolutional neural network on the mouth area image is as follows: Figure 4 As shown, including:

[0118] S410: Perform a first convolution operation, a first BN operation, a first ReLU operation, and a first pooling operation on the mouth area image to obtain a first feature map.

[0119] For example, the mouth area image (data) is a 3-channel image with a width of 80 pixels and a height of 60 pixels. The convolution kernel (kemel size) of the first convolution operation (conv1) is 7*7 and the stride (stride) is 4. The mouth area image is subjected to the first convolution operation to obtain a feature map of 36 channels with a size of 19*14. The first BN operation (batch1) and the first ReLU operation (relu1) are performed on the feature map, and the obtained feature map is also a feature map of 36 channels with a size of 19*14.

[0120] The first pooling operation (pool1) is performed on the feature map of 36 channels with a size of 19*14. The first pooling operation is a Maxpooling operation with a size of 3*3 and a step size of 2, and a 36-channel sampling result with a size of 9*7 is obtained, which is the first feature map.

[0121] S420: Expand the edge of the first feature map by a number of pixels, and perform a second convolution operation, a second BN operation, a second ReLU operation, and a second pooling operation to obtain a second feature map.

[0122] Exemplarily, the edge of the first feature map of 36 channels with a size of 9*7 is expanded by 2 pixels, and a second convolution operation (conv2) is performed with a convolution kernel of size 5*5 and a step size of 1 to obtain a feature map of 96 channels with a size of 9*7. The second BN operation (batch2) and the second ReLU (relu2) operation are performed on the obtained feature map, and the obtained feature map is also a feature map of 96 channels with a size of 9*7.

[0123] The second pooling operation (pool2) is performed on the above-mentioned feature map of 96 channels with a size of 9*7. The second pooling operation is a Maxpooling operation with a size of 3*3 and a step size of 2, and a 96-channel sampling result with a size of 4*3 is obtained, which is the second feature map.

[0124] S430: Expand the edge of the second feature map by several pixels, and perform a third convolution operation, a third BN operation, a third ReLU operation, and a third pooling operation to obtain a third feature map.

[0125] Exemplarily, the edge of the second feature map of 96 channels with a size of 4*3 is expanded by 1 pixel, and a convolution kernel with a size of 3*3 is used with a step size of 1 to obtain a feature map of 144 channels with a size of 4*3. The third BN operation (batch3) and the third ReLU operation (relu3) are performed on the obtained feature map, and the obtained feature map is also a feature map of 144 channels with a size of 4*3.

[0126] The third pooling operation (pool5) is performed on the feature map of 144 channels with a size of 4*3. The third pooling operation is a Maxpooling operation with a size of 3*3 and a step size of 2, and a 144-channel sampling result with a size of 2*1 is obtained, which is the third feature map.

[0127] S440: Perform a first full connection operation, a fourth ReLU operation, a first dropout operation, a second full connection operation, a fifth ReLU operation, and a second dropout operation on the third feature map to obtain a fourth feature map.

[0128] Exemplarily, a first fully connected operation (fc6) is performed on the third feature map of 144 channels with a size of 2*1 to obtain a feature vector with a dimension of 512, and then a fourth ReLU operation (relu6) and a first dropout operation (drop6) with a ratio of 0.5 are performed on the feature vector.

[0129] The result of the first dropout operation is subjected to a second fully connected operation (fc7) to obtain a feature vector with a dimension of 512. Then, the fifth ReLU operation (relu7) and the second dropout operation (drop7) with a ratio of 0.5 are performed on this feature vector to obtain the fourth feature map.

[0130] S450: Perform a third fully connected operation (fc8) with an output dimension of 2 on the fourth feature map, and use Softmax to classify it to obtain the probability that the mouth area image is a mouth and the probability that it is not a mouth.

[0131] The goal of the convolutional neural network classification in the present invention is to determine whether the mouth area image is a mouth. The output of the network is a two-dimensional vector, the first dimension represents the probability value of the image is not a mouth, and the second dimension represents the probability value of the image is a mouth.

[0132] S460: If the probability that the mouth area image is the mouth exceeds the set occlusion threshold, it is determined that the mouth is not occluded, otherwise it is determined that the mouth is occluded.

[0133] This step uses the output value of the second dimension (the probability value of the mouth) to equivalently replace the possibility that the mouth is not blocked, denoted as p. Set the occlusion threshold T. If p>T, it means that the mouth is not blocked, otherwise it means that the mouth is blocked.

[0134] The network used in the present invention is not only effective but also has fewer parameters. It has a good description effect on the characteristics of whether the mouth is blocked, ensuring the accuracy of classification. At the same time, the depth and width of the network are controlled, the time complexity is reduced, and the classification has a higher accuracy and a faster speed.

[0135] The aforementioned convolutional neural network needs to be pre-trained before use. The training method is as follows: Figure 5 As shown, including:

[0136] S100': construct a training set, the training set includes positive samples and negative samples, the positive samples are unobstructed mouth images, and the negative samples are mouth images obstructed by non-mouth images.

[0137] The positive samples of the training set are normal, unobstructed mouth images. For negative samples, a variety of non-mouth area images can be used to obstruct the mouth image for construction.

[0138] S200': perform data expansion on the training set, where the data expansion includes translating, rotating, modifying contrast, and / or adding noise to the positive samples and negative samples.

[0139] In order to obtain better training results, both positive and negative samples need to be expanded to enhance data diversity.

[0140] S300': Train the convolutional neural network using the data augmented training set.

[0141] The aforementioned non-mouth image includes a non-skin color image and a skin color image outside the mouth area. The non-skin color image may be a cardboard image, a black block, etc., and the skin color image outside the mouth area may be a skin color image of the hand, face, etc.

[0142] In order to obtain better results in the classification of slightly open mouth images, positive and negative samples of slightly open mouth can be added in the training data preparation, so that both the positive samples and the negative samples include several slightly open mouth images, in which the mouth is open but the degree of opening of the mouth does not reach the degree of opening of the mouth in the open mouth image.

[0143] Embodiment 2:

[0144] The embodiment of the present invention provides a mouth occlusion judgment device, such as Fig.10 As shown, the device comprises:

[0145] The image acquisition module 10 is used to acquire a face image and locate key points of the face.

[0146] The mouth opening detection module 20 is used to calculate the mouth opening degree according to the key points of the human face, and classify the human face image into an open-mouth image or a non-open-mouth image according to the mouth opening degree.

[0147] The image extraction module 30 is used to obtain a mouth area image from the non-mouth-opening image according to the mouth key points in the facial key points.

[0148] The second judgment module 40 is used to input the mouth area image into a pre-trained convolutional neural network to determine whether the mouth is blocked.

[0149] The present invention first calculates the degree of mouth opening through facial key points, and divides the facial image into an open-mouth image (an image with a wide mouth) or a non-open-mouth image (an image with a slightly open mouth or an image with a closed mouth) according to the degree of mouth opening. For the open-mouth image, it is directly judged that the mouth is not blocked or the facial image does not meet the requirements and is rejected for use. For the non-open-mouth image, a binary classification convolutional neural network is continued to be used for classification to determine whether the mouth is blocked.

[0150] Compared with the prior art, the present invention solves the problem of misjudgment of mouth occlusion in images with open mouth, and has a high judgment accuracy; and uses a convolutional neural network to perform binary classification of mouth area images as to whether they are occluded, with high accuracy, good generalization, and strong robustness; compared with using the entire face image for classification, the present invention has less computational effort and faster speed, and can also more easily control the depth and width of the convolutional neural network, so that the classification has a higher accuracy and a faster speed.

[0151] The aforementioned mouth opening detection module 20 is as follows Fig.11 As shown, it includes:

[0152] The upper and lower lip distance calculation unit 21 is used to calculate the vertical distance between the upper lip key point and the lower lip key point in the face key points to obtain the upper and lower lip distance h.

[0153] The left-right eye distance calculation unit 22 is used to calculate the left-right eye distance w according to the left eye key point and the right eye key point in the face key points.

[0154] The mouth opening degree calculation unit 23 is used to calculate the mouth opening degree h / w according to the distance h between the upper and lower lips and the distance w between the left and right eyes.

[0155] The first judgment unit 24 is used to determine that the face image is an open-mouth image if the mouth opening degree h / w reaches a set mouth opening threshold, otherwise the face image is a non-open-mouth image.

[0156] like Fig.12 As shown, the image extraction module 30 includes:

[0157] The rotation unit 31 is used to calculate the tilt angle of the mouth according to the mouth key points in the face key points, and rotate and align the face image according to the tilt angle.

[0158] The normalization unit 32 is used to cut out the mouth area from the aligned face image according to the mouth key point among the face key points and perform a normalization operation to obtain a mouth area image.

[0159] The second judgment module 40 is as follows Fig.13 As shown, it includes:

[0160] The first operation unit 41 is used to perform a first convolution operation, a first BN operation, a first ReLU operation and a first pooling operation on the mouth area image to obtain a first feature map.

[0161] The second operation unit 42 is used to expand the edge of the first feature map by a number of pixels, and perform a second convolution operation, a second BN operation, a second ReLU operation and a second pooling operation to obtain a second feature map.

[0162] The third operation unit 43 is used to expand the edge of the second feature map by several pixels, and perform a third convolution operation, a third BN operation, a third ReLU operation and a third pooling operation to obtain a third feature map.

[0163] The fourth operation unit 44 is used to perform a first full connection operation, a fourth ReLU operation, a first dropout operation, a second full connection operation, a fifth ReLU operation, and a second dropout operation on the third feature map to obtain a fourth feature map.

[0164] The classification unit 45 is used to perform a third fully connected operation with an output dimension of 2 on the fourth feature map, and use Softmax to perform classification to obtain the probability that the mouth area image is a mouth and the probability that it is not a mouth.

[0165] The second judgment unit 46 is configured to judge that the mouth is not blocked if the probability that the mouth area image is the mouth exceeds a set blocking threshold, otherwise, judge that the mouth is blocked.

[0166] like Fig.14 As shown in the figure, the aforementioned convolutional neural network is trained through the following modules:

[0167] The training set construction module 10' is used to construct a training set, the training set includes positive samples and negative samples, the positive samples are unobstructed mouth images, and the negative samples are mouth images obstructed by non-mouth images.

[0168] The data expansion module 20' is used to expand the data of the training set, and the data expansion includes translating, rotating, modifying the contrast and / or adding noise to the positive and negative samples.

[0169] The training module 30' is used to train the convolutional neural network using the training set after data expansion.

[0170] The non-mouth images include non-skin color images and skin color images outside the mouth area; the positive samples and negative samples both include a number of slightly open mouth images, and the mouth opening degree of the slightly open mouth images does not reach the mouth opening degree of the open mouth images.

[0171] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those in the aforementioned method embodiment 1. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment 1. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment 1, and will not be repeated here.

[0172] Embodiment 3:

[0173] The method described in the above-mentioned embodiment 1 provided in this specification can realize the business logic through a computer program and record it on a storage medium, and the storage medium can be read and executed by a computer to realize the effect of the scheme described in embodiment 1 of this specification. Therefore, the present invention also provides a computer-readable storage medium for mouth occlusion judgment, including a memory for storing processor-executable instructions, and when the instructions are executed by the processor, the steps of the mouth occlusion judgment method of embodiment 1 are realized.

[0174] Compared with the prior art, the present invention solves the problem of misjudgment of mouth occlusion in images with open mouth, and has a high judgment accuracy; and uses a convolutional neural network to perform binary classification of mouth area images as to whether they are occluded, with high accuracy, good generalization, and strong robustness; compared with using the entire face image for classification, the present invention has less computational effort and faster speed, and can also more easily control the depth and width of the convolutional neural network, so that the classification has a higher accuracy and a faster speed.

[0175] The storage medium may include a physical device for storing information, which is usually a medium that digitizes the information and then stores it in an electrical, magnetic or optical manner. The storage medium may include: a device that stores information in an electrical energy manner, such as various memories, such as RAM, ROM, etc.; a device that stores information in a magnetic energy manner, such as a hard disk, a floppy disk, a magnetic tape, a magnetic core memory, a magnetic bubble memory, a USB flash drive; a device that stores information in an optical manner, such as a CD or a DVD. Of course, there are other readable storage media, such as quantum memory, graphene memory, etc.

[0176] The above-mentioned device may also include other implementations according to the description of method embodiment 1. The specific implementation methods may refer to the description of the relevant method embodiment 1, and will not be described one by one here.

[0177] Embodiment 4:

[0178] The present invention also provides a device for judging mouth occlusion, which may be a separate computer, or may include an actual operating device using one or more of the methods or one or more of the embodiments of this specification. The device for judging mouth occlusion may include at least one processor and a memory storing computer executable instructions, and when the processor executes the instructions, the steps of the method for judging mouth occlusion described in any one or more of the embodiments 1 are implemented.

[0179] Compared with the prior art, the present invention solves the problem of misjudgment of mouth occlusion in images with open mouth, and has a high judgment accuracy; and uses a convolutional neural network to perform binary classification of mouth area images as to whether they are occluded, with high accuracy, good generalization, and strong robustness; compared with using the entire face image for classification, the present invention has less computational effort and faster speed, and can also more easily control the depth and width of the convolutional neural network, so that the classification has a higher accuracy and a faster speed.

[0180] The above-mentioned device may also include other implementation modes according to the description of the method or device embodiment. The specific implementation mode may refer to the description of the relevant method embodiment 1, and will not be described one by one here.

[0181] It should be noted that the device or system described above in this specification may also include other implementation methods according to the description of the relevant method embodiments. The specific implementation methods can refer to the description of the method embodiments, which will not be described one by one here. The various embodiments in this specification are described in a progressive manner, and the same and similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the hardware + program type and storage medium + program embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0182] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0183] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0184] For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing one or more of the present specification, the functions of each module can be implemented in the same or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0185] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules for implementing the method and structures within the hardware component.

[0186] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0187] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0188] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0189] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0190] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method or device including the element.

[0191] It should be understood by those skilled in the art that one or more embodiments of the present specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0192] One or more embodiments of this specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of this specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0193] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. In the description of this specification, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, in the absence of contradiction, a person skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0194] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. They should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for judging mouth occlusion, characterized in that: The method comprises: Obtain a face image and locate key points of the face; Calculating the mouth opening degree according to the facial key points, and dividing the facial image into an open-mouth image or a non-open-mouth image according to the mouth opening degree; For a non-open mouth image, a mouth area image is obtained from the non-open mouth image according to the mouth key points in the face key points; The mouth area image is input into a pre-trained convolutional neural network to determine whether the mouth is occluded.

2. The mouth occlusion determination method according to claim 1, characterized in that: The step of calculating the mouth opening degree according to the facial key points and classifying the facial image into an open-mouth image or a non-open-mouth image according to the mouth opening degree comprises: Calculate the vertical distance between the upper lip key point and the lower lip key point in the facial key points to obtain the upper and lower lip distance h; The left-right eye distance w is calculated based on the left-eye key point and the right-eye key point in the face key points; The mouth opening degree h / w is calculated according to the distance h between the upper and lower lips and the distance w between the left and right eyes; If the mouth opening degree h / w reaches the set mouth opening threshold, the face image is an open-mouth image, otherwise the face image is a non-open-mouth image.

3. The mouth occlusion determination method according to claim 2, characterized in that: The method of acquiring the mouth region image from the non-mouth-opening image according to the mouth key points in the face key points includes: Calculating the tilt angle of the mouth according to the mouth key points in the facial key points, and rotating and aligning the facial image according to the tilt angle; The mouth region is cut out from the aligned face image according to the mouth key points among the face key points and normalized to obtain a mouth region image.

4. The mouth occlusion determination method according to any one of claims 1 to 3, characterized in that: The step of inputting the mouth region image into a pre-trained convolutional neural network to determine whether the mouth is blocked includes: Performing a first convolution operation, a first BN operation, a first ReLU operation, and a first pooling operation on the mouth area image to obtain a first feature map; Expanding the edge of the first feature map by a number of pixels, and performing a second convolution operation, a second BN operation, a second ReLU operation, and a second pooling operation to obtain a second feature map; Expanding the edge of the second feature map by a number of pixels, and performing a third convolution operation, a third BN operation, a third ReLU operation, and a third pooling operation to obtain a third feature map; Performing a first full connection operation, a fourth ReLU operation, a first dropout operation, a second full connection operation, a fifth ReLU operation, and a second dropout operation on the third feature map to obtain a fourth feature map; Performing a third full connection operation with an output dimension of 2 on the fourth feature map, and using Softmax to perform classification, to obtain the probability that the mouth area image is a mouth and the probability that it is not a mouth; If the probability that the mouth area image is the mouth exceeds the set occlusion threshold, the mouth is judged to be unoccluded, otherwise the mouth is judged to be occluded.

5. The mouth occlusion determination method according to claim 4, characterized in that: The convolutional neural network is trained by the following method: Constructing a training set, wherein the training set includes positive samples and negative samples, wherein the positive samples are unobstructed mouth images, and the negative samples are mouth images obstructed by non-mouth images; Performing data expansion on the training set, wherein the data expansion includes translating, rotating, modifying contrast, and / or adding noise to positive samples and negative samples; The convolutional neural network is trained using the data augmented training set.

6. The mouth occlusion determination method according to claim 5, characterized in that: The non-mouth images include non-skin color images and skin color images outside the mouth area; the positive samples and negative samples both include a number of slightly open mouth images, and the mouth opening degree of the slightly open mouth images does not reach the mouth opening degree of the open mouth images.

7. A mouth occlusion judgment device, characterized in that: The device comprises: An image acquisition module is used to acquire a face image and locate key points of the face; A mouth opening detection module, used for calculating the mouth opening degree according to the key points of the human face, and classifying the human face image into an open-mouth image or a non-open-mouth image according to the mouth opening degree; An image extraction module, for obtaining a mouth area image from a non-open mouth image according to a mouth key point in a human face key point; The second judgment module is used to input the mouth area image into a pre-trained convolutional neural network to determine whether the mouth is blocked.

8. The mouth occlusion determination device according to claim 7, characterized in that: The second judgment module includes: A first operation unit, configured to perform a first convolution operation, a first BN operation, a first ReLU operation, and a first pooling operation on the mouth region image to obtain a first feature map; A second operation unit is used to expand the edge of the first feature map by a number of pixels, and perform a second convolution operation, a second BN operation, a second ReLU operation, and a second pooling operation to obtain a second feature map; A third operation unit, used to expand the edge of the second feature map by a number of pixels, and perform a third convolution operation, a third BN operation, a third ReLU operation and a third pooling operation to obtain a third feature map; a fourth operation unit, configured to perform a first full connection operation, a fourth ReLU operation, a first dropout operation, a second full connection operation, a fifth ReLU operation, and a second dropout operation on the third feature map to obtain a fourth feature map; A classification unit, configured to perform a third fully connected operation with an output dimension of 2 on the fourth feature map, and classify the image using Softmax to obtain a probability that the mouth region image is a mouth and a probability that the mouth region image is not a mouth; The second judgment unit is used to judge that the mouth is not blocked if the probability that the mouth area image is the mouth exceeds a set blocking threshold, otherwise, judge that the mouth is blocked.

9. A computer-readable storage medium for mouth occlusion determination, characterized in that: It comprises a memory for storing processor executable instructions, and when the instructions are executed by the processor, the steps of the mouth occlusion determination method according to any one of claims 1 to 6 are implemented.

10. A device for judging mouth occlusion, characterized in that: The method comprises at least one processor and a memory storing computer executable instructions, wherein when the processor executes the instructions, the steps of the mouth occlusion determination method according to any one of claims 1 to 6 are implemented.

Citation Information

Patent Citations

  • Cascaded convolutional neural network based human face occlusion detection method

    CN105868689A

  • Mouth opening / closing state detection method based on deep learning

    CN106250840A