Methods for detecting facial organ status and blinking, network training methods and devices

Abstract

This application discloses a method, network training method, and apparatus for detecting the state of facial organs and blinking. The method for detecting the state of facial organs includes: extracting an image of the target's facial organs and a contour map of the facial organs from an image containing the target face; overlaying the facial organ image and the contour map to obtain an overlaid image; and detecting the overlaid image to determine the open / closed state of the facial organs. This application can improve the accuracy of facial organ state detection.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, network training method and apparatus for detecting the state of facial organs and blinking. Background Technology

[0002] With the development of image intelligent processing technology, the detection of facial organs such as eyes and mouth, as well as their movements, has been widely applied. However, current methods for detecting the state of facial organs still have some problems, such as low detection accuracy. Summary of the Invention

[0003] This application provides a method, network training method, and apparatus for detecting the state of facial organs and blinking, which can improve the detection accuracy.

[0004] To address the above problems, this application provides a method for detecting the state of facial organs, the method comprising:

[0005] Extract facial organ images and contour maps of the target from images containing the target face;

[0006] The image of facial organs and the outline of facial organs are superimposed to obtain a superimposed image;

[0007] The superimposed images are inspected to determine the opening and closing states of facial organs.

[0008] The steps of extracting facial organ images and contour maps of the target based on images containing the target face include:

[0009] Extract facial organ images of the target from an image containing the target face;

[0010] An edge-aware network is used to process facial organ images to obtain contour maps.

[0011] The step of processing facial organ images using an edge-aware network includes the following:

[0012] The orientation information of the target face and the aspect ratio of the facial organs are determined based on the image containing the target face;

[0013] If, based on orientation information, it is determined that the target face is approximately facing the device that acquires an image containing the target face, and the aspect ratio of the facial organs is less than the closure threshold, it is determined that the facial organs are in a closed state.

[0014] Otherwise, the step of processing the facial organ image using an edge-aware network is performed.

[0015] Among them, the eyes are the target facial organ.

[0016] The steps for determining the orientation information of the target face and calculating the aspect ratio of facial features based on an image containing the target face include:

[0017] The difference in aspect ratio between the left and right eyes of the target is calculated based on the image containing the target's face, and the degree to which the tip of the target's nose deviates from the center of the target's face is calculated. The difference and degree are used as orientation information.

[0018] In a case where the target face is approximately facing the device acquiring an image containing the target face based on orientation information, and the aspect ratio of the facial features is less than a closure threshold, the step of determining that the facial features are in a closed state includes:

[0019] If the deviation of the tip of the nose from the center of the face is less than the deviation threshold, the difference is less than the error threshold, and the aspect ratio of the facial features is less than the closure threshold, then the facial features are determined to be in a closed state.

[0020] Specifically, facial organ images of the target are extracted from images containing the target's face.

[0021] Extract facial feature points from an image containing the target face;

[0022] Determine the center point of facial organs based on facial feature points;

[0023] Using the center point of the facial organs as the center and the width of the facial organs as the side length, extract the facial organ image from the image containing the target face.

[0024] The step of overlaying facial organ images and facial organ contour maps to obtain an overlaid image includes:

[0025] The facial organ images and their outlines are overlaid on the channels to obtain an overlaid image.

[0026] The method includes:

[0027] Obtain a set of facial organ images;

[0028] An edge perception network and a state detection network were trained based on a set of facial organ images;

[0029] Among them, the edge perception network is used to process facial organ images to obtain facial organ contour maps, and the state detection network is used to detect the superimposed image formed by superimposing facial organ images and facial organ contour maps to determine the open and closed state of facial organs.

[0030] The steps for training the edge-aware network and the state detection network based on a set of facial organ images include:

[0031] An edge-aware network was trained using a set of facial organ images.

[0032] The facial organ images are input into the trained edge perception network to obtain the contour map of the facial organs;

[0033] The image of facial organs and the outline of facial organs are superimposed to obtain the superimposed image;

[0034] The state detection network is trained based on overlaid images.

[0035] The steps for training the edge-aware network based on a set of facial organ images include:

[0036] The facial organ images are input into the edge perception network to obtain the contour map of the facial organs;

[0037] The first loss is calculated using the connectivity graph of feature points in the contour map and facial organ image;

[0038] Update the parameters of the edge-aware network in the direction where the first loss decreases.

[0039] The steps involved in training the state detection network based on overlaid images are as follows:

[0040] The superimposed image corresponding to the facial organ image is input into the state detection network to obtain the detection value of the opening and closing state of the facial organs in the facial organ image;

[0041] The second loss is calculated using the detected and true values ​​of the open and closed states of facial organs in facial organ images;

[0042] Update the parameters of the state detection network in the direction of the second loss reduction.

[0043] The step of training the state detection network based on the overlay image includes:

[0044] The edge perception network and the state detection network are jointly trained based on a set of facial organ images.

[0045] The steps for jointly training the edge perception network and the state detection network based on a set of facial organ images include:

[0046] During joint training, the first loss and the second loss are calculated, and the weighted values ​​of the first loss and the second loss are calculated. The parameters of the edge perception network and the state detection network are updated in the direction of decreasing weighted values.

[0047] To address the aforementioned problems, this application also provides a blink detection method, which includes:

[0048] Based on the above facial organ state detection method, the opening and closing state of the target eye in the current frame image is detected;

[0049] Determine whether the target blinks based on the opening and closing state of the target's eyes in the current frame image.

[0050] To address the aforementioned problems, this application also provides an electronic device comprising a processor; the processor is configured to execute instructions to implement the aforementioned method.

[0051] To address the aforementioned problems, this application also provides a computer-readable storage medium for storing instruction / program data that can be executed to implement the above-described method.

[0052] This application first extracts facial organ images and contour maps of the target face from an image containing the target face. Then, it detects the superimposed image of the facial organ images and contour maps to determine the open / closed state of the target's facial organs. This allows the detection focus to be on the contours of the facial organs, enabling more accurate extraction of high-quality features near the edges of the facial organs, thus improving the accuracy of facial organ state detection. Furthermore, compared to existing methods that detect whether a target is blinking by multiplying a Gaussian blurred heatmap generated from the eye center point with the eye image, this application not only more completely preserves effective and global information about the facial organs but also better leverages the network's effective extraction of facial organ region features, resulting in higher accuracy for the facial organ state detection method. Moreover, the implementation of this method does not rely on auxiliary equipment or sensors, reducing usage costs. Attached Figure Description

[0053] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0054] Figure 1 This is a flowchart illustrating one embodiment of the facial organ state detection method of this application;

[0055] Figure 2 This is a schematic diagram of facial feature points extracted by the facial organ state detection method of this application;

[0056] Figure 3 This is a schematic diagram of the facial organ image extracted by the facial organ state detection method of this application;

[0057] Figure 4 This is a schematic diagram of the connection of feature points of facial organs in the facial organ state detection method of this application;

[0058] Figure 5 This is a schematic diagram of the outline of the facial organs in the facial organ state detection method of this application;

[0059] Figure 6 This is a flowchart illustrating another embodiment of the facial organ state detection method of this application;

[0060] Figure 7 This is a schematic diagram of the structure of the edge sensing network and the state detection network in the facial organ state detection method of this application;

[0061] Figure 8 This is a flowchart illustrating one embodiment of the blink detection method of this application;

[0062] Figure 9 This is a flowchart illustrating one implementation method of the training method network for the facial organ state detection method of this application;

[0063] Figure 10 This is a schematic diagram of the structure of one embodiment of the electronic device of this application;

[0064] Figure 11 This is a schematic diagram of one embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0065] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application. In addition, unless otherwise specified (e.g., "or additionally" or "or in alternatives"), the term "or" as used herein refers to a non-exclusive "or" (i.e., "and / or"). Furthermore, the various embodiments described herein are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments.

[0066] Specifically, such as Figure 1 As shown, the facial organ state detection method of this embodiment includes the following steps. It should be noted that the step numbers are for simplification only and are not intended to limit the execution order of the steps. The execution order of each step in this embodiment can be arbitrarily changed without departing from the technical concept of this application.

[0067] S101: Extract facial organ images and contour maps of the target based on an image containing the target face.

[0068] First, the facial organ images and contour maps of the target can be extracted from the image containing the target's face. Then, the facial organ images and contour maps of the target can be detected to determine the open and closed state of the target's facial organs.

[0069] The target of this application may be an animal with a face, such as a human, cat, or dog. Facial features may include eyes or a mouth, etc., and there is no limitation herein.

[0070] In one implementation, such as Figure 2 As shown, the coordinates of feature points such as facial contours, eyebrows, eyes, nose, and mouth areas on the target face in the image can be determined first; then, the positions of the detected facial organs can be determined based on the coordinates of the feature points of the target face in the image; and finally, the positions of the facial organs can be determined based on the positions of the facial organs. Figure 3 The image of the target's facial organs is extracted from the image containing the target's face; it can also connect the feature points on the detected facial organs (e.g., if the facial organ is the eye, such as...). Figure 4 As shown, the six feature points of the eye region can be connected to form a closed edge to obtain... Figure 5 The diagram shows the outline of the facial features.

[0071] Optionally, a face alignment algorithm can be used to determine the location of the detected facial organs and the feature points at those organs. Specifically, a rigid transformation, such as rotation or translation, can be used to transform the face image onto the template face to achieve face alignment.

[0072] Among them, based on the location of facial organs Figure 3 The steps for extracting the facial organ image of the target from an image containing the target face may include: calculating the coordinates of the center point of the facial organ; and then, using the center point of the facial organ as the center and the width of the facial organ as the side length, extracting the facial organ image from the image containing the target face. For example, assuming the facial organ is the eye, the average coordinates of the feature points at the two corners of each eye can be used as the coordinates of the center point of each eye. As another example, assuming the facial organ is the mouth, the average coordinates of the feature points at the two corners of the mouth can be used as the coordinates of the center point of the mouth.

[0073] In another implementation, to avoid inaccurate facial organ contour maps due to feature point extraction errors, and to address the issue that relying solely on facial organ feature points for contour and statistical index analysis is susceptible to fluctuations in feature point localization accuracy, which can lead to reduced state detection performance, a trained edge perception network can be used to extract facial organ contour maps from the target's facial organ image. The open / closed state of the facial organs can then be determined by detecting both the contour maps and the facial organ images.

[0074] S102: Overlay the facial organ image and the facial organ contour map to obtain an overlaid image.

[0075] After obtaining the facial organ image and the facial organ contour map of the target based on step S101, the facial organ image and the facial organ contour map can be superimposed to detect the superimposed image and determine the open and closed state of the facial organs.

[0076] Optionally, facial organ images and facial organ contour maps can be superimposed on a single channel to obtain a superimposed image. For example, if both the facial organ images and facial organ contour maps are 64*64 single-channel images, superimposing them will result in a 64*64 dual-channel image.

[0077] S103: Detect the superimposed image to determine the opening and closing state of facial organs.

[0078] The superimposed image obtained in step S102 can be used for detection to determine the open and closed state of facial organs. This allows the detection focus to be on the contours of the facial organs, enabling more accurate extraction of high-quality features near the edges of these organs, thus improving the accuracy of facial organ state detection. Furthermore, compared to existing methods that use a Gaussian blurred heatmap generated from the eye center point multiplied with the eye image to determine whether the target is blinking, this application not only more completely preserves effective and global information about facial organs but also better leverages the network's ability to extract facial organ region features, resulting in better performance and avoiding errors in network optimization due to environmental factors.

[0079] Alternatively, a state detection network can be used to detect the superimposed image to determine the open and closed state of facial organs based on the output of the state detection network.

[0080] For example, assuming that the higher the probability output by the state detection network, the more likely the facial features are to be in a closed state, then in step S103, if the probability output by the state detection network after processing the superimposed image is greater than a first threshold, then the facial features are in a closed state; otherwise, the facial features are in an open state. The first threshold can be set according to actual conditions and is not limited here; for example, it can be 0.5.

[0081] For example, assuming that the lower the probability output by the state detection network, the more likely the facial features are to be in a closed state, then in step S103, if the probability output by the state detection network after processing the superimposed image is less than the second threshold, the facial features are in a closed state; otherwise, the facial features are in an open state. The second threshold can be set according to actual conditions and is not limited here; for example, it can be 0.5 or 0.3.

[0082] In this embodiment, facial organ images and contour maps of the target face are first extracted from an image containing the target face. Then, the superimposed image of the target's facial organ images and contour maps is detected to determine the open / closed state of the target's facial organs. This allows the detection attention to focus on the contours of the facial organs through the contour maps, enabling more accurate extraction of high-quality features near the edges of the facial organs, thereby improving the accuracy of facial organ state detection. Furthermore, compared to existing methods that detect whether a target is blinking by multiplying a Gaussian blurred heatmap generated from the eye center point with the eye image, this application not only more completely preserves effective and global information about the facial organs but also better leverages the network's effective extraction of facial organ region features, resulting in higher accuracy for the facial organ state detection method. Moreover, the implementation of this method does not rely on auxiliary devices or sensors, reducing usage costs.

[0083] To avoid unnecessary time consumption caused by using edge sensing networks and state detection networks to judge all video frame sequences, this application can filter out images where facial organs are obviously in a closed state based on the aspect ratio of facial organs before processing facial organ images using edge sensing networks and state detection networks. This reduces the number of images detected by edge sensing networks and state detection networks, avoids unnecessary time consumption, and balances algorithm performance and time consumption in cooperative scenarios. Specifically, as shown in Figure 6... Figure 6 This is a flowchart illustrating another embodiment of the facial organ state detection method of this application. The facial organ state detection method of this embodiment includes the following steps. It should be noted that the step numbers are for simplification only and are not intended to limit the execution order of the steps. The execution order of each step in this embodiment can be arbitrarily changed without departing from the technical concept of this application.

[0084] S201: Extract the facial organ image of the target from the image containing the target face.

[0085] As shown in step S101, it will not be elaborated here.

[0086] S202: Determine the orientation information of the target face and calculate the aspect ratio of the facial organs based on the image containing the target face.

[0087] S203: Confirm whether facial features are clearly closed based on orientation information and aspect ratio.

[0088] The orientation information of the target face can be determined based on an image containing the target face, and the aspect ratio of the facial organs can be calculated. Based on the orientation information and aspect ratio, it can be confirmed whether the facial organs are obviously in a closed state.

[0089] Optionally, if, based on an image containing the target face, it is determined that the target face is approximately facing the device acquiring the image containing the target face, and the aspect ratio of the facial organs is less than the closure threshold, then the facial organs are determined to be in a closed state, and the process proceeds to step S203; otherwise, the process proceeds to step S204. Since the calculated aspect ratio of the facial organs differs significantly from the actual value when the target face is not directly facing the acquisition device (e.g., swaying left, right, up, or down), this application only uses the aspect ratio of the facial organs to determine their state when the target face is directly facing the acquisition device. This avoids misjudging the state of the facial organs by using their aspect ratio when the aspect ratio error is large. In other words, it constrains the impact of scenarios where the aspect ratio cannot reflect the true state of the facial organs due to posture on the detection of open / closed states, thus avoiding false detections, improving detection accuracy, and balancing algorithm performance and time consumption in different scenarios.

[0090] In addition, facial organs are Figure 4 Taking the eyes in the image as an example, the formula for calculating the eye aspect ratio (EAR) is as follows:

[0091] Taking the mouth as an example of a facial feature, using... Figure 2 The formula for calculating the aspect ratio of the mouth using feature points is as follows:

[0092] The method for determining the orientation information of the target face is as follows.

[0093] In one implementation, a vector can be calculated between a preset feature point of the target face and the center point of the target face bounding box. If the vector is within a preset range, the target face is approximately facing the device that acquires the image containing the target face; otherwise, the target face is not facing the device that acquires the image containing the target face.

[0094] In another implementation, the distance between preset feature points of the target face and the center point of the target face bounding box can be calculated. If the facial organ is the eye, the difference between the aspect ratios of the left and right eyes can be calculated. If the distance between the preset feature points of the target face and the center point of the target face bounding box is less than a deviation threshold, and the difference between the aspect ratios of the left and right eyes is less than an error threshold, then the target face is approximately facing the device that acquires the image containing the target face. Otherwise, if the distance between the preset feature points of the target face and the center point of the target face bounding box is greater than or equal to the deviation threshold, and / or the difference between the aspect ratios of the left and right eyes is greater than or equal to the error threshold, then the target face is not facing the device that acquires the image containing the target face.

[0095] The aforementioned preset feature points can be the tip of the nose or the corner of the eye of the target face, etc. There are no restrictions here, and the specific settings can be made according to the actual situation.

[0096] Furthermore, the aforementioned deviation threshold, error threshold, and closure threshold can be set according to actual circumstances, and are not restricted here.

[0097] According to the definition of aspect ratio, larger facial features generally have a larger aspect ratio, and the aspect ratio also fluctuates more significantly. Furthermore, due to artifacts and other issues, there is a certain deviation in feature point localization. This deviation usually manifests as the actual aspect ratio being smaller than the calculated value, meaning the aspect ratio cannot accurately reflect the true state of the facial features. In summary, an excessively large closure threshold can lead to a large number of false detections. Therefore, the closure threshold can be set smaller, for example, to 0.1 or 0.12, to ensure that the facial features are in a closed state when the aspect ratio threshold is triggered, thus avoiding false detections.

[0098] Furthermore, according to the definition of aspect ratio, as the target yaw angle increases, and when the distance between the upper and lower edges of facial features remains constant (i.e., the opening amplitude of facial features remains constant), the aspect ratio of facial features tends to increase due to the shortening of the lateral distance. In addition, when the facial feature is the eye, the aspect ratio of the inner eye is significantly larger than that of the outer eye, thus introducing an error. The error threshold is used to constrain the impact of scene conditions where the aspect ratio fails to reflect the true state of facial features due to posture on state detection. The error threshold can be 0.25 or 0.3 to ensure that the target face yaw angle rotation is minimal when the aspect ratio threshold is triggered.

[0099] Furthermore, according to the definition of aspect ratio, changes in the target's pitch angle directly alter the observation distance between the upper and lower edges of facial features, while the lateral distance of the facial features remains constant, causing the calculated aspect ratio to be smaller than the true value. The deviation threshold then constrains the change in the target face's pitch angle, ensuring that the aspect ratio of the facial features accurately reflects their true state. The deviation threshold can be 10 or 13 to ensure that the target face's pitch angle rotation is minimal when the aspect ratio threshold is triggered.

[0100] The “device with the target face roughly facing the acquired image” mentioned in this application may include a device with the target face completely facing the acquired image, or a device with the target face tilted at an angle less than a preset threshold (i.e., the target face is approximately facing the acquired image).

[0101] S204: Determine that the facial organs are in a closed state.

[0102] When the orientation information determines that the target face is approximately facing the device that acquires an image containing the target face, and the aspect ratio of the facial organs is less than the closure threshold, it can be determined that the facial organs are in a closed state.

[0103] S205: Process facial organ images using an edge-aware network.

[0104] In cases where the target face is not directly facing the device that acquires an image containing the target face based on orientation information, and / or the aspect ratio of the facial organs is greater than or equal to a closure threshold, the facial organ image can be processed using an edge-aware network to obtain a contour map of the facial organs.

[0105] Alternatively, it can be as follows Figure 7 As shown, the edge-aware network may include at least one convolutional layer and an hourglass network connected after the at least one convolutional layer. Each convolutional layer is further connected to a batch normalization (BN) layer. In step S205, the aforementioned edge-aware network takes a facial organ image as input, passes it through two convolutional layers, each followed by a BN layer, and uses ReLU to perform non-linear mapping. Then, low-level features are input into a three-layer hourglass network, and after feature extraction processing such as convolution and average pooling, edges are captured. The edges are then upsampled to the same size as the facial organ image to restore the contours of the facial organs in the original image, thereby obtaining a contour map of the facial organs.

[0106] S206: Overlay the facial organ image and the facial organ contour map to obtain an overlaid image.

[0107] S207: Detect the superimposed images to determine the opening and closing state of facial organs.

[0108] Optionally, a state detection network can be used to detect the superimposed images to determine the open and closed states of facial organs.

[0109] The structure of the state detection network can be as follows: Figure 7 As shown, it includes at least one convolutional layer and a fully connected layer connected after the at least one convolutional layer. Each convolutional layer in the state detection network may be followed by a batch normalization (BN) layer and an average pooling layer.

[0110] In step S207, the aforementioned state detection network takes the superimposed image obtained in step S206 as input, extracts classification features through four convolutional layers, and each convolutional layer is followed by a batch normalization (BN) layer and an average pooling layer, using ReLU as the activation function. Subsequently, the output features are input into a three-layer cascaded fully connected layer to obtain a binary classification result, and SoftMax is used to convert the feature values ​​into probabilities. Then, the open / closed state of facial organs is determined based on the probabilities.

[0111] If the facial organ is the eye, after confirming the open / closed state of the target eye in the current frame image based on the aforementioned facial organ state detection method, it is possible to determine whether the target is blinking based on the current frame image and previous frames. Specifically, as follows... Figure 8 As shown, Figure 8 This is a flowchart illustrating one embodiment of the blink detection method of this application. The blink detection method of this embodiment includes the following steps. It should be noted that the step numbers are for simplification only and are not intended to limit the execution order of the steps. The execution order of each step in this embodiment can be arbitrarily changed without departing from the technical concept of this application.

[0112] S301: Detect the opening and closing state of the target eye in the current frame image based on the above facial organ state detection method.

[0113] S302: Determine whether the target blinks based on the opening and closing state of the target's eyes in the current frame image.

[0114] Optionally, it can be determined whether the target is blinking based on the current frame image and the N frames prior to it.

[0115] Specifically, if it is confirmed based on the current frame image and the previous N frames that the target continuously opened its eyes and then entered a closed-eye state, and subsequently opened its eyes again, then it can be determined that the target blinked.

[0116] This application first determines the opening and closing state of the target's eyes in each frame based on a single frame image, and then determines whether the target blinks based on the opening and closing state of the target's eyes in multiple frames. This avoids using the temporal and texture information of consecutive multiple frames to determine the opening and closing state of the target's eyes in each frame image. It does not impose strict constraints on the temporal or spatial features of the input data and can obtain relatively accurate detection results. Combined with event logic, blinking events can be analyzed accurately and quickly.

[0117] Furthermore, to facilitate the implementation of the above-mentioned facial organ state detection method, this application provides a training method for a facial organ state detection method network, specifically as follows: Figure 9 As shown, Figure 9 This is a flowchart illustrating one embodiment of the training method for the facial organ state detection method network of this application. The training method for the facial organ state detection method network of this embodiment includes the following steps. It should be noted that the step numbers are for simplification only and are not intended to limit the execution order of the steps. The execution order of each step in this embodiment can be arbitrarily changed without departing from the technical concept of this application.

[0118] S401: Obtain a set of facial organ images.

[0119] For any input video frame, the existing real-time face alignment algorithm is used to locate the face region, determine the feature points at the contours of the facial organs, and obtain the facial organ region. At the same time, a line map of the feature points in the closed facial organs is generated. Combined with the labels of open and closed facial organs, an image set of facial organs is constructed.

[0120] The method for generating the line graph of feature points in the facial organ image described above can be as follows: connect the edge feature points in the facial organ image to form a closed edge; generate an edge binarization image with the same size as the facial organ image, with edge pixels set to 1 and other pixels set to 0 based on the closed edge; and align the edge binarization image with the facial organ image to obtain the line graph of feature points in the facial organ image.

[0121] In the image set of facial organs, the size of the facial organ images and / or the line graphs connecting the feature points in the facial organ images can be unified to the same size, for example, 64*64.

[0122] S402: Training an edge perception network and a state detection network based on a set of facial organ images.

[0123] Among them, the edge perception network is used to process facial organ images to obtain facial organ contour maps, and the state detection network is used to detect the superimposed image formed by superimposing facial organ images and facial organ contour maps to determine the open and closed state of facial organs.

[0124] In one implementation, the edge perception network and the state detection network can be jointly trained based on a set of facial organ images.

[0125] In another implementation, the edge perception network and the state detection network can be trained separately based on a set of facial organ images. Specifically, the edge perception network can be trained first, followed by the state detection network.

[0126] Specifically, the edge perception network is first trained based on a set of facial organ images; then, the facial organ images are input into the trained edge perception network to obtain the contour map of the facial organs; the facial organ images and the facial organ contour map are superimposed to obtain a superimposed image; and the state detection network is trained based on the superimposed image.

[0127] During the training of the edge perception network, the training can be supervised by the connection map of feature points in the facial organ image. Specifically, the facial organ image can be input into the edge perception network to obtain the contour map of the facial organ; the first loss is calculated using the contour map and the connection map of feature points in the facial organ image; the parameters of the edge perception network are updated in the direction of decreasing first loss, and the edge perception network is trained using the first loss for pixel-level classification. The trained edge perception network can obtain more accurate edge contour information, which plays an important role in the subsequent classification of open and closed states.

[0128] The formula for calculating the first loss can be:

[0129]

[0130] Where Y + For non-edge pixels, Y - For edge pixels, Y + ∈[0,1],Y - ∈[0,1], β=|Y - | M is the pixel index, Pr is the predicted probability value, x is the input facial organ image, w is the edge-aware network parameter, L k This is the first loss.

[0131] Optionally, the state detection network can be trained under supervised training using facial organ opening and closing states to achieve strong supervision and optimize the network. During training, the superimposed image corresponding to the facial organ image can be input into the network to obtain the detection values ​​of the opening and closing states of the facial organs in the image. Then, a second loss is calculated using the detected values ​​and the true values ​​of the opening and closing states of the facial organs in the image. The parameters of the state detection network are updated in the direction that the second loss decreases to achieve iterative optimization.

[0132] Among them, the second loss L fl The formula for calculating is shown below:

[0133]

[0134] Where γ can be 2, α can be 0.25, y is the true value of the opening and closing state of the facial organs, y=1 indicates that the facial organs are closed, y=0 indicates that the facial organs are open, and Pr is the model prediction result, Pr∈[0,1].

[0135] Taking the eyes as an example of facial organs, the state detection network can be trained based on focal loss. Focal loss is more suitable for the task of detecting open and closed states than cross-entropy loss. In video streams, the number of frames with closed eyes accounts for about 4% to 5% of the total number of frames. Therefore, there will be a problem of imbalance between positive and negative samples during training. The α and γ weights proposed by focal loss can effectively improve the above problem and enable the network to achieve better optimization results.

[0136] In another implementation, the edge perception network and the state detection network can be trained separately based on a set of facial organ images. Then, the separately trained edge perception network and state detection network are jointly trained based on the set of facial organ images. In this way, under the joint supervision of multiple loss signals, the network is iteratively optimized to the preset conditions, and finally, a high-performance edge perception network and state detection network are obtained.

[0137] During joint training, a set of facial organ images can be input into separately trained edge perception networks. Then, the contour map output by the edge perception network and the facial organ images are superimposed on a single channel. The superimposed image is then input into a state detection network to obtain detection values. A first loss and a second loss can be calculated based on the contour map and detection values ​​obtained during joint training. Then, the weighted values ​​of the first and second losses are calculated. The parameters of the edge perception network and the state detection network are updated in the direction that the weighted values ​​decrease. During joint training, the weighting coefficients of the first and second losses remain constant, but they can be set according to specific circumstances; for example, the weighting coefficient of the first loss could be 0.3, and the weighting coefficient of the second loss could be 0.7.

[0138] This application utilizes multi-loss joint supervision, which not only helps to express low-level edge features but also fully extracts high-dimensional semantic information, thus enabling the network to achieve better optimization results.

[0139] Please see Figure 10 , Figure 10 This is a schematic diagram of one embodiment of the electronic device 20 of this application. The electronic device 20 of this application includes a processor 22, which is used to execute instructions to implement the methods provided by any of the above embodiments of this application and any non-conflicting combinations thereof.

[0140] Electronic device 20 can be a mobile phone, laptop computer or other terminal, or it can also be a server.

[0141] Processor 22 can also be referred to as CPU (Central Processing Unit). Processor 22 may be an integrated circuit chip with signal processing capabilities. Processor 22 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. A general-purpose processor can be a microprocessor, or processor 22 can be any conventional processor, etc.

[0142] The electronic device 20 may further include a memory 21 for storing instructions and data required for the processor 22 to run.

[0143] Please see Figure 11 , Figure 11This is a schematic diagram of the structure of a computer-readable storage medium in an embodiment of this application. The computer-readable storage medium 30 in this embodiment stores instruction / program data 31. When executed, this instruction / program data 31 implements the methods provided in any embodiment of the above-described method of this application, as well as any non-conflicting combination thereof. The instruction / program data 31 can be formed into a program file and stored in the storage medium 30 in the form of a software product, so that a computer device (which may be a personal computer, server, or network device, etc.) or processor can execute all or part of the steps of the methods in various embodiments of this application. The aforementioned storage medium 30 includes various media capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or devices such as computers, servers, mobile phones, and tablets.

[0144] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0145] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0146] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0147] The above are merely embodiments of this application and do not limit the scope of this patent application. Any equivalent structural or procedural changes made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of this application.

Claims

1. A method for detecting the state of facial organs, characterized in that, The method includes: Extract facial organ images of the target from an image containing the target face; Based on the image containing the target face, determine the orientation information of the target face and calculate the aspect ratio of the facial organs; If, based on the orientation information, it is determined that the target face is approximately facing the device that acquires the image containing the target face, and the aspect ratio of the facial organ is less than the closure threshold, it is determined that the facial organ is in a closed state. Otherwise, the facial organ images are processed using an edge-aware network to obtain the contour maps of the facial organs; The facial organ image and the outline of the facial organ are superimposed to obtain a superimposed image; The superimposed image is detected to determine the open and closed state of the facial organs.

2. The method for detecting the state of facial organs according to claim 1, wherein the facial organ is the target's eyes, characterized in that... The steps of determining the orientation information of the target face and calculating the aspect ratio of the facial organs based on the image containing the target face include: Based on the image containing the target's face, the difference in aspect ratio between the target's left and right eyes is calculated, and the degree to which the tip of the target's nose deviates from the center of the target's face is calculated. The difference and the degree are used as the orientation information. The step of determining that the facial features are in a closed state when the orientation information determines that the target face is approximately facing the device that acquires the image containing the target face, and the aspect ratio of the facial features is less than a closure threshold, includes: If the degree to which the tip of the nose of the target deviates from the center of the target face is less than the deviation threshold, the difference is less than the error threshold, and the aspect ratio of the facial organ is less than the closure threshold, then the facial organ is determined to be in a closed state.

3. The method for detecting the state of facial organs according to claim 1, characterized in that, The facial organ images of the target are extracted based on the image containing the target face: Extract facial feature points from the image containing the target face; The center points of facial organs are determined based on the feature points of the face. Using the center point of the facial organ as the center and the width of the facial organ as the side length, the image of the facial organ is extracted from the image containing the target face.

4. The method for detecting the state of facial organs according to claim 1, characterized in that, The step of overlaying the facial organ image and the facial organ contour map to obtain an overlaid image includes: The facial organ image and the outline of the facial organ are superimposed on the channel to obtain the superimposed image.

5. A method for training a facial organ state detection network, characterized in that, The method includes: Obtain a set of facial organ images; An edge perception network and a state detection network were trained based on a set of facial organ images; The edge-aware network is used to process facial organ images to obtain a contour map of the facial organs when the orientation information of the target face and the aspect ratio of the facial organs cannot confirm that the facial organs are in a closed state; when the orientation information determines that the target face is approximately facing the device that acquires the facial organ images, and the aspect ratio of the facial organs is less than a closure threshold, the facial organs are determined to be in a closed state; the state detection network is used to detect the superimposed image formed by superimposing the facial organ images and the facial organ contour maps to determine the open / closed state of the facial organs.

6. The training method according to claim 5, characterized in that, The steps of training the edge-aware network and the state detection network based on the facial organ image set include: An edge-aware network was trained using a set of facial organ images. The facial organ images are input into the trained edge perception network to obtain the contour map of the facial organs; The superimposed image is obtained by overlaying the facial organ image and the outline of the facial organ; The state detection network is trained based on the overlaid image.

7. The training method according to claim 6, characterized in that, The steps for training the edge-aware network based on a set of facial organ images include: The facial organ images are input into an edge-aware network to obtain the contour maps of the facial organs; The first loss is calculated using the connection graph of feature points in the contour map and the facial organ image; The parameters of the edge-aware network are updated in the direction in which the first loss is reduced.

8. The training method according to claim 6, characterized in that, The step of training the state detection network based on the overlaid image: The superimposed image corresponding to the facial organ image is input into the state detection network to obtain the detection value of the open and closed state of the facial organs in the facial organ image; The second loss is calculated using the detected and true values ​​of the open and closed states of the facial organs in the facial organ image; The parameters of the state detection network are updated in the direction of the second loss reduction.

9. The training method according to claim 6, characterized in that, The step of training the state detection network based on the overlaid image is followed by: The edge perception network and the state detection network are jointly trained based on the facial organ image set.

10. The training method according to claim 9, characterized in that, The step of jointly training the edge perception network and the state detection network based on the facial organ image set includes: During joint training, a first loss and a second loss are calculated, and a weighted value of the first loss and the second loss is calculated. In the direction in which the weighted value decreases, the parameters of the edge perception network and the state detection network are updated.

11. A blink detection method, characterized in that, The method includes: The facial organ state detection method according to any one of claims 1-4 is used to detect the opening and closing state of the target eye in the current frame image; Based on the opening and closing state of the target's eyes in the current frame image, determine whether the target is blinking.

12. An electronic device, characterized in that, The electronic device includes a processor; the processor is configured to execute instructions to implement the steps of the method as described in any one of claims 1-11.

13. A computer-readable storage medium having a program and / or instructions stored thereon, characterized in that, When the program and / or instructions are executed, they implement the steps of the method according to any one of claims 1-11.

Images