A method, apparatus and storage medium for blink detection

Abstract

This application relates to the field of computers and discloses a method, apparatus, and storage medium for blink detection. The blink detection method is applied to a computing device and includes: extracting temporal and spatial features from multiple image groups containing human eyes in a video stream, wherein each image group includes at least three consecutive images, and the order of the at least three consecutive images in the image group is consistent with the order of the at least three consecutive images in the video stream; based on this, performing a three-dimensional convolution operation on the extracted temporal and spatial features of each image group to obtain a predicted probability value of blinking behavior in the video stream, and determining whether blinking behavior has occurred based on the predicted probability value. In the process of performing a three-dimensional convolution operation on multiple consecutive images, the continuous change of blinking action in time is obtained. At the same time, the spatial features of human eyes are combined, thereby improving the performance of the operation and enhancing the accuracy of blinking behavior detection.

Description

Technical Field

[0001] This application relates to computer technology, and in particular to a method, apparatus and storage medium for blink detection. Background Technology

[0002] Blinking is a normal physiological response, and it is difficult to avoid eye strain. Because blinking reflects a positive interaction between a person and the outside world, it has been widely applied in various scenarios, including liveness detection, driver fatigue monitoring, micro-expression analysis, and lie detection systems, thanks to the development of intelligent technology.

[0003] Currently, commonly used blink detection methods in the industry are mainly divided into three types: blink detection methods based on sensing devices, blink detection methods based on single-frame spatial features, and blink detection methods based on multi-frame temporal weighted fusion. However, in practice, all three blink detection methods have drawbacks, as follows:

[0004] The main drawbacks of blink detection methods based on sensing devices are as follows: They require multiple sensors or chips to capture voltage values ​​at sampling points in the human eye region, making it difficult to extend their application to other general scenarios. This method not only increases application costs but is also susceptible to noise interference, which in turn affects algorithm performance.

[0005] The main drawbacks of blink detection methods based on single-frame spatial features are as follows: Since single-frame images lack temporal information, there is an inherent loss of information for behavior recognition. Localization of existing feature points remains challenging, making error propagation and misjudgment likely. Furthermore, the detection process requires extensive logical judgments, making its implementation relatively complex.

[0006] The main drawbacks of blink detection methods based on multi-frame temporal weighted fusion are as follows: Long-Short Term Memory (LSTM) artificial neural networks are often used in the detection process. LSTM networks have high complexity, and the accuracy of blink detection results is limited. Furthermore, the training complexity of multi-frame LSTM-based methods is high, easily leading to the introduction of hyperparameter modules, which exacerbates the training difficulty and model convergence. Additionally, LSTM is susceptible to the influence of historical input states.

[0007] In summary, existing blink detection methods have various drawbacks, resulting in poor accuracy of blink detection results obtained using these methods. Summary of the Invention

[0008] This application provides a method, apparatus, and storage medium for blink detection, in order to improve the accuracy of blink detection results.

[0009] The specific technical solution provided in this application is as follows:

[0010] Firstly, a blink detection method, applied to a computing device, includes:

[0011] Feature extraction in the temporal and spatial domains is performed on multiple image groups containing human eyes in a video stream, wherein the image group includes at least three consecutive images, and the order of the at least three consecutive images in the image group is consistent with the order of the at least three consecutive images in the video stream.

[0012] A 3D convolution operation is performed on the temporal and spatial features extracted from each image group to obtain the predicted probability value of the video stream containing blinking behavior, and the blinking behavior is determined based on the predicted probability value.

[0013] Optionally, before performing temporal and spatial feature extraction on the image group containing the human eye in the video stream, the method further includes:

[0014] Extract the sequence number of at least three consecutive frames containing the human eye from the video stream;

[0015] The face is aligned by aligning at least three consecutive frames of images containing human eyes in the video stream, and the area where the human eyes are located is extracted from the at least three consecutive frames of aligned images using pre-selection boxes.

[0016] The area where the human eye is located is converted to grayscale. The grayscale areas where the human eye is located are then arranged in order of their sequence numbers to obtain an image group.

[0017] Optionally, before performing temporal and spatial feature extraction on the image group containing the human eye in the video stream, the method further includes:

[0018] Extracting multiple consecutive frames of images containing human eye views from a video stream;

[0019] Divide multiple consecutive frames of images into multiple image groups, wherein each pair of image groups includes at least three consecutive frames of images that are partially repeated or not repeated.

[0020] Optionally, temporal and spatial feature extraction is performed on multiple image groups containing human eyes in the video stream, including:

[0021] Perform the following operations for each group of images:

[0022] From at least three frames of images containing human eyes in a video stream, temporal features representing human eye movements are extracted, where human eye movements include opening and closing of the same part of the eye; and

[0023] Extract spatial texture features representing the location of the human eye from at least three frames of images containing the human eye in the video stream.

[0024] Optionally, a 3D convolution operation is performed on the temporal and spatial features extracted from each image group to obtain the predicted probability value of the video stream containing blinking behavior, including:

[0025] For each image group, perform the following operation: use a preset 3D convolution kernel to perform convolution operations on the temporal features and spatial texture features extracted from the image group;

[0026] For each image group, the following steps are performed: sum the results of each convolution operation to obtain the predicted probability values ​​of blinking behavior for multiple image groups.

[0027] Optionally, determining whether blinking has occurred based on the predicted probability value includes:

[0028] Compare whether the predicted probability value is greater than the preset blink threshold;

[0029] If the predicted probability value is greater than the preset blink threshold, then the blinking behavior is determined to have occurred.

[0030] If the predicted probability value is not greater than the preset blink threshold, then it is determined that no blinking behavior has occurred.

[0031] Secondly, a blink detection device includes:

[0032] The feature extraction unit is used to extract features in the temporal and spatial domains from multiple image groups containing human eyes in the video stream, wherein the image group includes at least three consecutive images, and the arrangement order of the at least three consecutive images in the image group is consistent with the chronological order of the at least three consecutive images in the video stream.

[0033] The convolution operation unit is used to perform three-dimensional convolution operations on the temporal and spatial features extracted from each image group to obtain the predicted probability value of the video stream containing blinking behavior, and to determine whether blinking behavior has occurred based on the predicted probability value.

[0034] Optionally, before performing temporal and spatial feature extraction on the image group containing the human eye in the video stream, an image group acquisition unit is further included, which is used to:

[0035] Extract the sequence number of at least three consecutive frames containing the human eye from the video stream;

[0036] The face is aligned by aligning at least three consecutive frames of images containing human eyes in the video stream, and the area where the human eyes are located is extracted from the at least three consecutive frames of aligned images using pre-selection boxes.

[0037] The area where the human eye is located is converted to grayscale. The grayscale areas where the human eye is located are then arranged in order of their sequence numbers to obtain an image group.

[0038] Optionally, before performing temporal and spatial feature extraction on the image group containing the human eye in the video stream, an image group partitioning unit is further included, which is used for:

[0039] Extracting multiple consecutive frames of images containing human eye views from a video stream;

[0040] Divide multiple consecutive frames of images into multiple image groups, wherein each pair of image groups includes at least three consecutive frames of images that are partially repeated or not repeated.

[0041] Optionally, temporal and spatial feature extraction is performed on multiple image groups containing human eyes in the video stream. The feature extraction unit is used for:

[0042] Perform the following operations for each group of images:

[0043] From at least three frames of images containing human eyes in a video stream, temporal features representing human eye movements are extracted, where human eye movements include opening and closing of the same part of the eye; and

[0044] Extract spatial texture features representing the location of the human eye from at least three frames of images containing the human eye in the video stream.

[0045] Optionally, a 3D convolution operation is performed on the temporal and spatial features extracted from each image group to obtain the predicted probability value of the video stream containing blinking behavior. The convolution operation unit is used for:

[0046] For each image group, perform the following operation: use a preset 3D convolution kernel to perform convolution operations on the temporal features and spatial texture features extracted from the image group;

[0047] For each image group, the following steps are performed: sum the results of each convolution operation to obtain the predicted probability values ​​of blinking behavior for multiple image groups.

[0048] Optionally, based on the predicted probability value, the convolution operation unit is used to determine whether a blinking behavior has occurred.

[0049] Compare whether the predicted probability value is greater than the preset blink threshold;

[0050] If the predicted probability value is greater than the preset blink threshold, then the blinking behavior is determined to have occurred.

[0051] If the predicted probability value is not greater than the preset blink threshold, then it is determined that no blinking behavior has occurred.

[0052] Thirdly, a terminal includes:

[0053] Memory, used to store executable instructions;

[0054] A processor for reading and executing executable instructions stored in memory to implement the method as described in any of the first aspects.

[0055] Fourthly, a computer-readable storage medium, when instructions in the storage medium are executed by a processor, enables the processor to perform the method described in any of the first aspects above.

[0056] In summary, this application discloses a blink detection method applied to a computing device, comprising: extracting temporal and spatial features from multiple image groups containing human eyes in a video stream, wherein each image group includes at least three consecutive images, and the order of the at least three consecutive images in the image group is consistent with the order of the at least three consecutive images in the video stream; based on this, performing a three-dimensional convolution operation on the extracted temporal and spatial features of each image group to obtain a predicted probability value of blinking behavior in the video stream, and determining whether blinking behavior has occurred based on the predicted probability value. In the process of performing a three-dimensional convolution operation on multiple consecutive images, the continuous change of blinking action in time is obtained, and at the same time, the spatial features of human eyes are combined, thereby making the computation performance more superior and improving the accuracy of blinking behavior detection. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the system architecture for detecting blinking behavior in an embodiment of this application;

[0058] Figure 2 This is a schematic diagram illustrating the process of a computing device detecting blinking behavior in an embodiment of this application.

[0059] Figure 3 This is a schematic diagram of a computing device acquiring an image group in an embodiment of this application;

[0060] Figure 4 This is a schematic diagram illustrating the process of acquiring an image group using another computing device in an embodiment of this application;

[0061] Figure 5 This is a schematic diagram of the feature extraction process performed by the computing device in the embodiments of this application;

[0062] Figure 6 This is a flowchart illustrating the two-dimensional convolution operation in the prior art;

[0063] Figure 7 This is a schematic diagram illustrating the process of a computing device performing a three-dimensional convolution operation in an embodiment of this application;

[0064] Figure 8 This is a schematic diagram illustrating the acquisition of multiple feature points of the human eye using a face alignment algorithm in an embodiment of this application;

[0065] Figure 9 This is a schematic diagram of the dynamic changes in the human eye image used to determine whether blinking has occurred, as shown in the embodiments of this application.

[0066] Figure 10 This is a schematic diagram of a computing device performing a three-dimensional convolution operation in an embodiment of this application;

[0067] Figure 11a This is a schematic diagram of the logical architecture of a device according to an embodiment of this application;

[0068] Figure 11b This is a schematic diagram of the logical architecture of another device in an embodiment of this application;

[0069] Figure 12 This is a schematic diagram of the physical architecture of the terminal in the embodiments of this application. Detailed Implementation

[0070] To improve the accuracy of blink detection, in this embodiment, the computing device extracts temporal and spatial features from multiple image groups containing human eyes in the video stream. Each image group includes at least three consecutive frames, and the order of these three consecutive frames in the image group corresponds to their chronological order in the video stream. Based on this, a three-dimensional convolution operation is performed on the extracted temporal and spatial features of each image group to obtain a predicted probability value for the video stream containing blinking behavior. The predicted probability value is then used to determine whether blinking behavior has occurred. During the three-dimensional convolution operation on multiple consecutive frames, the continuous temporal changes of blinking are captured. Simultaneously, the spatial features of the human eye are incorporated, resulting in superior computational performance and improved accuracy of blink detection.

[0071] The preferred embodiments of this application will now be described in detail with reference to the accompanying drawings. Firstly, in the embodiments of this application, the system includes at least a video stream provider and a computing device. (See attached figures...) Figure 1 As shown, the video stream provider is the device that provides the video stream to the computing device. Common video stream providers include cameras, video recorders, and computers. The computing device is the main entity in this system that performs blink detection; that is, after the video stream provider sends the video stream to the computing device, the computing device performs blink detection on the video stream. Of course, if the computing device itself has video stream acquisition capabilities, then during the execution process, the computing device can directly perform blink detection on the acquired video stream.

[0072] The following describes a method for performing blink detection using a computing device. (See also...) Figure 2 As shown in the embodiments of this application, the specific process of the computing device performing blink detection is as follows:

[0073] Step 201: Perform temporal and spatial feature extraction on multiple image groups containing human eyes in the video stream, wherein the image group includes at least three consecutive images, and the arrangement order of the at least three consecutive images in the image group is consistent with the chronological order of the at least three consecutive images in the video stream.

[0074] Since a video stream consists of multiple consecutive video frames, each of which contains an image, in the implementation of this application, since it is necessary to detect human eye behavior (i.e., blinking or not blinking), it is first necessary to determine that the video stream to be detected contains multiple groups of images of human eyes, and then further extract features in the temporal and spatial domains for each group of images.

[0075] It should be noted that before performing temporal and spatial feature extraction on the image set containing the human eye in the video stream, the image set must first be obtained. (See [link to relevant documentation]). Figure 3 As shown, it specifically includes:

[0076] Step 2001: Extract the sequence number of at least three consecutive frames containing the human eye from the video stream.

[0077] To ensure the temporal authenticity of the image group, during implementation, the sequence numbers of consecutive images containing human eyes are extracted from the video stream. For example, in case (1), the sequence numbers of consecutive images 1, 2, 3, 4... are extracted sequentially from the video stream; in case (2), the sequence numbers of consecutive images extracted sequentially from the video stream can also be 1, 3, 5, 6..., but in order to ensure the accuracy of human eye detection, during implementation, the sequence numbers of the extracted images are mainly those in case (1) above.

[0078] Furthermore, to comprehensively consider the continuous temporal changes in blinking and the spatial characteristics of the human eye, the aforementioned image set must contain at least three frames. Specifically, the temporal characteristics of the image set are characterized by the temporal changes in the human eye region between the first and second frames, and between the second and third frames. Simultaneously, the morphology of the human eye in each frame is extracted to characterize the spatial characteristics of the image set.

[0079] Step 2002: Align the face of at least three consecutive frames containing the human eye in the video stream, and use pre-selection boxes to extract the area where the human eye is located from the at least three consecutive frames after alignment.

[0080] After acquiring at least three consecutive frames of images, face alignment is performed on these at least three consecutive frames. For example, the dlib face feature point extraction algorithm is used to align the faces in the images, so that the positions of the eyes in the images are relatively fixed.

[0081] Considering that the proportion of a face in an image may vary, during implementation, after aligning the face in the image, a pre-selection box is used to extract the area where the eyes are located from at least three consecutive frames of the aligned image. The size of the pre-selection box can be flexibly set according to the actual use scenario. The principle is that the pre-selection box can cover the entire eye. In this way, the actions of opening and closing the eyes can be accurately identified, thereby further improving the accuracy of blink detection.

[0082] Step 2003: Convert the area where the human eye is located to grayscale, and arrange the grayscale regions of the human eye in the order of their sequence numbers to obtain an image group.

[0083] Given that normal images in existing video streams are color images, meaning they include R, G, and B channels, convolution operations would require more dimensional convolution kernels. However, blinking is unrelated to image color. Therefore, in practice, the region where the human eye is located is converted to grayscale, meaning all images in the video stream are converted to single-channel images to reduce the computational load of convolution operations.

[0084] To maintain the continuity of the images in the image group, during the process, the region where the human eye is located after grayscale processing is arranged in the order of the extracted sequence numbers to obtain the image group for convolution operation.

[0085] Additionally, it's necessary to explain the source of the image groups here, i.e., the process of acquiring multiple image groups from a video stream. Specifically, before performing temporal and spatial feature extraction on the image groups containing human eyes in the video stream, please refer to [link to relevant documentation]. Figure 4 As shown, it also includes:

[0086] Step 2001': Extract multiple consecutive frames of images containing human eyes from the video stream.

[0087] Since video frames are arranged sequentially in the video stream according to the time sequence, during the implementation process, multiple consecutive images containing human eyes are extracted from the video stream according to the order of the video frames. For example, after determining that the starting frame in the video stream is the first frame image, multiple consecutive images containing human eyes are extracted from the video stream, namely the first frame image, the second frame image, ... the Nth frame image, ... the last frame image.

[0088] Step 2002': Divide the multiple consecutive frames of images into multiple image groups, wherein each pair of image groups includes at least three consecutive frames of images that are either partially repeated or not repeated.

[0089] Since blinking is a continuous action of the human eye, the implementation process requires dividing multiple consecutive images into multiple image groups. Furthermore, each pair of image groups must contain at least three consecutive images that are partially identical or not identical.

[0090] The first scenario: at least three consecutive frames of images are repeated in every two image groups. For example, when a total of 15 frames are acquired in step 2001', the three consecutive frames in these 15 frames are divided into one image group, that is, the first three frames form image group 1, the third to fifth frames form image group 2, the fifth to seventh frames form image group 3, and so on, with the thirteenth to fifteenth frames forming image group 7.

[0091] The second scenario: Each pair of image groups includes at least three consecutive non-repeating images. For example, when a total of 15 images are acquired in step 2001', the three consecutive images in each of these 15 images are divided into one image group, that is, the first three images form image group 1, the fourth to sixth images form image group 2, and so on, with the thirteenth to fifteenth images forming image group 5.

[0092] During implementation, after acquiring multiple image groups, temporal and spatial feature extraction is performed on the multiple image groups containing human eyes in the video stream. (See [reference needed]). Figure 5 As shown, it includes:

[0093] Step 2011: Perform the following operations for each image group: extract temporal features representing human eye movements from at least three frames of images containing human eyes in the video stream, wherein human eye movements include eye-opening movements and eye-closing movements of the same part of the human eye.

[0094] To accurately describe blinking, the human eye is divided into multiple parts, specifically multiple points selected on both the upper and lower eyelids. The distance between these points differs significantly during eye-opening and eye-closing movements. For example, when the eyes are open, the distance between the midpoints of the upper and lower eyelids is 3 centimeters; when the eyes are closed, the distance is 0.2 centimeters.

[0095] During implementation, for each image group, temporal features representing human eye movements are extracted from its three frames, that is, the opening and closing movements of the same part of the human eye are determined.

[0096] as well as

[0097] Step 2012: For each group of images, perform the following operation: extract spatial texture features representing the location of the human eye from at least three frames of images in the video stream that contain the human eye.

[0098] During implementation, to accurately describe whether the shape of the human eye conforms to the blinking behavior, the following operations are performed for each image group: Spatial texture features representing the position of the human eye are extracted from at least three frames in the video stream containing the human eye. That is, morphological information of the human eye is extracted from each image group. For example, the spatial texture feature representing the position of the human eye in the eye-opening action is an ellipse, while the spatial texture feature representing the position of the human eye in the eye-closing action is a straight line.

[0099] A blinking action can only be determined to have occurred when multiple image groups containing human eyes in the video stream detect both eye-opening and eye-closing actions, and when the aforementioned spatial texture features also conform to a preset pattern.

[0100] Step 202: Perform a 3D convolution operation on the temporal and spatial features extracted from each image group to obtain the predicted probability value of the video stream containing blinking behavior, and determine whether blinking behavior has occurred based on the predicted probability value.

[0101] See Figure 6 As shown, the number of types of 2D convolutional kernels equals the number of dimensions of the output features, and the number of kernels of the same type equals the number of dimensions of the input features. During convolution, each convolutional kernel performs convolution on the input feature map in each dimension using a sliding window, thereby extracting the texture features of the area covered by the convolutional kernel in space. Since each dimension of the input features represents a frame of human eye image in the video stream, a single convolutional kernel can only extract feature information within the receptive field with the same kernel parameters. Therefore, the 2D convolution process is very effective for spatial features, but almost completely loses temporal information. Unlike existing multi-frame temporal weighted fusion blink detection methods (e.g., LSTM) that use 2D convolution operations, this embodiment uses 3D convolution operations.

[0102] Preferably, before performing the 3D convolution operation, the process further includes training the neural network performing the 3D convolution operation using a set of human eye images from the video stream. Specifically, multiple local features and one global feature are extracted from each set of human eye images. The local features are used to represent the feature information of the same part of the human eye in adjacent frames, and the global feature is used to represent the spatial texture features of the human eye. Each local feature and each global feature are input into the blink detection network for training. If the training result is that the probability of blinking occurs is 1 and the probability of blinking does not occur is 0, then the training process is stopped, and the trained neural network performing the 3D convolution operation is output.

[0103] The above-described 3D convolution operation is performed on the temporal and spatial features extracted from each image group to obtain the predicted probability value of the video stream containing blinking behavior. (See [link to relevant documentation]). Figure 7 As shown, it includes:

[0104] Step 2021: For each image group, perform convolution operations on the temporal features and spatial texture features extracted from the image group using a preset 3D convolution kernel.

[0105] During implementation, the preset 3D convolution kernel extracts and combines the same local feature information of at least three consecutive frames of images through a low-level feature layer to capture the temporal feature changes. At the same time, the preset 3D convolution kernel performs convolution operation on the spatial texture features of each image group using feature fusion, effectively passing the temporal and spatial features down to achieve spatiotemporal feature perception from local to global. That is, each image group obtains a convolution operation result after the 3D convolution kernel slides, and the value range of the result is 0-1.

[0106] Step 2022: Perform the following for each image group: sum the results of each convolution operation to obtain the predicted probability values ​​of blinking behavior for multiple image groups.

[0107] During implementation, after obtaining the convolution operation results for each image group, the following steps are performed for each image group: summing the results of each convolution operation to obtain the predicted probability values ​​of blinking behavior for multiple image groups. In other words, the results of each convolution operation are sequentially superimposed to finally obtain the predicted probability values ​​of blinking behavior for multiple image groups.

[0108] After obtaining the specific predicted probability value, the system determines whether blinking has occurred based on the predicted probability value, specifically including:

[0109] During implementation, the predicted probability value is compared to a preset blink threshold. Here, the preset blink threshold is the minimum probability value that can be determined as a blinking behavior based on historical data. The occurrence of a blinking behavior is determined by comparing the predicted probability value with the preset blink threshold.

[0110] Case 1): If the predicted probability value is greater than the preset blink threshold, then the blinking behavior is determined to have occurred.

[0111] If the predicted probability value obtained above is greater than the preset blink threshold, it is considered that the occurrence of eye-opening and eye-closing actions has been detected in multiple consecutive image groups in the video stream, thereby determining that blinking behavior has occurred.

[0112] Case 2): If the predicted probability value is not greater than the preset blink threshold, then it is determined that no blinking behavior has occurred.

[0113] If the predicted probability value obtained above is not greater than the preset blink threshold, it is considered that no eye-opening or eye-closing actions were detected in multiple consecutive image groups in the video stream, and therefore it is determined that no blinking behavior occurred.

[0114] The above embodiments will be further described in detail below using a specific application scenario.

[0115] Application Scenario 1:

[0116] See Figure 8 As shown, the existing open-source Dlib library is used to obtain the coordinates of 68 facial feature points of each image in the video stream in real time. These points include semantically informational points in the facial contour, eyebrows, eyes, nose, and mouth regions. Then, through rigid transformations such as rotation and translation, all facial images are transformed onto the template face, thereby achieving face alignment. Subsequently, based on the coordinates of the feature points at the left and right corners of the eyes, the Euclidean distance is calculated as the width of the eye, and the average coordinates of the left and right corners of the eyes are calculated as the center point of the eye. Then, using the center point of the eye as a reference, a square area with a side length equal to the width of the eye is cropped using a pre-selected bounding box as the eye image. Finally, the cropped eye image is converted to grayscale.

[0117] The grayscale data of the human eye in 16 consecutive frames following any starting frame are stacked along the channel to obtain multiple image groups. For details, please refer to [link to documentation]. Figure 9 As shown, the encoding methods for human eye changes (i.e., including eye opening and closing actions) are illustrated in three common scenarios. Scenario 0 shows the eye state changing from open to closed and back to open from left to right; for this type of video stream, blinking is determined to have occurred. Scenario 1 and 2 show two sequence states: fully open and fully closed, respectively; for this type of video stream, blinking is determined not to have occurred. Subsequently, the image group consisting of the 16 grayscale frames stacked across channels is input into the neural network for 3D convolution.

[0118] See Figure 10 As shown, the 3*3*3 grid represents the three-dimensional convolution kernel. Compared to the two-dimensional convolution kernel, the number of types of three-dimensional convolution kernels is three times that of two-dimensional convolution kernels. Thus, during the convolution process, each three-dimensional convolution kernel performs convolution simultaneously on the three dimensions of at least three consecutive frames of the image group. The convolution parameter values ​​used on different dimensions can be trained. Therefore, on a single input feature layer, the three-dimensional convolution kernel can efficiently extract the spatial texture features of the human eye position. On the input feature layer of at least three consecutive frames, the three-dimensional convolution kernel can express the changes in features with the same semantics in three adjacent frames, that is, the temporal features representing human eye movements. Specifically, human eye movements include the opening and closing of the same part of the human eye.

[0119] After a 3D convolutional layer completes a 3D convolution operation, each pixel value in the output feature map of the neural network can represent the spatiotemporal feature information of at least three consecutive frames of human eye images. Then, the second 3D convolutional layer completes the 3D convolution operation in the same way, which can represent the spatiotemporal feature information of nine adjacent frames of human eye images. As the 3D convolutional layers deepen, the spatiotemporal fusion network can accurately express the spatial texture characteristics and temporal change characteristics in the entire video stream. That is, the results of each convolution operation are summed to obtain the predicted probability value of blinking behavior corresponding to multiple image groups, which is used as the basis for judging the occurrence of blinking behavior. That is, if the predicted probability value is greater than the preset blinking threshold, it is determined that blinking behavior has occurred.

[0120] Based on the same inventive concept, see [reference] Figure 11a and Figure 11b As shown in the figure, this application provides a blink detection device, including:

[0121] The feature extraction unit 1110 is used to extract features in the temporal and spatial domains from multiple image groups containing human eyes in the video stream, wherein the image group includes at least three consecutive images, and the arrangement order of the at least three consecutive images in the image group is consistent with the chronological order of the at least three consecutive images in the video stream.

[0122] The convolution operation unit 1120 is used to perform three-dimensional convolution operations on the temporal features and spatial features extracted from each image group to obtain the predicted probability value of the video stream containing blinking behavior, and to determine whether blinking behavior has occurred based on the predicted probability value.

[0123] Optionally, before performing temporal and spatial feature extraction on the image group containing the human eye in the video stream, an image group acquisition unit 1100 is further included. The image group acquisition unit 1100 is used for:

[0124] Extract the sequence number of at least three consecutive frames containing the human eye from the video stream;

[0125] The face is aligned by aligning at least three consecutive frames of images containing human eyes in the video stream, and the area where the human eyes are located is extracted from the at least three consecutive frames of aligned images using pre-selection boxes.

[0126] The area where the human eye is located is converted to grayscale. The grayscale areas where the human eye is located are then arranged in order of their sequence numbers to obtain an image group.

[0127] Optionally, before performing temporal and spatial feature extraction on the image group containing the human eye in the video stream, an image group segmentation unit 1100' is further included, which is used for:

[0128] Extracting multiple consecutive frames of images containing human eye views from a video stream;

[0129] Divide multiple consecutive frames of images into multiple image groups, wherein each pair of image groups includes at least three consecutive frames of images that are partially repeated or not repeated.

[0130] Optionally, feature extraction is performed on multiple image groups containing human eyes in the video stream in both temporal and spatial domains. The feature extraction unit 1110 is used for:

[0131] Perform the following operations for each group of images:

[0132] From at least three frames of images containing human eyes in a video stream, temporal features representing human eye movements are extracted, where human eye movements include opening and closing of the same part of the eye; and

[0133] Extract spatial texture features representing the location of the human eye from at least three frames of images containing the human eye in the video stream.

[0134] Optionally, a three-dimensional convolution operation is performed on the temporal and spatial features extracted from each image group to obtain the predicted probability value of the video stream containing blinking behavior. The convolution operation unit 1120 is used for:

[0135] For each image group, perform the following operation: use a preset 3D convolution kernel to perform convolution operations on the temporal features and spatial texture features extracted from the image group;

[0136] For each image group, the following steps are performed: sum the results of each convolution operation to obtain the predicted probability values ​​of blinking behavior for multiple image groups.

[0137] Optionally, based on the predicted probability value, the convolution operation unit 1120 is used to determine whether a blinking behavior has occurred, and to perform the following operations:

[0138] Compare whether the predicted probability value is greater than the preset blink threshold;

[0139] If the predicted probability value is greater than the preset blink threshold, then the blinking behavior is determined to have occurred.

[0140] If the predicted probability value is not greater than the preset blink threshold, then it is determined that no blinking behavior has occurred.

[0141] Based on the same inventive concept, see [reference] Figure 12 As shown, this application embodiment provides a terminal, including:

[0142] Memory 1201 is used to store executable instructions;

[0143] Processor 1202 is configured to read and execute executable instructions stored in memory to implement the method as described in any of the first aspects.

[0144] Among them, Figure 12 In this context, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 1202) and memory (memory 1201). The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver may be multiple components, including transmitters and transceivers, providing a unit for communicating with various other devices over a transmission medium. Processor 1202 is responsible for managing the bus architecture and general processing, and memory 1201 may store data used by processor 1202 during operation.

[0145] The processor 1202 is responsible for managing the bus architecture and general processing, while the memory 1201 can store the data used by the processor 1200 when performing operations.

[0146] The memory 1201 and processor 1202 cooperate with each other to implement any one of the methods executed by the computing device in steps 201-202 of the above embodiments, which will not be described in detail here.

[0147] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium that, when the instructions in the storage medium are executed by a processor, enables the processor to perform the method described in any of the first aspects above.

[0148] In summary, this application discloses a blink detection method applied to a computing device, comprising: extracting temporal and spatial features from multiple image groups containing human eyes in a video stream, wherein each image group includes at least three consecutive images, and the order of the at least three consecutive images in the image group is consistent with the order of the at least three consecutive images in the video stream; based on this, performing a three-dimensional convolution operation on the extracted temporal and spatial features of each image group to obtain a predicted probability value of blinking behavior in the video stream, and determining whether blinking behavior has occurred based on the predicted probability value. In the process of performing a three-dimensional convolution operation on multiple consecutive images, the continuous change of blinking action in time is obtained, and at the same time, the spatial features of human eyes are combined, thereby making the computation performance more superior and improving the accuracy of blinking behavior detection.

[0149] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program product systems. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product system implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0150] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program product systems according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0151] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0152] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0153] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for blink detection, characterized in that, Applied to a computing device, the method includes: The first three-dimensional convolutional layer in the neural network is used to extract features in the temporal and spatial domains from multiple image groups containing human eyes in the video stream. The image group includes at least three consecutive images, and the order of the at least three consecutive images in the image group is consistent with the order of the at least three consecutive images in the video stream. The neural network includes multiple three-dimensional convolutional layers. The temporal and spatial features extracted from each image group are subjected to three-dimensional convolution operations using the three-dimensional convolutional layers after the first three-dimensional convolutional layer to obtain the predicted probability value of the video stream containing blinking behavior, and the blinking behavior is determined based on the predicted probability value. The step of performing a three-dimensional convolution operation on the temporal and spatial features extracted from each image group to obtain the predicted probability value of the video stream containing blinking behavior includes: For each image group, the following steps are performed: using a preset three-dimensional convolution kernel to extract and combine the same local feature information of at least three consecutive images through a low-level feature layer to capture the temporal feature changes, and using a preset three-dimensional convolution kernel to perform convolution operation on the spatial texture features of each image group using feature fusion. For each image group, the following steps are performed: sum the results of each convolution operation to obtain the predicted probability value of blinking behavior corresponding to the multiple image groups.

2. The method as described in claim 1, characterized in that, Before performing temporal and spatial feature extraction on the image group containing the human eye in the video stream, the method further includes: Extract the sequence number of at least three consecutive frames containing the human eye from the video stream; At least three consecutive frames of images containing human eyes in the video stream are face aligned, and the area where the human eyes are located is extracted from the at least three consecutive frames of aligned images using pre-selection boxes. The region where the human eye is located is converted to grayscale, and the grayscale regions where the human eye is located are arranged in the order of the sequence numbers to obtain the image group.

3. The method as described in claim 1, characterized in that, Before performing temporal and spatial feature extraction on the image group containing the human eye in the video stream, the method further includes: Extract multiple consecutive frames of images containing human eye views from the video stream; The multiple consecutive images are divided into multiple image groups, wherein each two image groups include at least three consecutive images that are partially repeated or not repeated.

4. The method as described in claim 1, characterized in that, The extraction of features in the temporal and spatial domains from multiple image groups containing human eyes in the video stream includes: Perform the following operations for each of the image groups: From at least three frames of images containing human eyes in the video stream, temporal features representing human eye movements are extracted, wherein the human eye movements include opening and closing of the same part of the eye; and Spatial texture features representing the location of the human eye are extracted from at least three frames of images containing the human eye in the video stream.

5. The method according to any one of claims 1-4, characterized in that, The step of determining whether blinking has occurred based on the predicted probability value includes: Compare whether the predicted probability value is greater than a preset blink threshold; If the predicted probability value is greater than the preset blink threshold, then it is determined that a blinking behavior has occurred. If the predicted probability value is not greater than the preset blink threshold, then it is determined that no blinking behavior has occurred.

6. A device for detecting blinks, characterized in that, include: The feature extraction unit is used to extract features in the temporal and spatial domains from multiple image groups containing human eyes in a video stream using the first three-dimensional convolutional layer in the neural network. The image group includes at least three consecutive images, and the order of the at least three consecutive images in the image group is consistent with the order of the at least three consecutive images in the video stream. The neural network includes multiple three-dimensional convolutional layers. The convolution operation unit is used to perform three-dimensional convolution operations on the temporal and spatial features extracted from each image group using the three-dimensional convolution layer after the first three-dimensional convolution layer, to obtain the predicted probability value of the video stream containing blinking behavior, and to determine whether blinking behavior has occurred based on the predicted probability value. The temporal and spatial features extracted from each image group are subjected to a 3D convolution operation to obtain the predicted probability value of blinking behavior in the video stream. The convolution operation unit is used for: For each image group, the following steps are performed: using a preset three-dimensional convolution kernel to extract and combine the same local feature information of at least three consecutive images through a low-level feature layer to capture the temporal feature changes, and using a preset three-dimensional convolution kernel to perform convolution operation on the spatial texture features of each image group using feature fusion. For each image group, the following steps are performed: sum the results of each convolution operation to obtain the predicted probability value of blinking behavior corresponding to the multiple image groups.

7. The apparatus as claimed in claim 6, characterized in that, Before performing temporal and spatial feature extraction on the image group containing the human eye in the video stream, an image group acquisition unit is further included, which is used for: Extract the sequence number of at least three consecutive frames containing the human eye from the video stream; At least three consecutive frames of images containing human eyes in the video stream are face aligned, and the area where the human eyes are located is extracted from the at least three consecutive frames of aligned images using pre-selection boxes. The region where the human eye is located is converted to grayscale, and the grayscale regions where the human eye is located are arranged in the order of the sequence numbers to obtain the image group.

8. The apparatus as claimed in claim 6, characterized in that, Before performing temporal and spatial feature extraction on the image group containing the human eye in the video stream, an image group partitioning unit is included, which is used for: Extract multiple consecutive frames of images containing human eye views from the video stream; The multiple consecutive images are divided into multiple image groups, wherein each two image groups include at least three consecutive images that are partially repeated or not repeated.

9. The apparatus as claimed in claim 6, characterized in that, The feature extraction unit performs temporal and spatial feature extraction on multiple image groups containing human eyes in the video stream, wherein the feature extraction unit is used for: Perform the following operations for each of the image groups: From at least three frames of images containing human eyes in the video stream, temporal features representing human eye movements are extracted, wherein the human eye movements include opening and closing of the same part of the eye; and Spatial texture features representing the location of the human eye are extracted from at least three frames of images containing the human eye in the video stream.

10. The apparatus according to any one of claims 6-9, characterized in that, The convolution operation unit is used to determine whether blinking has occurred based on the predicted probability value. Compare whether the predicted probability value is greater than a preset blink threshold; If the predicted probability value is greater than the preset blink threshold, then it is determined that a blinking behavior has occurred. If the predicted probability value is not greater than the preset blink threshold, then it is determined that no blinking behavior has occurred.

11. A terminal, characterized in that, include: Memory, used to store executable instructions; A processor for reading and executing executable instructions stored in the memory to implement the method as described in any one of claims 1-5.

12. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor, the processor is able to perform the method as described in any one of claims 1-5.

Images