Multi-person-oriented blink and sight line integrated detection model construction method
By constructing an integrated blink and line of sight detection model for multiple people, using self-attention interaction and dynamic filters, combined with multi-task loss function, the real-time and accuracy of blink detection and line of sight estimation in multi-person scenarios in the prior art is solved. The problem is to realize efficient feature multiplexing and interaction, and improve detection performance in multiple instance scenarios.
Patent Information
- Application Number
- CN202410982935.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-12
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing blink detection methods are difficult to achieve real-time operation in multi-person scenarios, and there is a lack of a framework that can satisfy blink detection and line of sight estimation at the same time, resulting in insufficient accuracy and speed under unrestricted conditions.
Build an integrated detection model of blinking and line of sight for multiple people. Through feature extraction network and queryer initialization and update modules, combined with multi-task prediction heads, the joint detection and estimation of blinking and line of sight is realized, and self-attention interaction and dynamic filter are adopted. Improve feature extraction robustness and fine-grainedness, and use multi-task loss function for joint optimization.
In multiple instances, the accuracy and speed of blink detection and line of sight estimation are improved, efficient feature multiplexing and interaction are achieved, and good real-time and generalization performance are good, and the running speed is not sensitive to the number of people.
Smart Images

Figure CN118942145A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision, and more specifically, relates to a method for constructing a blink and sight line integrated detection model for multiple people. Background Art
[0002] Blinking and gaze movement are important physiological behaviors of the human body, which can reflect the physiological and psychological state of the face. With the development of scientific research, blink detection and gaze estimation have been widely used in medical treatment, criminal investigation, assisted driving, liveness authentication, human-computer interaction, virtual reality and other fields, with great application value.
[0003] Existing blink detection methods generally follow a multi-stage reasoning paradigm, which requires face detection, facial key point detection, eye location, local eye region feature extraction, and blink recognition in sequence. Under this paradigm, each submodule is relatively independent, and the features of each stage cannot be jointly optimized, so it is easy to obtain a suboptimal solution. In addition, the blink feature only relies on local eye features and lacks global perception. It is also difficult to accurately locate the eye region under unrestricted conditions, resulting in the blink detection performance in unrestricted scenarios still not being able to reach the accuracy required for practical applications. In addition, in multi-person scenarios, the reasoning time of this multi-stage paradigm is positively correlated with the number of people, making it difficult to meet the requirements of real-time operation in multi-person scenarios.
[0004] At the same time, existing methods for blink detection and gaze estimation are relatively independent, and there is a lack of a framework that can simultaneously meet the needs of real-time blink detection and gaze estimation in multi-instance scenarios. Summary of the invention
[0005] In view of the defects of the prior art and the need for improvement, the present invention provides a method for constructing an integrated blink and gaze detection model for multiple people, which aims to improve the accuracy and speed of instance positioning, blink detection and gaze estimation in multi-instance unrestricted scenarios.
[0006] To achieve the above object, according to one aspect of the present invention, a method for constructing a blink and gaze integrated detection model for multiple people is provided, comprising the following steps:
[0007] Construct an integrated blink and gaze detection network, including: a feature extraction network for extracting high-level semantic features F of RGB video clips v; A query initialization module is used to initialize N groups of instance-level face query and instance-level eye query and each group of corresponding face proposal boxes and eye proposal boxes, wherein the value of N is not less than the number of faces in each frame of the RGB video clip; a query update module is used to perform self-attention interaction between the face query features and eye query features corresponding to the same frame of the current query in each group of queries, and use the features obtained by the interaction as the update value of the corresponding feature; using each group of current face query and eye query, according to the corresponding face proposal box P i and eye proposal box p i , from F v Extract P i 、p i The corresponding local features are used to update the corresponding query; the multi-task prediction head is used to update the corresponding query based on the current face query Q i Predict the existence of the i-th face and the face frame boundary coordinates L i ; Based on the current eye query q i Predict the existence of the i-th eye and the eye frame boundary coordinates l i , blink probability of each frame and eye gaze direction of each frame;
[0008] The RGB video clip sample set is used to iteratively train the blink and gaze integrated detection network to obtain the blink and gaze integrated detection model and complete the construction. After each iteration, the current L i , l i As the new face proposal box P for the next iteration i , Eye suggestion framep i .
[0009] Further, the initialization method is:
[0010] Set the initial eigenvector And the corresponding initial proposal box Will and P i 、p i Copy T times along the time dimension to form Q i ,q i and P i 、p i , as a set of instance-level face query Q corresponding to the i-th face initialized i and instance-level eye query q i , and the initialized face proposal box P i and eye proposal box p i ; Wherein, T represents the total number of frames of the RGB video clip; the superscript 4 represents the number of vertices of the proposal box, the superscript C represents the number of channels per frame, and i ranges from 1 to N.
[0011] Furthermore, the query update module is further used to perform the following feature interaction before performing local feature extraction:
[0012] Interacting the features corresponding to the same frame image between each current face query and other face query devices, and using the features obtained by the interaction as feature update values corresponding to the same frame image in the face query device;
[0013] Self-attention interaction is performed between the features corresponding to different frame images within each current query, and the features obtained from the interaction are used as the update values of the corresponding features.
[0014] Furthermore, the specific implementation of the query update module when extracting local features is as follows:
[0015] Using the current face query Q i and Eye Finder q i , according to the corresponding face proposal box P i and eye proposal box p i , for F v Perform RoI align and get P i 、p i The corresponding local features; generating a dynamic filter according to each group of face query and eye query; filtering the local features by the dynamic filter, and the filtered features are linearly projected and respectively compared with Q i ,q i Added together, as the new face query Q i and Eye Finder q i ; Where i ranges from 1 to N.
[0016] Further, the multi-task prediction head includes:
[0017] Face prediction head, used to predict the face based on the current face query Q i Predict the existence of the i-th face And the face frame boundary coordinates T represents the total number of frames of the RGB video segment;
[0018] The eye prediction head is used to predict the eye based on the current eye query q i Predict the existence of the i-th eye And the eye frame boundary coordinates
[0019] Blink prediction head, used to predict the blink probability b of the i-th eye in each frame based on the current eye query qi i ;
[0020] The gaze direction prediction head is used to predict the direction of the eye based on the current eye query q i Predict the eye gaze direction g of the i-th eye in each frame i .
[0021] Furthermore, the loss function is calculated after each iteration as follows:
[0022] By minimizing the cost function, the optimal pairing relationship between the multi-face prediction results of the multi-task prediction head and the labels of the real multi-face is determined
[0023] Based on the best pairing relationship Calculate the training loss function;
[0024] Wherein, the cost function is:
[0025]
[0026] The training loss function is expressed as:
[0027]
[0028] Where N gt is the actual number of faces in the training sample; T represents the total number of frames of the RGB video segment; Represents the N instance-level prediction results obtained by the model prediction; represents the actual label; The best pairing relationship The corresponding N instance-level prediction results and labels Matching instance-level prediction results; is the classification loss used to calculate existence; is the regression loss used to calculate positioning; For face localization and tracking loss; Loss of eye positioning and tracking; is instance-level blink loss; It is the instance-level gaze direction loss; is the regression loss function.
[0029] The present invention also provides a method for integrated blink and sight line detection for multiple people, comprising:
[0030] The RGB video clip to be detected is input into the blink and gaze integrated detection model constructed by the blink and gaze integrated detection model construction method as described above, and the blink probability of each frame and the eye gaze direction of each frame of each face in the RGB video clip are predicted.
[0031] Furthermore, it also includes:
[0032] According to the preset blink threshold, the blink probability of each face frame is set to 1 if it is greater than the blink threshold, and is set to 0 if it is less than the blink threshold, and the continuous frames corresponding to 1 are taken as a blink interval of the face.
[0033] The present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the method for constructing an integrated blink and gaze detection model as described above and / or the method for integrated blink and gaze detection as described above.
[0034] The present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for constructing a blink and gaze integrated detection model as described above and / or the method for detecting blink and gaze integrated as described above are implemented.
[0035] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:
[0036] (1) The present invention provides a method for constructing an integrated detection model of blink detection and gaze estimation in an unrestricted multi-instance scenario. The integrated detection model of blink detection and gaze estimation includes a query initialization module and a query update module. The query initialization module is used to initialize N groups of instance-level face queries and instance-level eye queries and the face proposal frames and eye proposal frames corresponding to each group; the query update module is used to perform self-attention interaction between the face query features and the eye query features corresponding to the same frame image in each current group of queries in sequence, and use the features obtained by the interaction as the update values of the corresponding features; each current group of face queries and eye queries are used to update the face proposal frames P according to the corresponding face proposal frames P. i and eye proposal box p i , from F v Extract P i 、p i The corresponding local features are used to update the corresponding query. The query obtained in this way is used as a detection feature, and multi-task prediction is used to simultaneously realize the positioning and tracking of the face position and eye position of each instance in a multi-person scene, the blink time period detection and the line of sight direction estimation. That is, different subtasks share features, realizing efficient feature reuse and interaction, having high generalization performance for each subtask, and having better real-time performance. The running speed is insensitive to the number of people in the scene, and efficiently realizing the positioning and tracking of instance faces and eyes, blink detection and line of sight estimation in a framework, improving the accuracy and speed of instance positioning, blink detection and line of sight estimation in multi-instance unrestricted scenes.
[0037] (2) Based on the spatial interaction between face query units, the present invention also proposes the spatial interaction between face and eye query units, as well as the spatial interaction within each of the face and eye query units, so that the eye query unit contains richer global facial information, improves the robustness of eye feature extraction, and enhances the fine-grainedness of face query unit features.
[0038] (3) The present invention proposes a multi-task loss function based on one-to-one matching. By jointly optimizing multiple tasks, the network is ultimately capable of processing multiple tasks such as face positioning and tracking, eye positioning, blink detection, and line of sight estimation. The information interaction between the multiple tasks plays a complementary role, which can further improve the recognition accuracy of each task. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 A block diagram of a method for constructing a blink and gaze integrated detection model for multiple people provided by an embodiment of the present invention;
[0040] Figure 2 A block diagram of another method for constructing a blink and gaze integrated detection model for multiple people provided by an embodiment of the present invention;
[0041] Figure 3 A schematic diagram of a process of converting frame-level blink probability into interval-level blink provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0042] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0043] Embodiment 1
[0044] A method for constructing an integrated blink and gaze detection model for multiple people, such as Figure 1 As shown, the following steps are included:
[0045] Construct an integrated blink and gaze detection network, including: a feature extraction network for extracting high-level semantic features F of RGB video clips v; A query initialization module is used to initialize N groups of instance-level face queries and instance-level eye queries and the corresponding face proposal boxes and eye proposal boxes of each group, wherein the value of N is not less than the number of faces in each frame of the RGB video clip, and each group of queries corresponds to one face in the RGB video clip; a query update module is used to perform self-attention interactions between the face query features and the eye query features corresponding to the same frame of the current query in each group of queries, and use the features obtained by the interaction as the update values of the corresponding features; using each group of current face queries and eye queries, according to the corresponding face proposal box P i and eye proposal box p i , from F v Extract P i 、p i The corresponding local features are used to update the corresponding query; the multi-task prediction head is used to update the corresponding query based on the current face query Q i Predict the existence of the i-th face and the face frame boundary coordinates L i ; Based on the current eye query q i Predict the existence of the i-th eye and the eye frame boundary coordinates l i , blink probability of each frame and eye gaze direction of each frame;
[0046] The RGB video clip sample set is used to iteratively train the blink and gaze integrated detection network to obtain the blink and gaze integrated detection model and complete the construction. After each iteration, the current L i , l i As the new face proposal box P for the next iteration i 、Eye suggestion framep i .
[0047] In the above method, features are extracted from the input RGB video through a feature extraction network, as described below:
[0048] Takes an RGB video clip as input, whose dimensions are Where T is the number of frames of the input video, H and W are the height and width resolution of the video frame respectively. The feature extraction backbone network takes the video segment as input and extracts the high-level semantic features of the video. Where C is the number of feature channels, H′ and W′ are the high and wide resolutions of the feature, respectively.
[0049] In the above method, regarding the instance-level face query and the eye query, each query is essentially a feature, as described below:
[0050] Building N-Instance Face Querier Each instance face query Q i The dimension is T×C, that is, Qi Represents instance-level facial features. Each instance face query is associated with an eye query Paired, both have the same feature dimension (i.e. ). The eye query is essentially the instance-level eye-related features. Each instance face and eye query will also be associated with a proposal box P i 、p i The proposal box is used to determine the rough position of the face / eye of the instance, and its dimension is Where 4 represents the coordinates of the four vertices of the rectangular box. As a preferred implementation, the initialization method of each instance query feature and proposal box is: set the initial feature vector And its paired initial proposal box are learnable parameters. and P i 、p i Will be copied T times along the time dimension to form Q i ,q i and P i 、p i . and It is optimized and learned via back-propagation during the training phase and fixed during the testing phase.
[0051] In a more specific implementation, the most common ResNet-50-FPN is selected as the feature extraction network. The RGB video clip is used as input, and its dimension is Where T is the number of frames of the input video, and H and W are the height and width resolutions of the video frames, respectively. In this implementation case, the values of T, H, and W are 11, 360, and 640, respectively. It is worth noting that when the input video length exceeds T, this example processes the entire video in sequence in a sliding window manner in the time dimension. In addition, the choice of feature extraction network and the values of T, H, and W can be flexibly set according to the actual input and the capabilities of its own computing resources. The feature extraction network inputs each frame of the video in parallel, extracts high-level semantic features of each frame of the video and stacks them into Where C is the number of feature channels, which is 256 in this case. H′ and W′ are the height and width resolutions of the feature, respectively. Their values depend on the feature extraction network used. In this case, they are 90 and 160.
[0052] In the above method, regarding feature interaction, that is, self-attention interaction is performed in sequence between the face query feature and the eye query feature corresponding to the same frame image in each current query group, and the features obtained by the interaction are used as the update value of the corresponding feature, which can be expressed as:
[0053]
[0054] On this basis, it can be further preferred that Figure 2 As shown, the query update module is also used to perform the following feature interactions before performing local feature extraction:
[0055] The features corresponding to the same frame image of each current face query are interacted with those of other face query devices, and the features obtained by the interaction are used as the feature update values corresponding to the same frame image in the face query device, which is expressed as:
[0056] Self-attention interaction is performed between the features corresponding to different frame images within each current query, and the features obtained by the interaction are used as the update values of the corresponding features, expressed as:
[0057] As a preferred implementation, the query update module is specifically used for:
[0058] Using the current face query Q i and Eye Finder q i , according to the corresponding face proposal box P i and eye proposal box p i , for F v Perform RoI align and get P i 、p i The corresponding local features; at the same time, a dynamic filter is generated according to each group of face query and eye query; the local features are filtered by the dynamic filter, and the filtered features are linearly projected and respectively compared with Q i ,q i Added together, as the new face query Q i and Eye Finder q i ; Where i ranges from 1 to N.
[0059] Regarding the value of N, in this case the value is 100. There is no special restriction on the value of this hyperparameter. In theory, it can be larger than the number of people in the input video.
[0060] As a preferred implementation, the multi-task prediction head performs multi-task prediction, as described below:
[0061] Face prediction: Face prediction is implemented by the face prediction head. Given the updated face query features, face presence prediction is implemented using a multi-layer perceptron with sigmoid normalization:
[0062] C i =Sigmoid(MLP C (Q i ))
[0063] in Indicates the probability of the presence of a face in each frame, and MLP stands for Multi-Layer Perceptron. Face frame location prediction is achieved in a similar way:
[0064] L i =MLP L (Q i )
[0065] in Represents the boundary coordinates of the face frame in each frame. i As a new proposal frame P i , used for the next iteration.
[0066] Eye prediction: Eye prediction is implemented by the eye prediction head, which is implemented in a similar way to the face prediction head. Given the updated eye query feature q i , use another multilayer perceptron with sigmoid normalization to achieve eye presence prediction:
[0067] c i =Sigmoid(MLP c (q i ))
[0068] in Indicates the probability of the existence of eyes in each frame, and MLP stands for multi-layer perceptron. Eye box location prediction is achieved in a similar way:
[0069] l i =MLP l (q i )
[0070] in Represents the boundary coordinates of the eye box for each frame. i As a new proposal frame p i , used for the next iteration.
[0071] Blink interval prediction: Blink interval prediction is implemented by the blink prediction head, whose structure is a multi-layer perceptron with sigmoid normalization. It acts on the updated eye query feature q i , corresponding to q i Blink prediction for prediction example:
[0072] b i =Sigmoid(MLP b (q i ))
[0073] in, It is the frame-level blink prediction, which indicates the probability that each frame is within a blink event.
[0074] Gaze direction prediction: Gaze direction prediction is implemented by the gaze direction prediction head, which is a multi-layer perceptron with sigmoid normalization. i , corresponding to q i Prediction of the sight direction of the prediction instance:
[0075] g i =Sigmoid(MLP g (q i ))
[0076] in, is the prediction result of the sight direction of each frame.
[0077] The number of training iterations is M, such as Figure 1 As shown, it can be used as a preferred implementation mode, the way to calculate the loss function after each iteration is:
[0078] By minimizing the cost function, the optimal pairing relationship between the multi-face prediction results of the multi-task prediction head and the labels of the real multi-faces is determined; based on the optimal pairing relationship, the training loss function is calculated;
[0079] Among them, the cost function is:
[0080]
[0081] The training loss function is expressed as:
[0082]
[0083] Where N gt is the actual number of faces in the training sample; T represents the total number of frames of the RGB video segment; Represents the N instance-level prediction results obtained by the model prediction; represents the actual label; The best pairing relationship The corresponding N instance-level prediction results and labels Matching instance-level prediction results; The classification loss used to calculate existence is not limited to a specific form, and cross entropy loss or focal loss can be used; It is used to calculate the regression loss for positioning. The specific form is not limited, and L1 loss and GIoU loss can be used; For face localization and tracking loss; Loss of eye positioning and tracking; is instance-level blink loss; It is the instance-level gaze direction loss; Represents the loss function used for regression problems. The specific form is not limited, and L1 loss and L2 loss can be used.
[0084] That is, the network is trained using a multi-task loss function based on one-to-one matching, as follows:
[0085] The instance-level prediction corresponding to the instance query is matched with the instance-level label, and the method of this embodiment obtains N instance-level prediction results:
[0086]
[0087] For the current input training video sample, it contains N gt Instances, with labels in the form of:
[0088]
[0089] Confirm the pairing relationship between instance-level prediction and instance-level label, and the cost function is:
[0090]
[0091] in, The classification loss used to calculate the presence of faces (the specific form is not limited, cross entropy loss or focal loss can be used, etc.), It is used to calculate the regression loss for face positioning (the specific form is not limited, and L1 loss and GIoU loss can be used, etc.).
[0092] By minimizing the cost function Achieve one-to-one pairing of instance-level predictions and instance-level labels to obtain the best pairing relationship
[0093] Get the best pairing between instance-level predictions and labels After that, the training loss function is:
[0094]
[0095] in is with the label Matching instance-level prediction results; For face localization and tracking loss, its form is the same as Consistent, that is:
[0096]
[0097] It is the instance-level blink loss, specifically:
[0098]
[0099] It is the instance-level sight direction loss, specifically:
[0100]
[0101] in, Represents the loss function used for regression problems (the specific form is not limited, and L1 loss and L2 loss can be used, etc.).
[0102] On this basis, constraints are imposed on the prediction results related to the eyes, so that the network can acquire the ability to locate the eyes through training. The eye loss will be defined by the following formula:
[0103]
[0104] For predictions that do not match the true label instances, only the loss is used The output value c of the probability of the existence of the supervised face i As close to 0 as possible.
[0105] First, obtain multiple RGB videos, select part of the data as training samples, and the rest of the data as test samples. In order to train the model, it is necessary to ensure that the training samples have the face and eye frame position labels of each frame of each instance, as well as the blink interval and the line of sight direction labels of each frame. Considering that it is difficult to collect data with both blink and line of sight annotations, in order to further reduce the cost of training sample annotations, samples with blink interval annotations and line of sight direction annotations can be trained separately. This embodiment uses the training set in the MPEblink dataset as a training sample for multi-person blink detection, and uses Gaze360 as a training sample for line of sight estimation. Finally, the test set in the MPEblink dataset is used as a test sample for the overall multi-task prediction network.
[0106] That is, considering that it is difficult to collect data with both blink and gaze annotations, in order to further reduce the cost of training sample annotations, samples with blink interval annotations and gaze direction annotations can be trained separately, and the gaze estimation error loss is not calculated on the data with blink annotations, and the blink loss is not calculated on the data with gaze direction annotations. Taking this implementation as an example, the MPEblink dataset is first used to train blink detection without calculating the gaze estimation loss, and then the Gaze360 dataset is used to train gaze estimation without calculating the blink detection loss. Under the premise of having labeled data, end-to-end training can be used to further improve the generalization performance of the feature.
[0107] Embodiment 2
[0108] A method for integrated blink and gaze detection for multiple persons, comprising:
[0109] The RGB video clip to be detected is input into the blink and gaze integrated detection model constructed by the blink and gaze integrated detection model construction method described in Example 1, and the blink probability of each frame and the eye gaze direction of each frame of each face in the RGB video clip are predicted.
[0110] It can be used as a preferred embodiment, such as Figure 3 As shown, the detection method also includes:
[0111] According to the preset blink threshold, the blink probability of each face frame is set to 1 if it is greater than the blink threshold, and is set to 0 if it is less than the blink threshold, and the continuous frames corresponding to 1 are taken as a blink interval of the face.
[0112] That is, adjacent frame-level blink predictions are merged according to a threshold (e.g., 0.3) to produce the final blink interval where k∈K is the kth blink among K blink predictions.
[0113] In practical applications, an RGB video is input and the test results can be obtained through the integrated model designed by the present invention, including the face position, eye position, sight direction and blinking interval of each instance in each frame of the video.
[0114] Embodiment 3
[0115] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method for constructing an integrated blink and gaze detection model as described above and / or the method for integrated blink and gaze detection as described above are implemented.
[0116] The electronic device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The memory may be used to store computer programs and / or modules, and the processor may perform various functions of the electronic device by running or executing computer programs and / or modules stored in the memory, and calling data stored in the memory.
[0117] The relevant technical solutions are the same as those in Embodiment 1 and Embodiment 2 and will not be described in detail here.
[0118] Embodiment 4
[0119] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for constructing a model for integrated blink and gaze detection as described above and / or the method for integrated blink and gaze detection as described above are implemented.
[0120] Specifically, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0121] The relevant technical solutions are the same as those in Embodiment 1 and Embodiment 2 and will not be described in detail here.
[0122] In general, the present invention can simultaneously realize multiple tasks such as positioning and tracking faces and eyes of each instance, detecting blink intervals, and estimating gaze direction in multiple instance scenarios. Different subtasks share features, achieve efficient interaction, have good generalization performance, and have better real-time performance. The running speed is not sensitive to the number of people in the scene.
[0123] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for constructing an integrated blink and gaze detection model for multiple people, characterized in that: The following steps are involved: Construct an integrated blink and gaze detection network, including: a feature extraction network for extracting high-level semantic features F of RGB video clips v ; A query initialization module is used to initialize N groups of instance-level face query and instance-level eye query and each group of corresponding face proposal boxes and eye proposal boxes, wherein the value of N is not less than the number of faces in each frame of the RGB video clip; a query update module is used to perform self-attention interaction between the face query features and eye query features corresponding to the same frame of the current query in each group of queries, and use the features obtained by the interaction as the update value of the corresponding feature; using each group of current face query and eye query, according to the corresponding face proposal box P i and eye proposal box p i , from F v Extract P i 、p i The corresponding local features are used to update the corresponding query; the multi-task prediction head is used to update the corresponding query based on the current face query Q i Predict the existence of the i-th face and the face frame boundary coordinates L i , and based on the current eye query q i Predict the existence of the i-th eye and the eye frame boundary coordinates l i , blink probability of each frame and eye gaze direction of each frame; The RGB video clip sample set is used to iteratively train the blink and gaze integrated detection network to obtain the blink and gaze integrated detection model and complete the construction. After each iteration, the current L i , l i As the new face proposal box P for the next iteration i 、Eye suggestion framep i .
2. The method for constructing an integrated blink and gaze detection model according to claim 1, characterized in that: The initialization method is: Set the initial eigenvector And the corresponding initial proposal box Will and P i 、p i Copy T times along the time dimension to form Q i ,q i and P i 、p i , as a set of instance-level face query Q corresponding to the i-th face initialized i and instance-level eye query q i , and the initialized face proposal box P i and eye proposal box p i ; Wherein, T represents the total number of frames of the RGB video clip; the superscript 4 represents the number of vertices of the proposal box, the superscript C represents the number of channels per frame, and i ranges from 1 to N.
3. The method for constructing an integrated blink and sight detection model according to claim 1, characterized in that: Before performing local feature extraction, the query update module is also used to perform the following feature interaction: Interacting the features corresponding to the same frame image between each current face query and other face query devices, and using the features obtained by the interaction as feature update values corresponding to the same frame image in the face query device; Self-attention interaction is performed between the features corresponding to different frame images within each current query, and the features obtained from the interaction are used as the update values of the corresponding features.
4. The method for constructing an integrated blink and gaze detection model according to claim 1, characterized in that: The specific implementation of the query update module when extracting local features is as follows: Using the current face query Q i and Eye Finder q i , according to the corresponding face proposal box P i and eye proposal box p i , for F v Perform RoI align and get P i 、p i The corresponding local features; generating a dynamic filter according to each group of face query and eye query; filtering the local features by the dynamic filter, and the filtered features are linearly projected and respectively compared with Q i ,q i Added together, as the new face query Q i and Eye Finder q i ; Where i ranges from 1 to N.
5. The method for constructing an integrated blink and gaze detection model according to claim 1, characterized in that: The multi-task prediction head comprises: Face prediction head, used to predict the face based on the current face query Q i Predict the existence of the i-th face And the face frame boundary coordinates T represents the total number of frames of the RGB video segment; The eye prediction head is used to predict the eye based on the current eye query q i Predict the existence of the i-th eye And the eye frame boundary coordinates Blink prediction head for predicting eye blinks based on the current eye query q i Predict the blink probability b of the i-th eye in each frame i ; The gaze direction prediction head is used to predict the direction of the eye based on the current eye query q i Predict the eye gaze direction g of the i-th eye in each frame i .
6. The method for constructing an integrated blink and sight detection model according to claim 5, characterized in that: The loss function is calculated after each iteration as: By minimizing the cost function, the optimal pairing relationship between the multi-face prediction results of the multi-task prediction head and the labels of the real multi-face is determined. Based on the best pairing relationship Calculate the training loss function; Wherein, the cost function is: The training loss function is expressed as: Where N gt is the actual number of faces in the training sample; T represents the total number of frames of the RGB video segment; Represents the N instance-level prediction results obtained by the model prediction; represents the actual label; The best pairing relationship The corresponding N instance-level prediction results and labels Matching instance-level prediction results; is the classification loss used to calculate existence; is the regression loss used to calculate positioning; For face localization and tracking loss; Loss of eye positioning and tracking; is instance-level blink loss; It is the instance-level gaze direction loss; is the regression loss function.
7. A method for integrated blink and sight detection for multiple people, characterized in that: include: The RGB video clip to be detected is input into the blink and gaze integrated detection model constructed by the blink and gaze integrated detection model construction method described in any one of claims 1 to 6, and the blink probability of each frame and the eye gaze direction of each frame of each face in the RGB video clip are predicted.
8. The method for integrated blink and sight line detection according to claim 7, characterized in that: Also includes: According to the preset blink threshold, the blink probability of each face frame is set to 1 if it is greater than the blink threshold, and is set to 0 if it is less than the blink threshold, and the continuous frames corresponding to 1 are taken as a blink interval of the face.
9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, it implements the method for constructing a blink and gaze integrated detection model as described in any one of claims 1 to 6 and / or the method for detecting blink and gaze integrated as described in any one of claims 7 or 8.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the method for constructing a blink and gaze integrated detection model as described in any one of claims 1 to 6 and / or the method for detecting blink and gaze integrated as described in any one of claims 7 or 8.
Citation Information
Patent Citations
Blink detection model under unlimited condition and construction method and application thereof
CN110969109A
Early fatigue detection method and system based on fine eye movement features
CN112434611A
Non-calibration eye movement interaction method and device
CN113419623A
System and method for eye-tracking and blink detection
US20060215017A1
Driver gaze detection system
US20140204193A1
