User state monitoring method and device

Abstract

The invention provides a user state monitoring method and device. The method comprises the following steps: collecting a scene image of a target scene through a plurality of cameras; for each frame of scene image collected by each camera, judging whether a face of a target user and / or facial key points of the face are detected in the current frame of scene image collected by the camera; if yes, determining that the target user is in an on-duty state; extracting a region of interest corresponding to the target eye based on the current frame scene image; determining whether the target user is in a long eye closing state based on the region of interest; if the target user is not in the long eye closing state, selecting a specified key point from the face key points; and determining whether the target user is in a focused state based on the specified key point. According to the method, the on-duty state, the long eye closing state and the concentration state of the target user can be comprehensively judged under the constraints of visible light, no wearing, no complex calibration and the like, so that the state of the monitored user can be stably recognized in the natural environment.

Description

User Status Monitoring Method and Device Technical Field

[0001] This invention relates to the fields of computer vision and human-computer interaction technology, and in particular to a user status monitoring method and device. Background Technology

[0002] With the widespread adoption of online teaching and remote work, the demand for non-contact attention and fatigue monitoring is rapidly increasing. Traditional methods often rely on single-channel cameras and fixed threshold rules. These solutions are highly sensitive to posture changes, lighting variations, and occlusion, making it difficult to distinguish between brief blinks, natural head tilts, and genuine distraction, leading to false alarms and missed alarms. While deep learning methods can improve the recognition accuracy of a single modality, their stability remains insufficient, especially under conditions of side profiles, head tilts, and partial occlusion. From an engineering perspective, real-time monitoring also faces challenges such as camera interruptions, frame rate fluctuations, and resolution changes, making it difficult for the system to operate continuously for extended periods. Solutions relying on dedicated infrared eye trackers or head-mounted devices can improve robustness, but their high cost and intrusiveness limit large-scale deployment on general-purpose visible light cameras and general-purpose computing platforms. In summary, existing technologies struggle to reliably identify the monitored user's state in natural environments. Summary of the Invention

[0003] The purpose of this invention is to provide a user status monitoring method and apparatus to stably identify the status of the monitored user in a natural environment.

[0004] This invention provides a user status monitoring method, comprising: responding to a collection command, acquiring scene images of a target scene through multiple cameras; for each frame of scene image acquired by each camera, determining whether a target user's face and / or facial key points are detected in the current frame of scene image acquired by that camera; if a face and / or facial key points are detected, determining that the target user is in an on-duty state; extracting the region of interest corresponding to the target eye based on the current frame of scene image; determining whether the target user is in a state of long-closed eyes based on the region of interest; if the target user is not in a state of long-closed eyes, selecting specified key points from the facial key points; wherein, the specified key points include: left eye iris index, right eye iris index, left eye corner index, and right eye corner index; and determining whether the target user is in a focused state based on the specified key points.

[0005] Furthermore, the steps for determining whether a target user is in a prolonged eye-closed state based on the region of interest include: preprocessing the region of interest to obtain a preprocessed region of interest; inputting the preprocessed region of interest into a pre-trained recognition model, and outputting the current eye state of the target user through the recognition model; obtaining the historical eye state of the most recent preset number of times; calculating the eye-closed ratio and eye-closed duration based on the current eye state and the historical eye state; if the eye-closed ratio is greater than a preset ratio threshold and the eye-closed duration is greater than a preset time threshold, determining that the target user is in a prolonged eye-closed state; if the eye-closed ratio is not greater than the preset ratio threshold and / or the eye-closed duration is not greater than the preset time threshold, determining that the target user is not in a prolonged eye-closed state.

[0006] Furthermore, there are multiple left-eye iris indices, multiple right-eye iris indices, multiple left-eye corner indices, and multiple right-eye corner indices. The steps for determining whether the target user is in a focused state based on specified key points include: determining the center of the left iris based on multiple left-eye iris indices; determining the center of the right iris based on multiple right-eye iris indices; determining the center of the left eye based on multiple left-eye corner indices; determining the center of the right eye based on multiple right-eye corner indices; and determining whether the target user is in a focused state based on the centers of the left and right irises, the center of the left eye, and the center of the right eye.

[0007] Furthermore, the steps for determining whether the target user is in a focused state based on the centers of the left and right irises, the center of the left eye, and the center of the right eye include: calculating the difference between the centers of the left and right irises to obtain the left eye gaze deviation value; calculating the difference between the centers of the right and right irises to obtain the right eye gaze deviation value; determining the left eye gaze deviation direction based on the left eye gaze deviation value; determining the right eye gaze deviation direction based on the right eye gaze deviation value; if the left eye gaze deviation direction is the same as the right eye gaze deviation direction, and the absolute values ​​of both the left and right eye gaze deviation values ​​are greater than a preset deviation threshold, the target user is determined to be in an unfocused state; if the left eye gaze deviation direction is the same as the right eye gaze deviation direction, and the absolute values ​​of both the left and right eye gaze deviation values ​​are less than or equal to a preset deviation threshold, the target user is determined to be in a focused state.

[0008] Furthermore, the preset offset thresholds include: a left eye offset threshold and a right eye offset threshold; the left eye offset threshold and the right eye offset threshold are determined by: determining the left eye width based on multiple left eye corner indices; determining the right eye width based on multiple right eye corner indices; multiplying the left eye width by a preset ratio value to obtain the left eye offset threshold; multiplying the right eye width by a preset ratio value to obtain the right eye offset threshold.

[0009] Furthermore, if the left eye's gaze deviation direction is the same as the right eye's gaze deviation direction, and the absolute values ​​of both the left and right eye gaze deviation values ​​are greater than a preset deviation threshold, the steps to determine that the target user is in a state of inattention include: if the left eye's gaze deviation direction is the same as the right eye's gaze deviation direction, the absolute value of the left eye's gaze deviation value is greater than the left eye deviation threshold, and the absolute value of the right eye's gaze deviation value is greater than the right eye deviation threshold, then the target user is determined to be in a state of inattention; if the left eye's gaze deviation direction is the same as the right eye's gaze deviation direction, and the absolute values ​​of both the left and right eye gaze deviation values ​​are less than or equal to a preset deviation threshold, the steps to determine that the target user is in a state of focus include: if the left eye's gaze deviation direction is the same as the right eye's gaze deviation direction, the absolute value of the left eye's gaze deviation value is less than or equal to the left eye deviation threshold, and the absolute value of the right eye's gaze deviation value is less than or equal to the right eye deviation threshold, then the target user is determined to be in a state of inattention.

[0010] Furthermore, the method also includes: when the target user is in an open-eyed state and looking directly at the first camera among multiple cameras, acquiring multiple first images containing the target user's face through the first camera according to a preset acquisition number; extracting facial feature vectors from each first image; calculating the average value of the facial feature vectors in multiple first images to obtain the target session baseline code corresponding to the target user; when the specified user is in an open-eyed state and looking directly at the first camera, acquiring multiple second images containing the specified user's face through the first camera according to a preset acquisition number; extracting facial feature vectors from each second image; calculating the average value of the facial feature vectors in multiple second images to obtain the specified session baseline code corresponding to the specified user; calculating the similarity between the target session baseline code and the specified session baseline code; if the similarity is lower than a preset similarity threshold, determining that the target user and the specified user are different users; ending the monitoring process for the target user, saving the monitoring information corresponding to the target user based on the user ID of the target user and the camera ID of each camera, and starting the monitoring process for the specified user.

[0011] Furthermore, the method also includes: if it is determined that the second camera among multiple cameras fails to acquire scene images, or the scene images acquired by the second camera fail to open, the second camera is determined to enter a disconnected state; and the second camera is controlled to reconnect at preset time intervals.

[0012] Furthermore, the monitoring information includes: for each camera, the monitoring status of the target user determined based on each frame of scene image captured by that camera; wherein, the monitoring status corresponding to each camera is presented in the form of a timeline color band; the monitoring status includes: focused state, unfocused state, long-term closed-eye state, and off-duty state.

[0013] This invention provides a user status monitoring device, comprising: a data acquisition module for acquiring scene images of a target scene via multiple cameras in response to a data acquisition command; a judgment module for determining whether a target user's face and / or facial key points are detected in the current frame scene image acquired by each camera for each frame of scene image acquired by that camera; a first determination module for determining that the target user is in an on-duty state if a face and / or facial key points are detected; an extraction module for extracting the region of interest corresponding to the target eye based on the current frame scene image; a second determination module for determining whether the target user is in a state of long-closed eyes based on the region of interest; a selection module for selecting specified key points from facial key points if the target user is not in a state of long-closed eyes; wherein the specified key points include: left eye iris index, right eye iris index, left eye corner index, and right eye corner index; and a third determination module for determining whether the target user is in a focused state based on the specified key points.

[0014] The user status monitoring method and apparatus provided by this invention, in response to a collection command, acquires scene images of a target scene through multiple cameras; for each frame of scene image acquired by each camera, it determines whether the target user's face and / or facial key points are detected in the current frame of scene image acquired by that camera; if the face and / or facial key points are detected, it determines that the target user is in an on-duty state; based on the current frame of scene image, it extracts the region of interest corresponding to the target eye; based on the region of interest, it determines whether the target user is in a state of long-closed eyes; if the target user is not in a state of long-closed eyes, it selects specified key points from the facial key points; wherein, the specified key points include: left eye iris index, right eye iris index, left eye corner index, and right eye corner index; based on the specified key points, it determines whether the target user is in a focused state. This method can comprehensively determine the target user's on-duty state, long-closed eye state, and focused state under visible light, without wearing, and without complex calibration constraints, thereby enabling stable identification of the monitored user's state in a natural environment. Attached Figure Description

[0015] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 is a flowchart of a user status monitoring method provided in an embodiment of the present invention; Figure 2 is a schematic diagram of a status fusion priority decision tree provided in an embodiment of the present invention; Figure 3 is a flowchart of another user status monitoring method provided in an embodiment of the present invention; Figure 4 is a flowchart of another user status monitoring method provided in an embodiment of the present invention; Figure 5 is a structural schematic diagram of a user status monitoring device provided in an embodiment of the present invention; Figure 6 is a structural schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0017] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Currently, with the widespread adoption of online teaching and remote work, the demand for non-contact attention and fatigue monitoring is rapidly increasing. Traditional methods often rely on single-channel cameras and fixed threshold rules (such as using eye opening / closing ratio or blink frequency as the basis for judgment). Such solutions are highly sensitive to posture changes, lighting changes, and occlusion, making it difficult to distinguish between short blinks, natural head tilts, and genuine distraction, easily leading to false alarms and missed alarms. While deep learning methods can improve the recognition accuracy of a single modality task, if they only focus on "whether there is a face" or "eye opening / closing" without integrating information such as gaze direction and presence, their stability remains insufficient, especially under conditions of side profiles, head tilts, and partial occlusion. From an engineering perspective, real-time monitoring also faces issues such as camera disconnection, frame rate jitter, and resolution changes. Without disconnection detection and self-healing reconnection mechanisms, the system cannot operate continuously for long periods. At the same time, scenarios with multiple users rotating or changing users require clear session boundaries and historical archiving; otherwise, data ownership becomes chaotic, failing to meet the needs of teaching assessments or post-audit. While solutions relying on dedicated infrared eye trackers or head-mounted devices can improve robustness, their high cost and intrusiveness limit large-scale deployment on general-purpose visible light cameras and general-purpose computing platforms. In summary, existing technologies struggle to reliably identify the "off-duty, long-term eye-closed, inattentive, and focused" states of monitored users in natural environments, and cannot support timeline-level recording and historical review.

[0019] To facilitate understanding of this embodiment, a user status monitoring method disclosed in this invention will first be introduced, as shown in Figure 1. The method includes the following steps: Step S102, in response to a collection command, scene images of a target scene are collected using multiple cameras; the target scene can be an office, workshop, or similar scene; the number of cameras can be set according to actual needs; for example, taking three cameras, for each camera dev_id∈{0,1,2}, an independent collection thread is started (decoupled from the GUI main thread), and the video stream is opened using cv2.VideoCapture(dev_id, cv2.CAP_DSHOW), with a preferred target resolution of 640×480 (allowing each camera to fall back / rise according to its own capabilities). After opening, wait approximately 0.3 seconds for the camera to stabilize, then read and record the actual resolution and frame rate for diagnostics and logging. Each acquisition thread stores its latest frame buffer (only one copy) and a mutex lock to avoid memory and latency increases caused by large queues; the GUI main thread can use a timer (e.g., 30ms) to poll and copy the "latest frame" and present it through the display interface. A proportional scaling strategy (Qt.KeepAspectRatio) can be used to maintain the screen ratio without distortion during scaling.

[0020] Step S104: For each frame of scene image captured by each camera, determine whether the target user's face and / or facial key points are detected in the current frame of scene image captured by that camera; the target user can be understood as the monitored user in the target scene; in actual implementation, for each frame of scene image captured by each camera, face concatenation can be performed (such as haarcascade_frontalface_alt.xml, a trained frontal face detection model) to determine whether the target user's face and / or facial key points can be detected in the current frame of scene image. For example, the facial key points may include: the most prominent point between the eyebrows, the root of the nose, the corner of the mouth, etc. The set of facial key points can be obtained by using MediaPipe FaceMesh (refine_landmarks=True).

[0021] Step S106: If a face and / or facial key points are detected, the target user is determined to be in an on-duty state; if no face is detected and facial key points are also detected, the target user is considered to be off-duty during the time corresponding to the acquisition of the current frame scene image; if a face and / or facial key points are detected, the target user is considered to be in an on-duty state during the time corresponding to the acquisition of the current frame scene image.

[0022] Step S108: Based on the current frame scene image, extract the region of interest (ROI) corresponding to the target eye; Step S110: Based on the ROI, determine whether the target user is in a state of closed eyes; the target eye can be the target user's left or right eye; in actual implementation, if it is determined that the target user is in an on-duty state, the current frame scene image can be converted into a grayscale image first, and OpenCV Haar cascade can be used to detect the ROI (Region of Interest) of the left or right eye in the grayscale image respectively. Regions of interest (ROIs) can be prioritized for detection of the right eye (haarcascade_righteye_2splits.xml). If the right eye is not detected, the left eye is then detected (haarcascade_lefteye_2splits.xml). Typically, maximum and minimum size thresholds are pre-set. When the size boundary of the ROI exceeds the maximum size threshold, the ROI can be cropped. When the size boundary of the ROI is less than the minimum size threshold, the ROI is generally considered too small, and subsequent processing of the current frame scene image can be abandoned, maintaining stability based on the previous time-series smoothing result. Based on the extracted ROIs, it can be further determined whether the target user is in a state of long-closed eyes.

[0023] Step S112: If the target user is not in a state of long-closed eyes, select a designated key point from the facial key points; wherein, the designated key points include: left eye iris index, right eye iris index, left eye corner index, and right eye corner index; if it is determined that the target user is not in a state of long-closed eyes, the designated key point can be selected from the above set of facial key points. Specifically, the designated key point can be: left eye iris index {468,469,470,471,472}, right eye iris index {473,474,475,476,477}, left eye corner index {33,133}, and right eye corner index {362,263}.

[0024] Step S114: Based on the specified key points, determine whether the target user is in a focused state.

[0025] Based on the specified key points mentioned above, it can be further determined whether the target user is in a focused state. For example, when the target user's gaze is towards the center, they can be considered to be in a focused state, and when the target user's gaze is towards the left or right, they can be considered to be in a non-focused state.

[0026] The aforementioned user status monitoring method, in response to a data acquisition command, acquires scene images of the target scene using multiple cameras. For each frame of scene image acquired by each camera, it determines whether the target user's face and / or facial key points are detected in the current frame of scene image acquired by that camera. If a face and / or facial key points are detected, the target user is determined to be on duty. Based on the current frame of scene image, the region of interest corresponding to the target eye is extracted. Based on the region of interest, it is determined whether the target user is in a state of long-closed eyes. If the target user is not in a state of long-closed eyes, specified key points are selected from the facial key points. The specified key points include: left eye iris index, right eye iris index, left eye corner index, and right eye corner index. Based on the specified key points, it is determined whether the target user is in a state of focus. This method can comprehensively determine the target user's on-duty status, long-closed eyes state, and focus state under visible light, without wearing a mask, and without complex calibration constraints, thus enabling stable identification of the monitored user's status in a natural environment.

[0027] This invention also provides another user status monitoring method, which is implemented based on the method in the above embodiments. The method includes the following steps: Step 1, in response to a collection command, scene images of the target scene are collected through multiple cameras; Step 2, for each frame of scene image collected by each camera, it is determined whether the target user's face and / or facial key points are detected in the current frame of scene image collected by the camera; Step 3, if the face and / or facial key points are detected, the target user is determined to be in an on-duty state; For example, in one embodiment, when detecting the on-duty status of the target user, the on-duty condition can be "face detection successful (face_ok, i.e., face detected) or FaceMesh successful (iris_ok, i.e., facial key points detected)": Present = face_ok or iris_ok; if present=False, "off-duty" can be directly output, i.e., the target user is in an off-duty state, and a corresponding color block is added to the timeline.

[0028] Step 4: Based on the current frame scene image, extract the region of interest (ROI) corresponding to the target eye; Step 5: Preprocess the ROI to obtain the preprocessed ROI; In actual implementation, after extracting the ROI, it can be scaled, for example, scaled to 24×24, normalized to [0,1], and transformed into (24,24,1), that is, the ROI is adjusted to the standard size of 24 pixels × 24 pixels, and the pixel values ​​are linearly transformed from the original range (such as 0-255) to between 0 and 1, reshaping the data shape to 24 height × 24 width × 1 channel (grayscale image), to obtain the preprocessed ROI.

[0029] Step six involves inputting the preprocessed region of interest (ROI) into a pre-trained recognition model, which then outputs the current eye state of the target eye. This recognition model can be a lightweight CNN (Convolutional Neural Network), etc. In practice, the preprocessed ROI can be input into the trained recognition model to obtain Hat{y_t}=softmax(f_θ(ROI)), where the label space is y_t∈{Open, Close}. Here, f_θ(ROI) represents the preprocessed ROI processed by a model function with parameter θ; softmax represents converting the original output of the model function into a probability distribution. When the label is Open, the target eye's current eye state is open; when the label is Close, the target eye's current eye state is closed. To avoid blocking, Keras inference is executed on the acquisition thread, retaining only the classification results and necessary debugging information (such as the probability distribution or the original output argmax of the model function).

[0030] Step 7: Obtain the most recent preset number of historical eye states; Step 8: Calculate the closed eye ratio and closed eye duration based on the current and historical eye states; the preset number of times can be set according to actual needs, and the historical eye states can be open or closed; in actual implementation, to suppress the shaking caused by blinking, the closed eye ratio and closed eye duration can be calculated based on the current and historical eye states. For example, a double-ended queue of length L=10 can be maintained to store the most recent L eye classification results (i.e., the total number of eye classification results corresponding to the current and historical eye states), and the closed eye ratio can be defined as follows:

[0031] Where r represents the proportion of eyes closed; L represents the total number of eye classification results corresponding to the current eye state and historical eye states; This represents the eye classification result for the kth time, i.e., the eye state for the kth time.

[0032] Step 9: If the proportion of closed eyes is greater than a preset proportion threshold and the duration of closed eyes is greater than a preset time threshold, determine that the target user is in a long-closed-eye state. Step 10: If the proportion of closed eyes is not greater than a preset proportion threshold, and / or the duration of closed eyes is not greater than a preset time threshold, determine that the target user is not in a long-closed-eye state. The above-mentioned preset proportion threshold and preset time threshold can be set according to actual needs. For example, the preset proportion threshold can be 0.5, and the preset time threshold T_s can be 1s, etc. In actual implementation, when the proportion of closed eyes r is greater than the preset proportion threshold and the duration of closed eyes is greater than the preset time threshold T_s, it can be determined that the target user is in a "long-closed-eye state". If the proportion of closed eyes drops, the candidate state of "long-closed-eye state" can be cleared immediately. Preferably, the parameters are as follows: L∈[8,15], T_s∈[1,2]s; in low frame rate (<20 fps) or high noise scene, L and T_s can be appropriately increased to prioritize steady state.

[0033] If the proportion of closed eyes is not greater than a preset proportion threshold, or the duration of closed eyes is not greater than a preset time threshold, or the proportion of closed eyes is not greater than a preset proportion threshold and the duration of closed eyes is not greater than a preset time threshold, it can be determined that the target user is not in a state of long-term closed eyes.

[0034] Step 11: If the target user is not in a state of closed eyes, select designated key points from facial key points; these designated key points include: left eye iris index, right eye iris index, left eye canthus index, and right eye canthus index; there can be multiple left eye iris indices, right eye iris indices, left eye canthus indexes, and right eye canthus indexes; Step 12: Determine the center of the left eye iris based on the multiple left eye iris indices; for example, if there are 5 left eye iris indices, specifically {468,469,470,471,472}, the center of the left eye iris can be calculated using the following formula: ;in, Indicates the coordinates of the center of the left iris; Indicates the first A left eye iris index, Indicates the first The x and y coordinates of the key points corresponding to the left eye iris index.

[0035] Step 13: Determine the center of the right iris based on multiple right eye iris indices. For example, if there are 5 right eye iris indices, specifically {473,474,475,476,477}, the center of the right iris can be calculated using the following formula: ;in, Indicates the coordinates of the center of the right iris; Indicates the first A right eye iris index, Indicates the first The x and y coordinates of the key points corresponding to the right eye iris index.

[0036] Step fourteen: Determine the center of the left eye based on multiple left eye corner indices. For example, if there are two left eye corner indices, specifically {33, 133}, representing the inner and outer corners of the left eye respectively, the average of the x and y coordinates of these two key points can be calculated to obtain the coordinates of the left eye center. .

[0037] Step 15: Determine the center of the right eye based on multiple right eye corner indices. For example, if there are two right eye corner indices, specifically {362, 263}, representing the inner and outer corners of the right eye respectively, the average of the x and y coordinates of these two key points can be calculated to obtain the coordinates of the right eye center. .

[0038] Step sixteen: Determine whether the target user is in a focused state based on the center of the left iris, the center of the right iris, the center of the left eye, and the center of the right eye.

[0039] Step sixteen can be achieved through the following steps 160 to 165: Step 160, calculate the difference between the center of the left iris and the center of the left eye to obtain the left eye visual deviation value; specifically, it can be expressed by the following formula: ;in, This indicates the left eye's line of sight deviation. Indicates the coordinates of the center of the left iris; This indicates the coordinates of the center of the left eye.

[0040] Step 161: Calculate the difference between the center of the right iris and the center of the right eye to obtain the right eye visual deviation value; this can be expressed using the following formula: ;in, This indicates the right eye's line of sight deviation. Indicates the coordinates of the center of the right iris; This indicates the coordinates of the center of the right eye.

[0041] Step 162: Determine the direction of the left eye's line of sight based on the left eye's line of sight offset value. In actual implementation, the direction of the left eye's line of sight offset value can be determined based on the calculated value. For example, assuming the vertical coordinate of the left eye's line of sight offset value is zero, if the horizontal coordinate of the left eye's line of sight offset value is greater than zero, it can be considered that the left eye's line of sight is deviated to the left. If the horizontal coordinate of the left eye's line of sight offset value is less than zero, it can be considered that the left eye's line of sight is deviated to the right, and so on.

[0042] Step 163: Determine the direction of the right eye's line of sight based on the right eye's line of sight offset value. In actual implementation, the direction of the right eye's line of sight offset value can be determined based on the right eye's line of sight offset value calculated above. For example, assuming the vertical coordinate of the right eye's line of sight offset value is zero, if the horizontal coordinate of the right eye's line of sight offset value is greater than zero, it can be considered that the right eye's line of sight is deviated to the left. If the horizontal coordinate of the right eye's line of sight offset value is less than zero, it can be considered that the right eye's line of sight is deviated to the right, and so on.

[0043] Step 164: If the left eye's gaze deflection direction is the same as the right eye's gaze deflection direction, and the absolute values ​​of both the left and right eye gaze deflection values ​​are greater than a preset deflection threshold, it is determined that the target user is in a state of inattention. The preset deflection threshold is specifically a pixel threshold, which can be set according to actual needs, such as 5px. When both eyes deflect in the same direction, and the absolute values ​​of both the left and right eye gaze deflection values ​​are greater than the preset deflection threshold, it can be determined that the target user is in a state of inattention, such as "Looking Left / Right".

[0044] The preset offset thresholds include: left eye offset threshold and right eye offset threshold; the left eye offset threshold and right eye offset threshold are determined by the following steps A to D: Step A, determine the left eye width based on multiple left eye corner indices; Step B, determine the right eye width based on multiple right eye corner indices; Step C, multiply the left eye width by a preset ratio value to obtain the left eye offset threshold; Step D, multiply the right eye width by a preset ratio value to obtain the right eye offset threshold.

[0045] For ease of explanation, taking the left eye corner index {33, 133} and the right eye corner index {362, 263} as examples, the left eye width can be expressed as W^{(L)}=|x_{133}-x_{33}|, and the right eye width can be expressed as W^{(R)}=|x_{362}-x_{263}|. The preset ratio value ρ can be set according to actual needs, for example, ρ∈[0.08, 0.12]. The product of the left eye width and the preset ratio value can be used as the left eye offset threshold, and the product of the right eye width and the preset ratio value can be used as the right eye offset threshold. This method can mitigate the impact of different resolutions and variations in face scale.

[0046] This embodiment can also have visualization and debugging functions, that is, the key points corresponding to the iris can be drawn in the current frame scene image with colored dots to facilitate debugging and calibration; the rendering process can be completed in the acquisition thread, but only one compressed RGB (R:Red; G:Green; B:Blue) image is transmitted to the GUI main thread to reduce the amount of cross-thread data.

[0047] Step 164 may include: if the left eye gaze deviation direction is the same as the right eye gaze deviation direction, the absolute value of the left eye gaze deviation value is greater than the left eye deviation threshold, and the absolute value of the right eye gaze deviation value is greater than the right eye deviation threshold, it is determined that the target user is in an unfocused state; in actual implementation, when the two eyes deviate in the same direction, and the absolute value of the left eye gaze deviation value is greater than the left eye deviation threshold, and the absolute value of the right eye gaze deviation value is greater than the right eye deviation threshold, it can be determined that the target user is in an unfocused state, such as "Looking Left / Right".

[0048] Step 165: If the left eye gaze deviation direction is the same as the right eye gaze deviation direction, and the absolute values ​​of the left eye gaze deviation value and the right eye gaze deviation value are both less than or equal to the preset deviation threshold, it is determined that the target user is in a focused state.

[0049] When both eyes deviate in the same direction, and the absolute values ​​of the left and right eye line deviations are both less than or equal to a preset deviation threshold, the target user can be determined to be in a focused state, i.e., "Looking Center".

[0050] Step 165 may include: if the left eye gaze deviation direction is the same as the right eye gaze deviation direction, the absolute value of the left eye gaze deviation value is less than or equal to the left eye deviation threshold, and the absolute value of the right eye gaze deviation value is less than or equal to the right eye deviation threshold, then the target user is determined to be in a state of inattention.

[0051] In practice, when both eyes deviate in the same direction, and the absolute value of the left eye's visual deviation is less than or equal to the left eye's deviation threshold, and the absolute value of the right eye's visual deviation is less than or equal to the right eye's deviation threshold, the target user can be determined to be in a focused state, i.e., "Looking Center".

[0052] Based on the above process, it can be seen that this solution adopts a multimodal fusion strategy with priority settings. For each frame of scene image, three types of criteria can be constructed: on-duty status detection, long-term closed-eye status monitoring, and gaze deviation detection, and fused according to priority, as shown in the state fusion priority decision tree in Figure 2: ① present=False → Off-duty; Determine whether a face / or facial key points are detected. If not, determine that the target user is off-duty; if yes, determine that the target user is on-duty and continue to the next step; ② sleepy=True (long-term closed-eye condition satisfied) → Long-term closed-eye; If the target user is not in a long-term closed-eye state, continue to the next step; ③ gaze≠Center → Looking left and right; That is, it can be determined whether the gaze is off-center. If yes, determine that the target user is not focused; ④ Other → Focused.

[0053] This sequence ensures that "leaving the post" and "closing the eyes for a long time" take precedence over "gaze deviation," avoiding the misinterpretation of closing the eyes / leaving the post as "lack of focus."

[0054] This embodiment also features an acoustic alarm function. Specifically, alarm sound resources can be built into the system, with looping alarms disabled by default. For example, the logic of "continuous alarm when state = prolonged eye closure, stop upon recovery" can be enabled in the deployment parameters. To avoid multiple threads controlling the audio simultaneously, the "play / stop" function is uniformly controlled by the acquisition thread, and jitter is removed when the state changes.

[0055] Step 17: With the target user's eyes open and looking directly at the first camera among multiple cameras, capture multiple first images containing the target user's face using the first camera, according to a preset number of captures; Step 18: Extract the facial feature vector from each first image; Step 19: Calculate the average value of the facial feature vectors from the multiple first images to obtain the target session baseline code corresponding to the target user; The first camera can be any one of the multiple cameras; The preset number of captures can be set according to actual needs, such as 5 times; The facial feature vector e_i∈ ^128 (face_recognition), i.e., a 128-dimensional face feature vector; this embodiment can realize manual baseline input and session management. Specifically, after the target user clicks "Input Face", the acquisition mode is entered: when the target user's "eyes=Open and gaze=Center" conditions are met simultaneously, the face feature vector e_i∈ is extracted from each first image. ^128 (face recognition), the average of m=5 times is used as the target session baseline encoding for the target user:

[0056] in, Indicates the target session baseline encoding; m represents the total number of the first images; Indicates the first The facial feature vector of the first image.

[0057] The data collection process is presented in a progress dialog box (0 / 5→5 / 5). Once completed, it automatically switches to "monitoring mode", starts timeline recording, and records the session start time t_0 into memory.

[0058] Step 20: With the designated user's eyes open and looking directly at the first camera, capture multiple second images containing the designated user's face using the first camera, following a preset number of captures. Step 21: Extract the facial feature vector from each second image. Step 22: Calculate the average of the facial feature vectors from the multiple second images to obtain the designated session baseline code corresponding to the designated user. The designated user and the target user may be the same user or different users. When a designated user enters the target scene, the same method can be used to extract the facial feature vector e_i∈ from each second image when the designated user's "eyes = Open and gaze = Center" conditions are met simultaneously. ^128 (face_recognition), the average value after accumulating m=5 times is used as the base encoding for the specified session corresponding to the specified user.

[0059] Step 23: Calculate the similarity between the target session baseline code and the specified session baseline code; Step 24: If the similarity is lower than a preset similarity threshold, determine that the target user and the specified user are different users; The preset similarity threshold can be set according to actual needs; In actual implementation, the similarity between the target session baseline code and the specified session baseline code can be calculated, and the similarity can specifically be cosine similarity; If the similarity is greater than or equal to the preset similarity threshold, the specified user and the target user can be considered to be the same user; If the similarity is lower than the preset similarity threshold, the specified user and the target user can be considered to be different users.

[0060] Step 25: End the monitoring process for the target user, save the monitoring information corresponding to the target user based on the user ID of the target user and the camera ID of each camera, and start the monitoring process for the specified user.

[0061] After determining that the designated user and the target user are different users, the monitoring process for the target user can be terminated, and archiving and reset can be triggered. Specifically, based on the target user's user ID and the camera ID of each camera, the monitoring information corresponding to the target user can be saved, and the monitoring process for the designated user can be started. The aforementioned monitoring information includes: for each camera, the monitoring status of the target user determined based on each frame of scene image captured by that camera; wherein, the monitoring status corresponding to each camera is presented in the form of a timeline color band; the monitoring status includes: focused state, unfocused state, long-term closed-eye state, and off-duty state.

[0062] This embodiment features timeline rendering and tracking capabilities. The GUI thread maintains a timeline that grows to the right over time. Each camera has its own corresponding display window in the main GUI thread, displaying the respective timeline for each camera. A color block with a width w=0.5 is appended to each frame: gray (focused), dark blue (unfocused), orange (eyes closed), and red (away from duty). The system automatically "follows the latest" when the scroll bar nears its end; tracking pauses when monitoring personnel drag to view history, and a "return to latest" button resets the tracking.

[0063] When monitoring of the target user ends (or the application is closed) and the timeline is not empty, the session can be persisted to history_{camera_idx} / user_{user_id}.json, with the following format: {"camera_index": 1,"user_id": 3,"start_time": "YYYY-MM-DD HH:MM:SS","status_history": ["Focused","Focused","Not Focused", "..."],"duration_seconds": 1832}. After successful saving, the user_id of the camera is incremented to reserve a number for the next user. To avoid data loss, time-based segmentation / timed disk write-to-disk can be set in long sessions. That is, this embodiment can use manual input to ensure controllable session boundaries; in optional embodiments, automatic user replacement detection is introduced.

[0064] This embodiment also has a history query and read-only preview function. By entering the user number through "Query History", the corresponding JSON is read and previewed in a read-only timeline pop-up window. This pop-up window does not modify the historical data and is mainly used for quick verification and auditing.

[0065] Step 26: If it is determined that the second camera among multiple cameras fails to capture scene images, or the scene images captured by the second camera fail to open, the second camera is determined to be in a disconnected state; Step 27: Control the second camera to reconnect at preset time intervals.

[0066] The second camera mentioned above can be any of multiple cameras; the preset time interval mentioned above can be set according to actual needs; this embodiment can realize disconnection detection and self-healing reconnection functions. Specifically, the acquisition thread enters the "disconnection state" every time cap.read() fails or cap.isOpened()==False, and immediately: ① Marks the camera as unavailable; ② Sends status text (such as "Camera disconnected") and a pure black frame to the GUI main thread as a placeholder; ③ Releases the acquisition thread every preset time interval, such as ΔT=3 s, and re-attempts to open the camera. After successful recovery: ④ Resets the "monitoring state / acquisition state" flag (returns to "No face recorded"); ⑤ Clears the timeline of that channel (to avoid data confusion before and after disconnection).

[0067] It should be noted that the acquisition thread typically only performs I / O (In / Out) and light processing to avoid time-consuming operations on the GUI main thread; the black screen placeholder ensures the continuity of the UI (User Interface), and both disconnection and resumption are done by updating the interface through signals / slots to prevent the thread from directly manipulating the controls.

[0068] This embodiment can also provide resource path and packaging compatibility functionality (optional). To be compatible with single-file packaging such as PyInstaller, a resource parsing function resource_path() is provided. At runtime, it prioritizes reading Haar cascade files and model files from the temporary unpacking directory _MEIPASS, and reads from the current working directory during degradation.

[0069] This solution also provides a user status monitoring system for implementing the above-mentioned user status monitoring methods. The system consists of an acquisition module, a feature extraction and recognition module, a status fusion and judgment module, a baseline input and archiving module, and a timeline recording and historical query module. The system runs on a general-purpose visible light camera and a general-purpose computing platform, following the engineering path of "multi-source stable input - multi-modal reliable recognition - structured result accumulation". As shown in the flowchart of another user status monitoring method in Figure 3, specifically, S1 multi-camera video acquisition and stabilization: independent acquisition threads are established for multiple physical cameras, and the stream is pulled according to a unified resolution and frame rate strategy, and anomalies such as stream interruption and frame drop are detected; when a device disconnection is detected, it automatically retryes and self-heals and reconnects at a fixed period (e.g., about 3 seconds). After recovery, the local view and session state are reset to ensure long-term continuous operation and adaptive interface display. S2 Eye Opening / Closing Recognition: Eye region localization is performed on each frame of the scene image (right eye prioritized, left eye regressed if failure occurs). The extracted eye ROIs are normalized and scaled, and input into a lightweight CNN classifier to output "Open / Close" judgments. Simultaneously, a short window temporal buffer is maintained to smoothly suppress transient phenomena such as blinking, providing a robust basis for subsequent "prolonged eye closure" recognition. S3 Iris Center Estimation and Gaze Direction Determination: Iris and corner eye key points are extracted based on FaceMesh regression. A geometric center and horizontal offset method is used for uncalibrated estimation of gaze direction. For example, when the offset directions of both eyes are consistent and exceed a set threshold, it is judged as "Looking Left / Right"; otherwise, it is considered "Looking Center." Stable gaze determination can still be obtained under natural light and slight posture changes. S4 Multimodal State Fusion and Manual Baseline Input: The system fuses "Presence Detection (Presence of Face / Facial Key Points)," "Long-Term Eye Closure Sequence Determination," and "Gaze Detachment" according to priority, outputting state results sequentially as "Off Duty → Long-Term Eye Closure → Lack of Focus → Focus." To ensure clear session boundaries and historical traceability, the system provides a baseline face input process with visualized progress. After several valid samplings, it enters the monitoring state, also reserving an interface for future expansion with automatic person replacement detection. S5 State Timeline Recording, Visualization, and Historical Retrospection: Real-time judgment results are incrementally presented in a timeline color band format and stored in a structured manner by camera and user ID, supporting retrieval, playback, and export. During long-term operation, the view supports "Back to Latest" and automatic follow-up, improving readability and operational efficiency. Compared with existing technologies, this solution integrates three core types of information—"in-situation, eye opening / closing, and line of sight"—under the constraints of visible light, no wearing, and no complex calibration. It also incorporates engineering-level features such as self-healing after line breakage, session management, and historical backtracking into a unified framework, significantly improving long-term stability, judgment accuracy, and maintainability, thus meeting the requirements for the implementation of non-intrusive monitoring.

[0070] Referring to the flowchart of another user status monitoring method shown in Figure 4, after the application starts, taking a three-camera instance as an example, the acquisition threads corresponding to the three cameras perform frame acquisition and status updates, and perform face detection, eye status, and gaze detection to determine whether the target user is focused, absent-minded, unfocused, or has had their eyes closed for an extended period. If any abnormality is detected, an alarm can be generated. The GUI main thread can periodically poll, copy the latest frames, and display them. In addition, the status history can be displayed in the form of a timeline based on the determined status of the target user. After the user clicks "Enter Face," facial feature vectors can be collected to generate the corresponding session baseline code. After the application is closed, the history can be saved and the acquisition threads corresponding to the cameras can be released. This solution can save the monitoring information of the target user collected by each camera, and the user can query it when clicking "Query History."

[0071] For ease of understanding, a specific embodiment is provided below. This embodiment uses a fixed three-camera deployment on a Windows platform. In this embodiment, the system is deployed in a Windows environment, using cv2.CAP_DSHOW to open three visible light USB cameras for parallel data acquisition. The software stack uses Python as the main control language, combined with OpenCV for video reading and writing and basic image processing. Mediapipe FaceMesh extracts facial key points and iris positions, Keras loads a lightweight eye CNN to perform open / closed eye classification, and face_recognition extracts face codes for conversation benchmarks. The front end uses PyQt5 to build the interface and interaction.

[0072] At the acquisition end, the target resolution for each camera can be set to 640×480, and the display layer maintains proportional scaling to prevent image distortion when the window size changes; the typical operating frame rate is approximately 30 fps. Eye classification uses a 24×24 single-channel ROI as input, and normalizes pixels to [0,1]. To suppress blink jitter and occasional false detections, the system maintains a sliding queue of length 10 to count the percentage of "closed eyes." For example, if the percentage exceeds 0.5 and the duration exceeds 1 second, it is judged as "long-term closed eyes." The gaze direction can be calculated based on FaceMesh to determine the horizontal offset between the iris center and the midpoint of the corner of the eye, and left and right off-axis is determined by a pixel threshold τ=5 px; at different resolutions or face scales, it can be switched to an equivalent implementation based on the eye width ratio threshold. The session baseline is entered through a "progress dialog box" that guides the user to complete 5 valid samples under the condition of "eyes open + looking straight ahead," and the average of the obtained face codes is used as the session baseline code. Once the entry is complete, the system enters the monitoring state and starts recording the timeline. The acquisition thread continuously monitors the device status. If a frame reading fails or the device disconnects, it enters a black screen position and attempts to reconnect at a cycle of approximately 3 seconds. After a successful reconnection, the timeline and status text are cleared, and the system waits for re-entry to avoid data mixing before and after the disconnection. The timeline adds color bands frame by frame with a fixed width (w=0.5 scene coordinate units) and updates the status text synchronously. Each time a session ends or the application is closed, the system creates three directories locally: history_1 / , history_2 / , and history_3 / to store the historical records of the corresponding cameras. The filenames are auto-incremented by the user number as user_{n}.json, and the content includes the camera number, user number, start time, frame-by-frame status sequence, and total session duration for easy retrieval and auditing.

[0073] The system operation process is explained below: After the application starts, the three cameras collect data in parallel, and the interface displays real-time images and a status bar. The target user clicks "Enter Face," and after completing 5 valid samples as prompted, the system automatically enters monitoring mode. Subsequently, the system outputs the status "Off Duty / Long-Term Eye Closure / Lack of Focus / Focus" for each frame of the scene image according to priority rules of on-site detection, long-term eye closure judgment, and gaze direction, and incrementally writes it to the timeline. If a disconnection occurs during operation, the interface goes black and periodically reconnects. After reconnection, the timeline is automatically cleared and the system waits for re-entry. When the monitoring personnel actively end the session or a new round of entry begins, the current session is immediately saved to the disk and the user number is incremented. Actual testing shows that under normal indoor lighting conditions and with slight changes in facial profile and posture, the system can stably output the timeline and status text. Historical queries can be quickly located and played back by camera and user number, meeting the traceability requirements for teaching duty and post-event auditing.

[0074] Without altering the overall approach, this solution enhances key components according to specific scenarios: First, it introduces automatic person-swapping detection, comparing the similarity between the current face encoding (corresponding to the specified session baseline encoding) and the session baseline (corresponding to the target session baseline encoding) using a cosine similarity threshold of 0.70 or an equivalent Euclidean distance threshold. If the similarity falls below the threshold, the current session is automatically archived and reset to a pending state. Second, regarding threshold adaptation, the gaze threshold is changed to an eye-width ratio metric to improve robustness under different resolutions and face scales, and the "long-term closed-eye" duration threshold is adaptively calculated based on the actual frame rate. Third, persistence is upgraded from JSON to SQLite / MySQL, enabling multi-condition retrieval and statistical reports based on time, status, user, and camera dimensions. Fourth, regarding algorithm replacement, RetinaFace or MTCNN can replace Haar to enhance the robustness of on-duty detection under low-light and side-face conditions, and lightweight variants of MobileNet / ResNet can replace the eye CNN to improve generalization. Fifth, cross-platform packaging is implemented via resource_path. The mechanism embeds models and cascading files, is compatible with packaging methods such as PyInstaller, and is easy to install and use on different hosts.

[0075] The aforementioned user status monitoring method involves a non-intrusive cognitive status monitoring system based on multi-channel video acquisition, facial key point regression, and a lightweight convolutional neural network (CNN) in natural scenes. This method eliminates the need for wearable hardware and complex calibration, enabling real-time determination of a target user's on-duty status, focused / distracted state, and prolonged eye-closed state. It generates a structured historical record in a timeline format, facilitating subsequent retrieval and backtracking. This approach is applicable to scenarios such as online teaching, smart office environments, driver assistance, and safety monitoring.

[0076] This solution aims to provide a non-intrusive cognitive state monitoring solution that can be deployed on general-purpose visible light cameras and general-purpose computing platforms, addressing issues such as the fragility of single-modal judgment, unstable engineering operation, and lack of historical management. This solution obtains a more stable input source through parallel acquisition from multiple cameras, uses facial keypoint regression and a lightweight CNN to complete eye opening and closing recognition, and estimates the gaze direction based on the geometric relationship between the iris center and the corner of the eye. Subsequently, a priority fusion rule is established with in-situ detection, prolonged eye closure judgment, and gaze deviation as core elements, outputting four states: absent from duty, prolonged eye closure, inattentiveness, and focusedness, explicitly reducing misjudgments caused by single signals. To balance low intrusion and session management, this solution provides a human-computer interaction process for manual baseline input, enabling monitoring to start with clear session boundaries and automatically form timeline-level structured records. At the engineering level, this solution introduces disconnection detection and periodic self-healing reconnection strategies, combined with adaptive display and lightweight computing paths, ensuring stable long-term operation and a real-time interactive experience. The above design achieves robust monitoring in natural scenes without relying on dedicated hardware and complex calibration, and reserves interfaces for related extensions (such as automatic person replacement detection, threshold adaptation, database storage, and more robust face / eye gaze model replacement), thus achieving an engineering balance between accuracy, usability, and scalability.

[0077] In summary, this solution has the following beneficial effects: 1. Refined recognition and robust noise reduction: By fusing multimodal priorities of "on duty - closed eyes - gaze", it effectively distinguishes situations such as being off duty, long-term closed eyes, short-term blinking and gaze shifting, reducing false alarms / missed alarms caused by the susceptibility of single-modal methods to lighting, side profile and occlusion.

[0078] 2. Long-term stability and engineering usability: Multi-threaded data acquisition and periodic self-healing reconnection ensure continuous input; automatic black screen and status reset in case of abnormality, and seamless return to monitoring state after recovery, suitable for long-term scenarios such as teaching / office work.

[0079] 3. Low-interference and low-cost deployment: It can run in real time on a regular PC (Personal Computer) by relying only on a general-purpose visible light camera and a lightweight CNN, without the need for head-mounted sensors or dedicated infrared equipment, making it easy to scale up.

[0080] 4. Traceable and easy to audit: Real-time results are stored as frame-level timelines and structured records, which can be retrieved and played back by camera and user number, meeting the traceability needs of teaching assessment, duty audit and behavior analysis.

[0081] 5. User-friendly and scalable: It provides baseline input with progress feedback and one-click history viewing, which not only lowers the threshold for use, but also reserves room for future expansion by introducing automatic person replacement detection, threshold adaptation, database storage, and more robust face / eye gaze model replacement.

[0082] In summary, this solution achieves non-intrusive cognitive state monitoring on general-purpose visible light cameras and general-purpose computing platforms through multimodal fusion of "on-duty detection—eye opening and closing—gaze direction," combined with interactive benchmark input, self-healing of disconnections, and timeline-level structured backtracking. This solution demonstrates continuity, robustness, and traceability in natural scenes, and reserves extension paths such as threshold adaptation, automatic personnel replacement, and algorithm and storage replacement, exhibiting good engineering feasibility and scalability.

[0083] This invention provides a user status monitoring device, as shown in Figure 5. The device includes: a collection module 50, used to collect scene images of a target scene through multiple cameras in response to a collection command; a judgment module 51, used to determine whether the target user's face and / or facial key points are detected in the current frame scene image collected by each camera for each frame scene image; a first determination module 52, used to determine that the target user is in an on-duty state if a face and / or facial key points are detected; an extraction module 53, used to extract the region of interest corresponding to the target eye based on the current frame scene image; a second determination module 54, used to determine whether the target user is in a long-closed-eye state based on the region of interest; a selection module 55, used to select specified key points from the facial key points if the target user is not in a long-closed-eye state; wherein the specified key points include: left eye iris index, right eye iris index, left eye corner index, and right eye corner index; and a third determination module 56, used to determine whether the target user is in a focused state based on the specified key points.

[0084] The aforementioned user status monitoring device can comprehensively determine the on-duty status, long-term closed-eye state, and focused state of the target user under constraints such as visible light, no wearing, and no complex calibration, thereby stably identifying the status of the monitored user in a natural environment.

[0085] Furthermore, the second determining module is also used for: preprocessing the region of interest to obtain a preprocessed region of interest; inputting the preprocessed region of interest into a pre-trained recognition model, and outputting the current eye state of the target eye through the recognition model; obtaining the historical eye state of the most recent preset number of times; calculating the closed eye ratio and the closed eye duration based on the current eye state and the historical eye state; if the closed eye ratio is greater than a preset ratio threshold and the closed eye duration is greater than a preset time threshold, determining that the target user is in a long closed eye state; if the closed eye ratio is not greater than the preset ratio threshold and / or the closed eye duration is not greater than the preset time threshold, determining that the target user is not in a long closed eye state.

[0086] Furthermore, there are multiple left-eye iris indices, multiple right-eye iris indices, multiple left-eye corner indices, and multiple right-eye corner indices; the third determining module is also used to: determine the center of the left iris based on multiple left-eye iris indices; determine the center of the right iris based on multiple right-eye iris indices; determine the center of the left eye based on multiple left-eye corner indices; determine the center of the right eye based on multiple right-eye corner indices; and determine whether the target user is in a focused state based on the center of the left iris, the center of the right iris, the center of the left eye, and the center of the right eye.

[0087] Furthermore, the third determining module is also used to: calculate the difference between the center of the left iris and the center of the left eye to obtain the left eye gaze offset value; calculate the difference between the center of the right iris and the center of the right eye to obtain the right eye gaze offset value; determine the left eye gaze offset direction based on the left eye gaze offset value; determine the right eye gaze offset direction based on the right eye gaze offset value; if the left eye gaze offset direction is the same as the right eye gaze offset direction, and the absolute values ​​of both the left and right eye gaze offset values ​​are greater than a preset offset threshold, determine that the target user is in a state of inattentiveness; if the left eye gaze offset direction is the same as the right eye gaze offset direction, and the absolute values ​​of both the left and right eye gaze offset values ​​are less than or equal to a preset offset threshold, determine that the target user is in a state of attentiveness.

[0088] Furthermore, the preset offset thresholds include: a left eye offset threshold and a right eye offset threshold; the left eye offset threshold and the right eye offset threshold are determined by: determining the left eye width based on multiple left eye corner indices; determining the right eye width based on multiple right eye corner indices; multiplying the left eye width by a preset ratio value to obtain the left eye offset threshold; multiplying the right eye width by a preset ratio value to obtain the right eye offset threshold.

[0089] Furthermore, the third determining module is also used to: determine that the target user is in a state of inattention if the left eye gaze deviation direction is the same as the right eye gaze deviation direction, the absolute value of the left eye gaze deviation value is greater than the left eye deviation threshold, and the absolute value of the right eye gaze deviation value is greater than the right eye deviation threshold; and determine that the target user is in a state of inattention if the left eye gaze deviation direction is the same as the right eye gaze deviation direction, the absolute value of the left eye gaze deviation value is less than or equal to the left eye deviation threshold, and the absolute value of the right eye gaze deviation value is less than or equal to the right eye deviation threshold.

[0090] Furthermore, the device is also used for: when the target user is in an open-eyed state and looking directly at the first camera among multiple cameras, acquiring multiple first images containing the target user's face through the first camera according to a preset acquisition number; extracting facial feature vectors from each first image; calculating the average value of the facial feature vectors in the multiple first images to obtain the target session baseline code corresponding to the target user; when the specified user is in an open-eyed state and looking directly at the first camera, acquiring multiple second images containing the specified user's face through the first camera according to a preset acquisition number; extracting facial feature vectors from each second image; calculating the average value of the facial feature vectors in the multiple second images to obtain the specified session baseline code corresponding to the specified user; calculating the similarity between the target session baseline code and the specified session baseline code; if the similarity is lower than a preset similarity threshold, determining that the target user and the specified user are different users; ending the monitoring process for the target user, saving the monitoring information corresponding to the target user based on the user ID of the target user and the camera ID of each camera, and starting the monitoring process for the specified user.

[0091] Furthermore, the device is also used to: determine that the second camera has entered a disconnected state if it is determined that the second camera among multiple cameras has failed to acquire scene images, or the scene images acquired by the second camera have failed to open; and control the second camera to reconnect at preset time intervals.

[0092] Furthermore, the monitoring information includes: for each camera, the monitoring status of the target user determined based on each frame of scene image captured by that camera; wherein, the monitoring status corresponding to each camera is presented in the form of a timeline color band; the monitoring status includes: focused state, unfocused state, long-term closed-eye state, and off-duty state.

[0093] The user status monitoring device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned user status monitoring method embodiment. For the sake of brevity, any parts not mentioned in the user status monitoring device embodiment can be referred to the corresponding content in the aforementioned user status monitoring method embodiment.

[0094] This invention also provides an electronic device, as shown in FIG6, which includes a processor 130 and a memory 131. The memory 131 stores machine-executable instructions that can be executed by the processor 130. The processor 130 executes the machine-executable instructions to implement the above-described user status monitoring method.

[0095] Furthermore, the electronic device shown in Figure 6 also includes a bus 132 and a communication interface 133, with the processor 130, the communication interface 133 and the memory 131 connected via the bus 132.

[0096] The memory 131 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 133 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network. The bus 132 may be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only a single bidirectional arrow is used in Figure 6, but this does not indicate that there is only one bus or one type of bus.

[0097] Processor 130 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 130 or by instructions in software form. Processor 130 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 131, and processor 130 reads the information in memory 131 and, in conjunction with its hardware, completes the steps of the method described in the foregoing embodiments.

[0098] This invention also provides a machine-readable storage medium storing machine-executable instructions. When these machine-executable instructions are invoked and executed by a processor, they cause the processor to implement the aforementioned user state monitoring method. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0099] The computer program product of the user status monitoring method and apparatus provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0100] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A user status monitoring method, characterized in that, The method includes: in response to a collection command, acquiring scene images of a target scene using multiple cameras; for each frame of scene image acquired by each camera, determining whether a target user's face and / or facial key points are detected in the current frame of scene image acquired by that camera; if the face and / or facial key points are detected, determining that the target user is in an on-duty state; extracting a region of interest corresponding to the target eye based on the current frame of scene image; determining whether the target user is in a state of long-closed eyes based on the region of interest; if the target user is not in the state of long-closed eyes, selecting a specified key point from the facial key points; wherein the specified key point includes: left eye iris index, right eye iris index, left eye corner index, and right eye corner index; determining whether the target user is in a focused state based on the specified key point.

2. The method according to claim 1, characterized in that, The step of determining whether the target user is in a long-term closed-eye state based on the region of interest includes: preprocessing the region of interest to obtain a preprocessed region of interest; inputting the preprocessed region of interest into a pre-trained recognition model, and outputting the current eye state of the target user through the recognition model; obtaining the historical eye state of the most recent preset number of times; calculating the closed-eye ratio and the closed-eye duration based on the current eye state and the historical eye state; if the closed-eye ratio is greater than a preset ratio threshold and the closed-eye duration is greater than a preset time threshold, determining that the target user is in a long-term closed-eye state; if the closed-eye ratio is not greater than the preset ratio threshold, and / or the closed-eye duration is not greater than the preset time threshold, determining that the target user is not in a long-term closed-eye state.

3. The method according to claim 1, characterized in that, There are multiple left-eye iris indices, multiple right-eye iris indices, multiple left-eye corner indices, and multiple right-eye corner indices. The step of determining whether the target user is in a focused state based on the specified key points includes: determining the center of the left iris based on the multiple left-eye iris indices; determining the center of the right iris based on the multiple right-eye iris indices; determining the center of the left eye based on the multiple left-eye corner indices; determining the center of the right eye based on the multiple right-eye corner indices; and determining whether the target user is in a focused state based on the left-eye iris center, the right-eye iris center, the left-eye center, and the right-eye center.

4. The method according to claim 3, characterized in that, The step of determining whether the target user is in a focused state based on the center of the left iris, the center of the right iris, the center of the left eye, and the center of the right eye includes: calculating the difference between the center of the left iris and the center of the right eye to obtain a left eye gaze offset value; calculating the difference between the center of the right iris and the center of the right eye to obtain a right eye gaze offset value; determining the left eye gaze offset direction based on the left eye gaze offset value; determining the right eye gaze offset direction based on the right eye gaze offset value; if the left eye gaze offset direction is the same as the right eye gaze offset direction, and the absolute values ​​of both the left and right eye gaze offset values ​​are greater than a preset offset threshold, the target user is determined to be in an unfocused state; if the left eye gaze offset direction is the same as the right eye gaze offset direction, and the absolute values ​​of both the left and right eye gaze offset values ​​are less than or equal to the preset offset threshold, the target user is determined to be in a focused state.

5. The method according to claim 4, characterized in that, The preset offset thresholds include: a left eye offset threshold and a right eye offset threshold; the left eye offset threshold and the right eye offset threshold are determined by: determining the left eye width based on multiple left eye corner indices; determining the right eye width based on multiple right eye corner indices; multiplying the left eye width by a preset ratio value to obtain the left eye offset threshold; multiplying the right eye width by the preset ratio value to obtain the right eye offset threshold.

6. The method according to claim 4, characterized in that, If the left eye gaze deviation direction is the same as the right eye gaze deviation direction, and the absolute values ​​of both the left and right eye gaze deviation values ​​are greater than a preset deviation threshold, the step of determining that the target user is in an unfocused state includes: if the left eye gaze deviation direction is the same as the right eye gaze deviation direction, the absolute value of the left eye gaze deviation value is greater than the left eye deviation threshold, and the absolute value of the right eye gaze deviation value is greater than the right eye deviation threshold, then the target user is determined to be in an unfocused state; if the left eye gaze deviation direction is the same as the right eye gaze deviation direction, and the absolute values ​​of both the left and right eye gaze deviation values ​​are less than or equal to the preset deviation threshold, the step of determining that the target user is in a focused state includes: if the left eye gaze deviation direction is the same as the right eye gaze deviation direction, the absolute value of the left eye gaze deviation value is less than or equal to the left eye deviation threshold, and the absolute value of the right eye gaze deviation value is less than or equal to the right eye deviation threshold, then the target user is determined to be in an unfocused state.

7. The method according to claim 1, characterized in that, The method further includes: when the target user is in an open-eyed state and looking directly at the first camera among the plurality of cameras, acquiring multiple first images containing the target user's face through the first camera according to a preset acquisition number; extracting facial feature vectors from each of the first images; calculating the average value of the facial feature vectors in the plurality of first images to obtain a target session baseline code corresponding to the target user; when a specified user is in an open-eyed state and looking directly at the first camera, acquiring multiple second images containing the specified user's face through the first camera according to the preset acquisition number; extracting facial feature vectors from each of the second images; calculating the average value of the facial feature vectors in the plurality of second images to obtain a specified session baseline code corresponding to the specified user; calculating the similarity between the target session baseline code and the specified session baseline code; if the similarity is lower than a preset similarity threshold, determining that the target user and the specified user are different users; ending the monitoring process for the target user; saving the monitoring information corresponding to the target user based on the user ID of the target user and the camera ID of each of the cameras; and starting the monitoring process for the specified user.

8. The method according to claim 1, characterized in that, The method further includes: if it is determined that the second camera among the plurality of cameras fails to acquire the scene image, or the scene image acquired by the second camera fails to open, determining that the second camera enters a disconnected state; and controlling the second camera to reconnect at a preset time interval.

9. The method according to claim 7, characterized in that, The monitoring information includes: for each camera, the monitoring status of the target user determined based on each frame of scene image captured by the camera; wherein, the monitoring status corresponding to each camera is presented in the form of a timeline color band; the monitoring status includes: focused state, unfocused state, long-term closed-eye state, and off-duty state.

10. A user status monitoring device, characterized in that, The device includes: an acquisition module for acquiring scene images of a target scene via multiple cameras in response to an acquisition command; a judgment module for determining whether a target user's face and / or facial key points are detected in the current frame scene image acquired by each camera for each frame scene image; a first determination module for determining that the target user is in an on-duty state if the face and / or facial key points are detected; an extraction module for extracting the region of interest corresponding to the target eye based on the current frame scene image; a second determination module for determining whether the target user is in a state of long-closed eyes based on the region of interest; a selection module for selecting a specified key point from the facial key points if the target user is not in the state of long-closed eyes; wherein the specified key point includes: left eye iris index, right eye iris index, left eye corner index, and right eye corner index; and a third determination module for determining whether the target user is in a focused state based on the specified key point.

Images