Methods, apparatuses, devices, media, products for detecting an eye-rubbing state

Abstract

This application relates to the field of computer technology, and particularly to a method, apparatus, device, medium, or product for detecting eye rubbing. The method includes: acquiring multiple consecutive frames of images; when performing eye rubbing detection on the i-th frame of the multiple frames, acquiring the coordinates of facial and hand key points of the target person; determining the eye rubbing radius based on the relative positional relationship between the facial key points; determining and marking the instantaneous state of the i-th frame based on the relationship between the distance between the coordinates of the eye key points and the coordinates of the hand key points and the eye rubbing radius; acquiring the instantaneous states of each frame within a sliding time window of the i-th frame, and calculating the percentage of images marked as eye rubbing within the sliding time window; if the percentage is greater than a preset percentage threshold, determining that the target person is in an eye rubbing state within the sliding time window. This method solves the problems of difficult training sample collection and high deployment costs when using a trained model for eye rubbing detection.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and in particular relates to a method, apparatus, device, medium, or product for detecting eye rubbing. Background Technology

[0002] In scenarios requiring focused monitoring, such as working, studying, or driving, eye-rubbing behavior is often used as an important indicator of a user's fatigue level or attention state.

[0003] In related technologies, a common approach to detect eye-rubbing behavior is to train a classification model. Specifically, a large number of image samples containing and not containing eye-rubbing behavior are collected in advance. After the samples are manually labeled, they are input into the model for training, allowing the model to learn the features of eye-rubbing behavior and thus output a classification result for whether eye-rubbing behavior exists on the input image frame.

[0004] However, the above approach has the following problems: Training the classification model requires a large number of samples, and the cost of sample collection and manual annotation is high, resulting in a long model training cycle. The classification model is computationally intensive, requiring high hardware computing power for deployment and incurring high implementation costs. Summary of the Invention

[0005] This application provides a method, apparatus, device, medium, and product for detecting eye rubbing, which can solve the problems of difficulty in collecting training samples and high deployment costs when using a trained model for eye rubbing detection.

[0006] In a first aspect, embodiments of this application provide a method for detecting eye rubbing, the method comprising: Acquire a series of consecutive frames of images, where at least one frame contains the face of the target person; When performing eye rubbing detection on the i-th frame of a multi-frame image, the coordinates of the facial key points and hand key points of the target person are obtained. The facial key points include the eye key points, and i is a positive integer. The radius of eye rubbing of the target person in the i-th frame image is determined based on the relative positional relationship between facial key points; Based on the relationship between the distance between the coordinates of the key points of the human eye and the coordinates of the key points of the hand and the radius of eye rubbing, the instantaneous state of the i-th frame image is determined and marked. The instantaneous state includes the eye rubbing state or the non-eye rubbing state. Get the instantaneous state of each frame within the sliding time window of the i-th frame image, and count the proportion of images marked as eye-rubbing state within the sliding time window. The sliding time window is a window with a preset duration, with the i-th frame image as the last frame. If the proportion of images marked as rubbing eyes within the sliding time window is greater than a preset proportion threshold, it is determined that the target person is rubbing their eyes within the sliding time window.

[0007] The beneficial effects of the first aspect are as follows: This method determines whether eye rubbing has occurred by detecting the distance between key points on the hand and key points on the eye. It leverages the spatial characteristic of eye rubbing as the hand key points entering the local area surrounding the eye key points, breaking down the complex problem of eye rubbing posture recognition into geometric distance calculations between these points, making the judgment rules clear and simple. The eye rubbing radius is determined by acquiring the distance between facial key points, allowing the eye rubbing judgment area to automatically change according to the size of the face in the image—the area increases when the person is closer to the camera and decreases when they are farther away. This overcomes the problem of missed or false detections that easily occur with fixed threshold schemes under different users and distances. During single-frame judgment, the pixel distance between each hand key point and the eye key point is calculated. If the distance between any hand key point and the eye key point is less than the eye rubbing radius, the frame is judged as an eye rubbing state. This judgment method, based on the actual spatial relationship between the hand and eye, takes into account key points at different positions on the hand, improving the accuracy of the recognition results. This method does not require training a separate eye-rubbing classification model, thus eliminating the need to collect and label a large number of eye-rubbing samples, resulting in a short development cycle and low deployment cost. Furthermore, given a single-frame judgment result, it uses a sliding time window to statistically analyze the proportion of eye-rubbing frames over a given period. Only when the proportion exceeds a threshold is an eye-rubbing state confirmed, effectively filtering out interference from occasional hand tremors and false single-frame detections, leading to more stable and accurate judgment results. It only requires a standard camera and the computing power of the terminal device itself, resulting in low deployment costs and meeting the needs of lightweight device deployment.

[0008] In one possible implementation, facial keypoints include nasal keypoints, and human eye keypoints include left eye keypoints and right eye keypoints. The radius of eye rubbing of the target person in the i-th frame image is determined based on the relative positional relationship between facial key points, including: Calculate the first distance between the nasal keypoint and the left eye keypoint; Calculate the second distance between the nose keypoint and the right eye keypoint; Calculate the third distance between the left eye keypoint and the right eye keypoint; The maximum value among the first distance, the second distance, and the third distance is determined as the reference distance; Obtain a preset radius coefficient, and determine the product of the radius coefficient and the reference distance as the rubbing radius of the target person in the i-th frame image.

[0009] This implementation determines the rub radius by acquiring the relative positional relationships between facial key points. This allows the rub detection area to automatically change according to the size of the face in the image; the area increases when the person is closer to the camera and decreases when the person is farther away. This overcomes the problem of missed or false detections that easily occur with fixed threshold schemes under different users and distances. It eliminates the need for separately training a rub classification model, avoiding the high costs of collecting and labeling a large number of rub samples, resulting in a short development cycle and low deployment cost.

[0010] In one possible implementation, there are multiple hand keypoints, and the human eye keypoints include left eye keypoints and right eye keypoints; Based on the relationship between the distance between the coordinates of key points on the human eye and the coordinates of key points on the hand and the radius of eye rubbing, the instantaneous state of the i-th frame image is determined and marked, including: Calculate the distance between the coordinates of each key hand point and the coordinates of each key eye point; If there exists at least one hand key point and the distance between the left eye key point or the right eye key point is less than or equal to the rubbing radius, determine and mark the instantaneous state of the i-th frame image as the rubbing state; If the distance between each key point on the hand and each key point on the eye is greater than the rubbing radius, the instantaneous state of the i-th frame image is determined and marked as the non-rubbing state.

[0011] In this implementation, single-frame determination is based on the pixel distance between hand and eye keypoints, utilizing the spatial geometric principle that eye-rubbing behavior manifests as hand keypoints entering the local area surrounding eye keypoints on the image plane. When the distance between any hand keypoint and eye keypoint is less than the eye-rubbing radius, the frame is determined to be an eye-rubbing state.

[0012] In one possible implementation, when performing eye-rubbing detection on the i-th frame of a multi-frame image, the coordinates of the facial and hand key points of the target person are obtained, including: Perform face detection on the i-th frame image to obtain the face region bounding box of the target person; Based on the position of the face region bounding box in the i-th frame image, human keypoint detection is performed on the i-th frame image to obtain the coordinates of multiple human keypoints of the target person and their corresponding confidence scores. The multiple human keypoints include facial keypoints and wrist keypoints. If the confidence level of the human eye key point in the facial key points reaches the preset eye confidence threshold, then the hand key points are detected in the i-th frame image based on the wrist key points to obtain the coordinates of the hand key points. The method also includes: If the confidence level of the human eye keypoint in the facial keypoint does not reach the preset eye confidence threshold, then the eye rubbing detection for the i-th frame image is cancelled.

[0013] In this implementation, face detection is combined with human body keypoint detection. Wrist keypoints are used to locate the hand search area, ensuring that the hand bounding box naturally belongs to the target person corresponding to the face, thus avoiding the problem of incorrect hand and face attribution in multi-person scenes. Simultaneously, the validity of eye keypoints is judged using confidence scores; subsequent steps are only executed when the detection results are reliable, avoiding erroneous judgments caused by model misdetections.

[0014] In one possible implementation, the method further includes: If no face region bounding box of the target person is detected in the i-th frame image, obtain the previous image where the face region bounding box of the target person was detected, and determine whether the interval between the previous image and the i-th frame image exceeds the preset anti-shake duration. If the time interval between the preceding image and the i-th frame does not exceed the preset anti-shake time interval, then the i-th frame is subjected to human key point detection based on the position of the face region bounding box in the preceding image. If the time interval between the preceding image and the i-th frame exceeds the preset anti-shake duration, then the eye rubbing detection for the i-th frame is cancelled.

[0015] In this implementation, by pre-setting the anti-shake duration and reusing the previous face region bounding box position for continued detection, the continuity of human body key point detection and eye rubbing judgment can still be maintained when the face is temporarily lost due to reasons such as hand occlusion.

[0016] In one possible implementation, the method further includes: When performing human key point detection on the i-th frame image, if no human key points of the target person are detected, the eye rubbing detection on the i-th frame image is cancelled.

[0017] In this implementation, the presence of key points on the human body is detected to determine whether the target person is still in the frame, avoiding invalid detection when the target person has left or has been off for a long time, thus achieving a balance between detection continuity and accuracy.

[0018] In one possible implementation, the instantaneous state of each frame within the sliding time window of the i-th frame is obtained, and the proportion of images marked as rubbing eyes within the sliding time window is counted, including: Obtain the actual cumulative duration of each frame within the sliding time window of the i-th frame image, and determine whether the actual cumulative duration has reached the preset duration; If the actual cumulative duration reaches the preset duration, the percentage of images marked as rubbing eyes within the sliding time window will be counted. The method also includes: If the actual cumulative time does not reach the preset time, the step of determining whether the target person is rubbing their eyes within the sliding time window will be canceled.

[0019] In this implementation, the anti-shake determination is based on the proportion of eye-rubbing frames within a sliding time window. Eye-rubbing is only confirmed when the proportion of images in the window exceeds a preset threshold. This effectively filters out momentary misjudgments caused by factors such as occasional hand tremors and single-frame detection errors in the model, improving the stability and reliability of the detection results. Furthermore, the calculation and final determination of the eye-rubbing frame proportion are only performed when the actual cumulative image duration within the sliding window reaches the preset duration; if the duration is insufficient, no results are output temporarily. This mechanism ensures a sufficient and effective sample size for statistical judgment, avoiding misjudgments due to insufficient data in the initial stage, and further improving the accuracy of the final detection results.

[0020] Secondly, embodiments of this application provide a device for detecting eye rubbing, comprising: The data acquisition module is used to acquire multiple consecutive frames of images, at least one of which includes the face of the target person. The detection module is used to obtain the coordinates of the facial key points and hand key points of the target person when performing eye rubbing detection on the i-th frame of a multi-frame image. The facial key points include the eye key points, and i is a positive integer. The data processing module is also used to determine the radius of eye rubbing of the target person in the i-th frame image based on the relative positional relationship between facial key points; The data processing module is also used to determine and mark the instantaneous state of the i-th frame image based on the relationship between the distance between the coordinates of the key points of the human eye and the coordinates of the key points of the hand and the radius of eye rubbing. The instantaneous state includes the eye rubbing state or the non-eye rubbing state. The data processing module is also used to obtain the instantaneous state of each frame image within the sliding time window of the i-th frame image, and to count the proportion of images marked as rubbing eyes within the sliding time window. The sliding time window is a window with a preset duration, with the i-th frame image as the last frame. The determination module is used to determine that the target person is rubbing their eyes within the sliding time window if the proportion of images marked as rubbing eyes within the sliding time window is greater than a preset proportion threshold.

[0021] Thirdly, this application also provides an electronic device. The electronic device includes a memory, one or more processors, and a computer program stored in the memory and executable on the processor. The electronic device executes the computer program to implement any of the implementations of the first aspect described above.

[0022] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the method of any of the implementations of the first aspect described above.

[0023] Fifthly, this application also provides a computer program product, including a computer program and a method for implementing any of the implementation methods of the first aspect when the computer program is executed by a processor.

[0024] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect above, and will not be repeated here. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of a method for detecting eye rubbing state provided in an embodiment of this application; Figure 2 This is a schematic diagram of a face region bounding box provided in an embodiment of this application; Figure 3 This is a schematic diagram of a palm region frame provided in an embodiment of this application; Figure 4 This is a schematic diagram of the distribution of key hand points provided in an embodiment of this application; Figure 5 This is a schematic diagram of an eye-rubbing determination area based on the eye-rubbing radius provided in an embodiment of this application; Figure 6 This is a structural block diagram of a device for detecting eye rubbing state provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] This solution provides a method for detecting eye rubbing, addressing the high cost of training samples and demanding requirements on terminal devices when deploying classification models for eye rubbing detection in related technologies. It detects eye rubbing behavior by leveraging the positional relationship between the hand and eyes during rubbing, eliminating the need to train a dedicated eye rubbing classification model and avoiding the costs of sample collection and model training. By adaptively determining the eye rubbing radius based on facial key points, the judgment area varies with individual face size and shooting distance, overcoming the poor generalization problem of fixed thresholds. Anti-shake judgment is performed based on the proportion of eye rubbing within a sliding time window, effectively filtering out false detections in single frames and improving the stability of detection results. A short-term face missing detection mechanism maintains the continuity of judgment even when the face is obscured by a hand.

[0028] This solution can be widely applied to various smart hardware scenarios that require monitoring user eye-rubbing behavior (such as mobile phone selfie detection, classroom behavior recognition, workstation action monitoring, and driving status monitoring) and have cameras and edge computing capabilities. For example, in a driving scenario, a camera deployed in the vehicle captures driver images in real time. This solution performs eye-rubbing detection on each frame of the image, obtains the single-frame detection result, and marks it. Combining the marked results of each frame of the image within a time window, continuous detection and recognition of the eye-rubbing state is achieved, serving as an auxiliary basis for judging fatigue driving and providing intervention reference for the vehicle's active safety system. When the driver is detected to be rubbing their eyes within a certain window, a fatigue intervention command (such as reducing vehicle speed) is output.

[0029] For example, in the scenario of a smart desk lamp, the built-in or external camera of the desk lamp can collect the student's facial images in real time. The images are then processed by this solution to identify whether the student is rubbing their eyes. The frequency and duration of eye rubbing are used to determine the student's level of fatigue and concentration. Once signs of fatigue or distraction are detected, the smart desk lamp can automatically adjust the brightness and color temperature of the light, or guide the student to rest through voice prompts, thereby improving learning efficiency and vision protection.

[0030] The specific implementation method of this solution will be described in detail below.

[0031] Figure 1 This is a flowchart of a method for detecting eye rubbing state provided by an exemplary embodiment of this application. The method can be applied to a terminal device that can be connected to a camera to acquire video frames / image frames captured by the camera in real time and apply the method to them. This application does not limit the scope of the method. The method includes the following steps.

[0032] Step 101: Acquire multiple consecutive frames of images.

[0033] Among these multiple images, at least one image contains the face of the target person.

[0034] This embodiment uses a smart desk lamp as an example for specific illustration. The smart desk lamp includes a chip with computing power. In this embodiment, the smart desk lamp can serve as a terminal device for executing the method. In this scenario, the smart desk lamp can be placed on a desk to provide lighting for the target person when they are in front of the desk. The external or built-in camera of the smart desk lamp can continuously capture multiple consecutive frames of images containing the user (i.e., the target person) at a preset frame rate (e.g., 15 to 30 frames per second, i.e., capturing 15 to 30 frames of images per second).

[0035] Each time an image frame is acquired, the method in the subsequent steps of this embodiment is used to analyze the image frame in real time, thereby realizing the dynamic recognition and tracking of the user's eye-rubbing behavior.

[0036] In this embodiment, the target person is a student sitting in front of a desk. The image of the area in front of the desk is captured by a camera built into or connected to the desk lamp, and a face detection model is used to detect faces in the image to lock the user's face area, which serves as the basis for subsequent key point detection and eye rubbing judgment.

[0037] As mentioned above, the method provided in this application is not limited to smart desk lamp scenarios, but can also be applied to driving scenarios, online classes, classroom behavior recognition, workstation action monitoring and other scenarios, and can be executed by devices with camera and edge computing capabilities.

[0038] When applied to other scenarios mentioned above, or when the image to be detected contains multiple target individuals, a face detection model can be used to detect each frame of the image to obtain a bounding box for each face region. Based on each bounding box, a local image of the face and body of the target individual corresponding to that bounding box is cropped out. For each local image, the subsequent key point detection, eye-rubbing radius determination, instantaneous state judgment, and sliding window anti-shake process of this embodiment are performed separately, and eye-rubbing state recognition and subsequent intervention decisions are made independently for each target individual.

[0039] For the smart desk lamp scenario used in the illustration, since the target person is usually a student sitting at a desk, the focus is on recognizing and responding to this single target. If multiple target persons appear in the image simultaneously, the same processing procedure is performed on each face area as described above. The implementation principle for other non-desk lamp scenarios is the same as the core idea of ​​the embodiments of this application, and will not be elaborated here.

[0040] It should be noted that eye-rubbing detection is based on face detection. If no face is detected in a certain frame, it means that the target person may have left the camera's range, or the entire face is largely obscured, making meaningful eye-rubbing detection impossible. Therefore, step 101 of this solution requires that at least one frame out of multiple frames contains the target person's face to ensure the basic input conditions for the detection process.

[0041] Step 102: When performing eye rubbing detection on the i-th frame of the multi-frame image, obtain the coordinates of the facial key points and hand key points of the target person.

[0042] Here, facial key points include eye key points, and i is a positive integer. The i-th frame image is used to refer to any frame image in the multi-frame image. That is, for each captured frame image, the processing flow of steps 102 to 106 is executed.

[0043] This scheme utilizes the spatial relationship of the eye-rubbing behavior as the proximity between the hand and the eye on the image plane. By obtaining the coordinates of facial key points and hand key points, the relative positions between the two are further analyzed to determine whether the current frame is in the state of eye-rubbing.

[0044] The detection principle of this method is briefly introduced as follows: First, the face is located to determine the position of the target person in the image. Based on this, key points of the human body are detected, obtaining the positions of wrist key points and eye key points. The wrist key points are used to locate the hand region, and the eye key points are used to locate the eye region. Further, the precise coordinates of the hand are obtained through hand key point detection, providing a data basis for calculating the distance between the hand and the eye. Since the eye rubbing detection requires a certain level of accuracy in the position between the hand and the eye, if the detection results of the eye key points are unreliable, the subsequent judgment based on the hand-eye distance will be meaningless. Therefore, after obtaining the human body key points, the confidence level of the eye key points needs to be judged, and subsequent steps are only performed if the detection results are reliable. The key points in this application can also be referred to as feature points.

[0045] Optionally, face detection can be performed on the i-th frame image to obtain the face region bounding box of the target person.

[0046] For example, a lightweight face detection model (such as the RetinaFace model) is used to detect faces in the i-th frame image. This model can detect faces from the input image and output the position and size of the face region bounding box, as well as five key points of the face (left eye, right eye, tip of the nose, left corner of the mouth, and right corner of the mouth) and a similarity score. The face region bounding box is the rectangular box defining the face region after the face detection model recognizes the face, and this box can contain most of the main areas of the face. Whether it is a frontal, side, or tilted face, the face detection model can accurately select the position and size of the face.

[0047] Indicative, such as Figure 2 As shown, Figure 2 This is a schematic diagram of a face region bounding box. Figure 2 In the image, the face of the target person 200 is outlined using a face region box 201.

[0048] Based on the position of the face region bounding box in the i-th frame image, human keypoint detection is performed on the i-th frame image to obtain the coordinates of multiple human keypoints of the target person and their corresponding confidence scores. The multiple human keypoints include facial keypoints and wrist keypoints.

[0049] Specifically, the face region bounding box is expanded outwards by a certain proportion to obtain a candidate region containing the upper body or full body of the target person. Then, the candidate region is cropped from the original image of the i-th frame to obtain the human body sub-image.

[0050] Optionally, when expanding the face region bounding box outwards by a certain ratio, the expansion ratios in the horizontal and vertical directions can be different. For example, the horizontal expansion can be 0.2 times and the vertical expansion can be 0.5 times, to ensure that the human body sub-image can completely contain the upper body and arm areas of the target person. The boundary of the expanded candidate region must be limited to the edge range of the i-th frame image.

[0051] By inputting a human body sub-image into a trained human keypoint detection model (such as the HRNet (High-Resolution Network) model in the field of deep learning pose estimation), the keypoint locations and confidence scores of 17 parts of the body can be obtained after inference by the human keypoint detection model.

[0052] The confidence score is the reliability rating given by the model for each detected human keypoint, used to characterize the reliability of the keypoint's location. A higher confidence score indicates that the model's judgment of the keypoint's location is more reliable; a lower confidence score indicates that the model's judgment of the keypoint's location is more uncertain, and may be affected by factors such as occlusion, pose changes, or image quality.

[0053] The 17 key points of the human body include: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0054] The key points used in this solution include facial key points (left eye key point, right eye key point, nose key point) and wrist key points (left wrist key point, right wrist key point). Key points such as left shoulder, right shoulder, left elbow, and right elbow are used to help determine the hand search area.

[0055] If the confidence level of the human eye keypoint in the facial keypoints reaches the preset eye confidence threshold, then the hand keypoints in the i-th frame image are detected based on the wrist keypoints to obtain the coordinates of the hand keypoints.

[0056] The preset eye confidence threshold can be set according to the actual application scenario and model accuracy, for example, it can be set to 0.5 or 0.6. When the confidence of the human eye key points reaches this threshold, it indicates that the eye position detection result is reliable, and subsequent hand key point detection and distance calculation can be performed.

[0057] In one optional embodiment, the detection of key hand points is implemented as follows.

[0058] First, based on the wrist and elbow keypoints obtained from human body keypoint detection, an adaptive hand detection region is generated. Specifically, based on the distance between the wrist and elbow keypoints on the same side (i.e., forearm length) and the coordinates of the wrist keypoint, a hand search region is generated near the wrist.

[0059] For example, using the wrist as the center and the distance from the wrist to the elbow as the reference length, a rectangular search area is obtained by expanding outwards according to a certain ratio. The size of this search area will adaptively adjust as the size of the target person in the image changes; the area becomes larger when the target person is closer to the camera and smaller when the target person is farther away from the camera.

[0060] Then, the hand search region is cropped from the original image of the i-th frame to obtain a hand sub-image. This hand sub-image is input into a trained palm detection model (such as a hand bounding box detection model). After model inference, the position and size of the palm bounding box, as well as 7 palm keypoints, can be obtained. The 7 palm keypoints include five finger root keypoints, palm center keypoints, and palm root keypoints.

[0061] Since the hand search region is generated based on the wrist and elbow keypoints of the target person, and these keypoints are obtained from the human keypoint detection guided by the face region bounding box of the target person, the hand bounding box detected in this way naturally belongs to the same target person as the face region bounding box. This ensures that in multi-person scenes, each face region bounding box can correspond to the hand bounding box belonging to the same target person, avoiding misidentification of other people's hands in the image as the current target person's hand.

[0062] Indicative, such as Figure 3 As shown, Figure 3 This is a schematic diagram of a palm area.

[0063] The rectangle surrounding the key point P1 on the wrist of the target character 300 is the palm area box 301.

[0064] In some embodiments, since the palm in the image captured by the camera may be in different postures such as tilting or rotating, directly using the original palm bounding box image for keypoint detection will affect the detection accuracy. Therefore, after obtaining the palm bounding box and 7 palm keypoints, the original palm bounding box is further processed by directional cropping and perspective transformation based on the 7 palm keypoints. Perspective transformation is the process of transforming the skewed palm image captured by the camera into a standard palm image with the palm facing upwards using mathematical methods.

[0065] Specifically, the deflection angle and tilt of the palm are determined based on seven key points of the palm, the perspective transformation matrix is ​​calculated, and the image of the palm region is aligned to a predetermined size. After the above processing, the region of interest (i.e., the hand region bounding box) of the palm region is aligned into a standard square image of 224×224 pixels, resulting in the aligned palm region image.

[0066] The aligned palm region image is input into a trained hand keypoint detection model (e.g., the MediaPipeHands model (a device-based real-time hand tracking model)). After model inference, the position coordinates and corresponding confidence scores of 21 key points of the hand can be obtained. In other words, the coordinates and corresponding confidence scores of each key point of the hand are obtained.

[0067] Indicative, such as Figure 4 As shown, Figure 4 This is a schematic diagram showing the distribution of key points on a hand.

[0068] Taking a 400-degree palm facing upwards as an example, Figure 4 The CCP includes 21 key points of the hand: 20 key points of finger joints P20 and one key point of the palm base P0.

[0069] The above steps obtain the coordinates and corresponding confidence scores of each facial keypoint. If the confidence score of the human eye keypoint among the facial keypoints does not reach the preset eye confidence threshold, the eye rubbing detection for the i-th frame image is cancelled. That is, subsequent steps 103 to 106 are not executed.

[0070] In actual detection, there may be instances where the face region bounding box is temporarily lost due to the target person turning around instantly, shifting their head, or having a large area of ​​their hand obscuring the view. Alternatively, the target person may leave the camera's field of view, which will also cause the face region bounding box to be lost. The inability to detect the face region bounding box means that subsequent eye-rubbing detection cannot be performed. However, considering that the target person's departure or occlusion usually does not occur instantaneously within a single frame, this method incorporates an image stabilization mechanism. This mechanism allows the reuse of previously detected face positions within a limited time window to maintain detection continuity.

[0071] Optionally, if the face region bounding box of the target person is not detected in the i-th frame image, the preceding image in which the face region bounding box of the target person was detected is obtained, and it is determined whether the interval between the preceding image and the i-th frame image exceeds the preset anti-shake duration.

[0072] If the time interval between the preceding image and the i-th frame does not exceed the preset anti-shake time interval, then human key point detection is performed on the i-th frame based on the position of the face region bounding box in the preceding image.

[0073] Considering that the actual spatial location of the target person and the camera's acquisition angle usually do not change significantly within a short time interval, the position of the face region bounding box detected in the previous image can approximately represent the target person's current position in the image, provided the interval duration does not exceed the preset stabilization duration. Based on this judgment, the face region bounding box position in the previous image can be reused to guide the detection of human key points in the i-th frame image (the current frame image).

[0074] For example, assume the preset stabilization duration is 0.5 seconds. If the acquisition time corresponding to the i-th frame is t1, and no face region bounding box of the target person is detected in this frame, then the previous image in which the face region bounding box of the target person was detected is acquired, and the acquisition time of this previous image is t0. Calculate the time difference Δt = t1 - t0 between t1 and t0. If Δt = 0.1 seconds, which does not exceed the preset stabilization duration of 0.5 seconds, then based on the position of the face region bounding box in the previous image, human keypoint detection is performed in the corresponding area of ​​the i-th frame. Since the time interval is short, the position of the target person has not moved significantly, and this reused position can basically cover the current area of ​​the target person, thereby ensuring the continuity of detection.

[0075] If the time interval between the preceding image and the i-th frame exceeds the preset anti-shake duration, then the eye rubbing detection for the i-th frame is cancelled.

[0076] If the preset stabilization time exceeds the limit, it means that the target person's face bounding box has not been detected for a considerable period, and the target person may have left or moved away from the camera's range for an extended period. In this case, the face bounding box position in the previous image can no longer effectively represent the target person's actual position at the current moment, and performing human keypoint detection and subsequent eye-rubbing detection based on this position is unreliable. Continuing to use this position for human keypoint detection is meaningless. Furthermore, if the target person has been in a state where their face is not visible for a long time, continuing to detect whether they have rubbed their eyes is also meaningless. Therefore, in the above situations, eye-rubbing detection for the i-th frame image is cancelled.

[0077] Optionally, when performing human key point detection on the i-th frame image, if no human key points of the target person are detected, then the eye rubbing detection on the i-th frame image is cancelled.

[0078] Regardless of whether the face region bounding box comes from the face detection result of the current frame or is reused from the face region bounding box position in the previous image, it must satisfy the requirement of being able to detect the body (human key points) of the target person in the i-th frame image.

[0079] Human keypoint detection is the intermediate step connecting face localization and hand localization. If no human keypoints are detected, it means that the target person's body parts cannot be located within the candidate region of the i-th frame image. In this case, even if a face region bounding box exists, it is impossible to find the wrist keypoint belonging to the target person through human skeleton association, thus preventing subsequent hand keypoint detection and hand-eye distance calculation. Therefore, if human keypoint detection fails, it is determined that the current frame does not meet the valid input conditions for eye rub detection, and eye rub detection for the i-th frame image is canceled.

[0080] Step 103: Determine the rubbing radius of the target person in the i-th frame image based on the relative positional relationship between facial key points.

[0081] Among them, facial key points include nasal key points, and human eye key points include left eye key points and right eye key points.

[0082] The core of eye-rubbing behavior detection is determining whether the distance between key points on the hand and key points on the eye is close enough. Since the individual facial sizes of different target persons are different, and the facial size of the same target person in the image is also different at different distances, it is necessary to determine an eye-rubbing radius that can adaptively change with facial scale so that the judgment area can be adapted to different target persons and different shooting distances.

[0083] Optionally, calculate the first distance between the nose keypoint and the left eye keypoint; calculate the second distance between the nose keypoint and the right eye keypoint; calculate the third distance between the left eye keypoint and the right eye keypoint; determine the maximum value among the first distance, the second distance, and the third distance as the reference distance; obtain a preset radius coefficient, and determine the product of the radius coefficient and the reference distance as the rubbing radius of the target person in the i-th frame image.

[0084] Because there is a stable proportional relationship between the local area around the eyes and facial key points, the interocular distance between the left and right eye key points, the distance between the left and right eye key points and the nose key point, and the distance between the right and right eye key points and the nose key point can all reflect the scale of the eye area in the image. Taking the maximum value of the above three as the reference distance can ensure that the rubbing radius covers the farthest area around the eyes.

[0085] For example, the radius coefficient is a fixed value, set to 1 / 3. In this case, the formula for calculating the eye-rubbing radius R is: R = reference distance D_max / 3, where the reference distance D_max = max(first distance D_1, second distance D_2, third distance D_3).

[0086] R is measured in pixels. In the current frame image, the circular area centered on the key points of the human eye and with R as the radius is the area for detecting eye rubbing. The size of this area adapts to the facial scale: when the target person is closer to the camera, D_max is larger, and R is correspondingly larger; when the target person is farther from the camera, D_max is smaller, and R is correspondingly smaller.

[0087] Among them, the key point of the left eye corresponds to one eye rubbing judgment area, and the key point of the right eye corresponds to another eye rubbing judgment area.

[0088] Indicative, such as Figure 5 As shown, Figure 5 This is a schematic diagram of the eye-rubbing judgment area determined based on the radius of eye-rubbing.

[0089] Figure 5 In the diagram, the key point Pl of the left eye of the target character 500 corresponds to the rubbing judgment area 501 of the left eye, and the key point Pr of the right eye of the target character 500 corresponds to the rubbing judgment area 502 of the right eye.

[0090] In some embodiments, if the calculated eye-rubbing radius R is less than a preset minimum pixel threshold, then R is assigned the value of that minimum pixel threshold. For example, the preset minimum pixel threshold is set to 12 pixels, that is: if R < 12, then R = 12.

[0091] The purpose of setting a minimum pixel threshold is that when the target person is far from the camera, the distance between facial key points may become very small. The R value obtained after multiplying by the radius coefficient may only be a few pixels, which is much smaller than the coordinate positioning error of the hand key point detection model (usually 2 to 3 pixels) and the pixel width occupied by the fingers in the image. If the eye-rubbing radius is too small, even if the target person is actually rubbing their eyes, the hand key points may fall outside the radius range due to slight deviations, resulting in missed detection. By setting a minimum threshold of 12 pixels, it can be ensured that the eye-rubbing radius can still cover the eye area in long-distance scenes, maintaining the effectiveness of detection.

[0092] Considering the differences in the physiological structure of the eyes of different target individuals, such as the varying lengths of the palpebral fissure (the horizontal width when the eyes are open), individuals with larger palpebral fissures will have a greater distance between the key points of their hand and the key points of their eyes when rubbing their eyes, while those with smaller palpebral fissures will have the opposite effect. Therefore, in some embodiments, the radius coefficient can be individually and adaptively adjusted according to the size of the target individual's eyes to improve the accuracy of the eye-rubbing radius adaptation for different target individuals.

[0093] Optionally, face detection is performed on the i-th frame image to obtain the face region bounding box of the target person; eye region detection is performed on the image within the face region bounding box to obtain the eye detection bounding box; the radius coefficient is determined based on the ratio of the eye detection bounding box to the face region bounding box, and the radius coefficient is positively correlated with the ratio.

[0094] The eye detection box here is for a single eye. Since the size of a person's eyes is almost the same, only the eye detection box for a single eye (left or right eye) is needed here.

[0095] For example, let the width of the eye detection bounding box be w1, and the width of the face region bounding box be w2. Calculate the ratio r = w1 / w2 between the eye detection bounding box and the face region bounding box. This ratio reflects the proportion of the target person's eye size relative to their face size. The radius coefficient k can be calculated using the following formula: k = r × a.

[0096] Where 'a' is the preset calibration coefficient, and 'r' is the ratio of the eye detection box to the face region box, with a typical value of 5 / 3.

[0097] Based on this, the formula for calculating the radius R of eye rubbing can be: R = D_max × k = D_max × r × a.

[0098] Where D_max is the reference distance (i.e., the maximum value among the first distance, the second distance, and the third distance), and k is the adaptively adjusted radius coefficient.

[0099] Through the above method, the eye-rubbing radius can be adaptively adjusted based on the facial dimensions and individual eye characteristics of the target person. This takes into account both changes in shooting distance and individual physiological differences between different users, further improving the accuracy of eye-rubbing detection.

[0100] Step 104: Determine and mark the instantaneous state of the i-th frame image based on the relationship between the distance between the coordinates of the key points of the human eye and the coordinates of the key points of the hand and the radius of eye rubbing.

[0101] The instantaneous state includes either an eye-rubbing state or a non-eye-rubbing state. It's important to note that the instantaneous state is not a semantic label indicating the actual posture of the target person in the image. Rather, it's a preliminary judgment result extracted from the geometric distance features between hand and eye key points in a single frame. It's used only as a marker for subsequent statistical judgments within a sliding time window. In other words, this instantaneous state is an intermediate result marker for the current frame of the image, not a final confirmation of the target person's eye-rubbing behavior.

[0102] Among them, there are multiple key points for the hand, and key points for the human eye include key points for the left eye and key points for the right eye.

[0103] Calculate the distance between the coordinates of each key point on the hand and the coordinates of each key point on the eye.

[0104] For each hand, obtain the coordinates of its 21 hand keypoints, and calculate the distance from each hand keypoint to the left eye keypoint and the distance to the right eye keypoint. That is, each hand corresponds to 21 × 2 = 42 sets of distances, which constitute a distance set. If both the left and right hands of the target person are detected in the i-th frame image, then the left and right hands are calculated independently, with each hand corresponding to 42 sets of distances, for a total of 84 sets of distances for both hands.

[0105] If there exists at least one hand keypoint and the distance between the left eye keypoint or the right eye keypoint is less than or equal to the rubbing radius, then the instantaneous state of the i-th frame image is determined and marked as the rubbing state.

[0106] In other words, if any one of the 21 key points of any hand (left or right) has a distance to the key point of the left or right eye that is less than or equal to the eye-rubbing radius, then the instantaneous state of the current frame is determined to be the eye-rubbing state.

[0107] As mentioned earlier, the eye-rubbing radius can be used to determine the eye-rubbing detection area. When any key point of the hand falls into any eye-rubbing detection area, the instantaneous state of that frame image is determined to be an eye-rubbing state.

[0108] If the distance between each key point on the hand and each key point on the eye is greater than the rubbing radius, the instantaneous state of the i-th frame image is determined and marked as the non-rubbing state.

[0109] The left and right hands independently perform the above determination process. As long as either hand meets the conditions for the eye-rubbing state, the instantaneous state of that frame is the eye-rubbing state.

[0110] By using the above method, single-frame determination based on the fine distance between hand key points and eye key points can accurately reflect the spatial positional relationship between the hand and eye in the current frame, providing a reliable frame-level label for statistical determination within the subsequent sliding time window.

[0111] Step 105: Obtain the instantaneous state of each frame within the sliding time window of the i-th frame image, and count the proportion of images marked as rubbing eyes within the sliding time window.

[0112] The sliding time window is a window with a preset duration, where the i-th frame is the last frame.

[0113] For each frame of an image, there exists a sliding time window with that frame as the last frame. For example, assuming the time interval between two adjacent frames is Δt (e.g., when the camera captures frames at a frame rate of 30 frames / second, Δt = 1 / 30 seconds), and the preset duration is T, then the number of frames N contained in this sliding time window is the preset duration divided by the frame interval, rounded down, and then 1 (including the current frame), that is: N = floor(T / Δt) + 1.

[0114] For example, if the preset duration T is 5 seconds and the frame interval Δt is 1 / 30 seconds, then N = floor(5 / (1 / 30)) + 1 = floor(150) + 1 = 151 frames, that is, the sliding time window includes the current frame and the 150 frames before it, for a total of 151 frames.

[0115] Eye rubbing is a process that lasts for a certain period of time. The instantaneous state judgment of a single frame may be misjudged due to factors such as occasional hand tremors or model detection errors. Only by accumulating a sufficient number of continuous frame data can the overall positional relationship between the target person's hand and eyes during that time period be reflected, thereby avoiding misreporting occasional misjudgments of a single frame as actual eye rubbing.

[0116] Optionally, the actual cumulative duration of each frame within the sliding time window of the i-th frame image is obtained, and it is determined whether the actual cumulative duration has reached the preset duration.

[0117] If the actual cumulative duration reaches the preset duration, the percentage of images marked as rubbing eyes within the sliding time window is counted. That is, the ratio of the number of images n marked as rubbing eyes to the total number N within the sliding time window is the percentage of images.

[0118] In some embodiments, if the actual cumulative duration does not reach the preset duration, the step of determining whether the target person is rubbing their eyes within the sliding time window is canceled.

[0119] When the actual cumulative duration of the sliding time window is insufficient, the amount of data within the window is insufficient to support reliable statistical judgment, and therefore no judgment result is output.

[0120] Step 106: If the proportion of images marked as rubbing eyes within the sliding time window is greater than a preset proportion threshold, it is determined that the target person is in a rubbing eye state within the sliding time window.

[0121] For example, assuming the preset duration of the sliding time window is 5 seconds and the camera captures frames at a rate of 30 frames per second, the sliding time window contains a total of 151 frames. The preset percentage threshold is 20%. If 40 of these 151 frames are marked as an instantaneous state of rubbing eyes, then the percentage of images in the rubbing eye state is 40 / 151≈26.5%, which is greater than the preset percentage threshold of 20%. Therefore, it is determined that the target person is in a state of rubbing eyes within this sliding time window.

[0122] In some embodiments, if the proportion of images marked as rubbing eyes within the sliding time window is less than or equal to a preset proportion threshold, it is determined that the target person is in a non-rubbing eye state within the sliding time window.

[0123] For example, continuing from the previous example, if only 20 of the 151 frames are marked as the eye-rubbing state, then the proportion of the eye-rubbing state images is 20 / 151≈13.2%, which is less than the preset proportion threshold of 20%. Therefore, it is determined that the target person is not in the eye-rubbing state within the sliding time window.

[0124] By using the above method, eye-rubbing behavior is only confirmed when the proportion of images in the sliding time window exceeds a preset threshold. This effectively filters out momentary misjudgments caused by factors such as occasional hand tremors and single-frame detection errors of the model, avoiding the misreporting of brief non-eye-rubbing actions as eye-rubbing behavior, and improving the stability and reliability of eye-rubbing state detection results.

[0125] In summary, the method for detecting eye rubbing provided in this application determines whether eye rubbing has occurred by detecting the distance relationship between key points on the hand and key points on the eye. It leverages the spatial characteristic of eye rubbing as the hand key points entering the local area surrounding the eye key points, thus decomposing the complex problem of eye rubbing posture recognition into the calculation of geometric distances between these points, making the judgment rules clear and simple. By obtaining the distance between facial key points to determine the eye rubbing radius, the eye rubbing judgment area automatically changes according to the size of the face in the image; the area increases when the person is closer to the camera and decreases when they are farther away, overcoming the problem of missed or false detections caused by fixed threshold schemes under different users and distances. During single-frame judgment, the pixel distance between each hand key point and the eye key point is calculated. When the distance between any hand key point and the eye key point is less than the eye rubbing radius, the frame is determined to be in an eye rubbing state. This judgment method, based on the actual spatial relationship between the hand and eye, takes into account key points at different positions on the hand, improving the accuracy of the recognition results. This method does not require training a separate eye-rubbing classification model, thus eliminating the need to collect and label a large number of eye-rubbing samples, resulting in a short development cycle and low deployment cost. Furthermore, given a single-frame judgment result, it uses a sliding time window to statistically analyze the proportion of eye-rubbing frames over a given period. Only when the proportion exceeds a threshold is an eye-rubbing state confirmed, effectively filtering out interference from occasional hand tremors and false single-frame detections, leading to more stable and accurate judgment results. It only requires a standard camera and the computing power of the terminal device itself, resulting in low deployment costs and meeting the needs of lightweight device deployment.

[0126] Corresponding to the method for detecting eye rubbing in the above embodiments, Figure 6 A structural block diagram of a device for detecting eye rubbing states provided in an embodiment of this application is shown. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0127] Reference Figure 6 The device 600 includes: The data acquisition module 610 is used to acquire multiple consecutive frames of images, wherein at least one frame of the multiple frames includes the face of the target person. The detection module 620 is used to obtain the coordinates of the facial key points and hand key points of the target person when performing eye rubbing detection on the i-th frame of a multi-frame image. The facial key points include the eye key points, and i is a positive integer. The data processing module 630 is also used to determine the rubbing radius of the target person in the i-th frame image based on the relative positional relationship between facial key points; The data processing module 630 is also used to determine and mark the instantaneous state of the i-th frame image based on the relationship between the distance between the coordinates of the key points of the human eye and the coordinates of the key points of the hand and the radius of eye rubbing. The instantaneous state includes the eye rubbing state or the non-eye rubbing state. The data processing module 630 is also used to obtain the instantaneous state of each frame image within the sliding time window of the i-th frame image, and to count the proportion of images marked as rubbing eyes within the sliding time window. The sliding time window is a window with a preset duration, with the i-th frame image as the last frame. The determination module 640 is used to determine that the target person is rubbing their eyes within the sliding time window if the proportion of images marked as rubbing eyes within the sliding time window is greater than a preset proportion threshold.

[0128] It should be noted that the information interaction and execution process between the above-mentioned devices / modules are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0129] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0130] To implement the above embodiments, this application also proposes an electronic device.

[0131] Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application.

[0132] like Figure 7 As shown, the above-mentioned electronic device 700 includes: The system includes a memory 710 and at least one processor 720, and a bus 730 connecting different components (including the memory 710 and the processor 720). The memory 710 stores a computer program, and when the processor 720 executes the program, it implements the method for detecting eye rubbing state according to the embodiments of this application.

[0133] Bus 730 represents one or more of several bus architectures, including memory buses or memory controllers, peripheral buses, graphics acceleration ports, processors, or local buses using any of the various bus architectures. For example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MCA) buses, Enhanced ISA buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses.

[0134] Electronic device 700 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by electronic device 700, including volatile and non-volatile media, removable and non-removable media.

[0135] The memory 710 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 740 and / or cache memory 750. The electronic device 700 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 760 may be used to read and write non-removable, non-volatile magnetic media (… Figure 7 Not shown; usually referred to as a "hard drive"). Although Figure 7 As not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 730 via one or more data media interfaces. Memory 710 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0136] A program / utility 780 having a set (at least one) of program modules 770 may be stored in, for example, memory 710. Such program modules 770 include—but are not limited to—an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 770 typically perform the functions and / or methods described in the embodiments of this application.

[0137] Electronic device 700 can also communicate with one or more external devices 790 (e.g., keyboard, pointing device, display 791, etc.), and with one or more devices that enable a user to interact with electronic device 700, and / or with any device that enables electronic device 700 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 795. Furthermore, electronic device 700 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 793. As shown, network adapter 793 communicates with other modules of electronic device 700 via bus 730. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0138] The processor 720 executes various functional applications and data processing by running programs stored in the memory 710.

[0139] It should be noted that the implementation process and technical principles of the electronic device in this embodiment are explained in the foregoing description of the method for detecting eye rubbing in this application embodiment, and will not be repeated here.

[0140] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0141] This application provides a computer program product that, when run on an electronic device, causes the electronic device to perform the steps described in the various method embodiments above.

[0142] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed by a processor, it implements the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographic device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some regions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0143] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0144] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0145] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0146] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0147] In the foregoing, specific details such as particular system architectures and techniques have been set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted to avoid unnecessary detail from obscuring the description of this application.

[0148] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0149] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0150] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0151] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0152] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0153] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for detecting eye rubbing, characterized in that, The method includes: Acquire a series of consecutive frames of images, wherein at least one frame of the multiple frames includes the face of the target person; When performing eye rubbing detection on the i-th frame of the multi-frame images, the coordinates of the facial key points and hand key points of the target person are obtained. The facial key points include the human eye key points, and i is a positive integer. The radius of eye rubbing of the target person in the i-th frame image is determined based on the relative positional relationship between the facial key points; Based on the relationship between the distance between the coordinates of the key points of the human eye and the coordinates of the key points of the hand and the radius of the eye rubbing, the instantaneous state of the i-th frame image is determined and marked, and the instantaneous state includes the eye rubbing state or the non-eye rubbing state; The instantaneous state of each frame image within the sliding time window of the i-th frame image is obtained, and the proportion of images marked as eye-rubbing state within the sliding time window is counted. The sliding time window is a window with a preset duration, with the i-th frame image as the last frame. If the proportion of images marked as rubbing eyes within the sliding time window is greater than a preset proportion threshold, it is determined that the target person is rubbing their eyes within the sliding time window.

2. The method according to claim 1, characterized in that, The facial key points include the nasal key points, and the human eye key points include the left eye key points and the right eye key points; Determining the rubbing radius of the target person in the i-th frame image based on the relative positional relationship between the facial key points includes: Calculate the first distance between the nasal keypoint and the left eye keypoint; Calculate the second distance between the nasal keypoint and the right eye keypoint; Calculate the third distance between the left eye key point and the right eye key point; The maximum value among the first distance, the second distance, and the third distance is determined as the reference distance; Obtain a preset radius coefficient, and determine the product of the radius coefficient and the reference distance as the rubbing radius of the target person in the i-th frame image.

3. The method according to claim 1, characterized in that, The number of key points on the hand is multiple, and the key points on the human eye include key points on the left eye and key points on the right eye; The step of determining and marking the instantaneous state of the i-th frame image based on the relationship between the distance between the coordinates of the key points of the human eye and the coordinates of the key points of the hand and the radius of the eye rubbing includes: Calculate the distance between the coordinates of each key hand point and the coordinates of each key eye point; If there exists at least one hand key point and the distance between the left eye key point or the right eye key point is less than or equal to the rubbing radius, the instantaneous state of the i-th frame image is determined and marked as the rubbing state; If the distance between each key point on the hand and each key point on the eye is greater than the rubbing radius, the instantaneous state of the i-th frame image is determined and marked as a non-rubbing state.

4. The method according to claim 1, characterized in that, When performing eye-rubbing detection on the i-th frame of the multi-frame image, obtaining the coordinates of the facial and hand key points of the target person includes: Face detection is performed on the i-th frame image to obtain the face region bounding box of the target person; Based on the position of the face region bounding box in the i-th frame image, human key point detection is performed on the i-th frame image to obtain the coordinates of multiple human key points of the target person and their corresponding confidence scores. The multiple human key points include the facial key points and wrist key points. If the confidence level of the human eye key point in the facial key points reaches the preset eye confidence threshold, then the hand key points of the i-th frame image are detected based on the wrist key points to obtain the coordinates of the hand key points; The method further includes: If the confidence level of the human eye key point in the facial key points does not reach the preset eye confidence threshold, then the eye rubbing detection for the i-th frame image is cancelled.

5. The method according to claim 4, characterized in that, The method further includes: If the face region bounding box of the target person is not detected in the i-th frame image, the preceding image in which the face region bounding box of the target person was last detected is obtained, and it is determined whether the interval between the preceding image and the i-th frame image exceeds the preset anti-shake duration. If the duration between the preceding image and the i-th frame does not exceed the preset anti-shake duration, then based on the position of the face region bounding box in the preceding image, human key point detection is performed on the i-th frame. If the time interval between the preceding image and the i-th frame exceeds the preset anti-shake duration, then the eye rubbing detection for the i-th frame is cancelled.

6. The method according to claim 4 or 5, characterized in that, The method further includes: When performing human body key point detection on the i-th frame image, if no human body key points of the target person are detected, the eye rubbing detection on the i-th frame image is cancelled.

7. The method according to claim 1, characterized in that, The step of obtaining the instantaneous state of each frame within the sliding time window of the i-th frame image and calculating the proportion of images marked as eye-rubbing states within the sliding time window includes: Obtain the actual cumulative duration of each frame within the sliding time window of the i-th frame image, and determine whether the actual cumulative duration reaches the preset duration; If the actual cumulative duration reaches the preset duration, then the percentage of images marked as rubbing eyes within the sliding time window is counted. The method further includes: If the actual cumulative duration does not reach the preset duration, the step of determining whether the target person is rubbing their eyes within the sliding time window will be cancelled.

8. A device for detecting eye rubbing, characterized in that, The device includes: The data acquisition module is used to acquire multiple consecutive frames of images, wherein at least one frame of the multiple frames includes the face of the target person; The detection module is used to obtain the coordinates of the facial key points and hand key points of the target person when performing eye rubbing detection on the i-th frame of the multi-frame images. The facial key points include the eye key points, and i is a positive integer. The data processing module is also used to determine the rubbing radius of the target person in the i-th frame image based on the relative positional relationship between the facial key points; The data processing module is further configured to determine and mark the instantaneous state of the i-th frame image based on the relationship between the distance between the coordinates of the key points of the human eye and the coordinates of the key points of the hand and the radius of the eye rubbing, wherein the instantaneous state includes an eye rubbing state or a non-eye rubbing state. The data processing module is also used to obtain the instantaneous state of each frame image within the sliding time window of the i-th frame image, and to count the proportion of images marked as eye-rubbing state within the sliding time window, wherein the sliding time window is a window with a preset duration of the i-th frame image as the last frame. The determination module is used to determine that the target person is rubbing their eyes within the sliding time window if the proportion of images marked as rubbing eyes within the sliding time window is greater than a preset proportion threshold.

9. An electronic device comprising a memory, one or more processors, and a computer program stored in the memory and executable on the one or more processors, characterized in that, When the one or more processors execute the computer program, the electronic device performs the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images