Method and system for identifying a proxy card punching behavior based on image feature comparison

Abstract

The present application relates to the technical field of image recognition, and particularly relates to a method and system for recognizing a proxy clock-in behavior based on image feature comparison, comprising: collecting continuous frames of original face images; extracting face key points from the continuous frames of original face images and obtaining key point information; determining a closed mouth reference frame and a mouth opening keeping frame; calculating a mouth opening change vector according to the key point information in the closed mouth reference frame and the mouth opening keeping frame; calculating a support chain conduction amount; determining a nodding peak frame of a nodding action and calculating a nodding rigid driving consistency amount; calculating a structure coupling keeping degree by using the support chain conduction amount and the nodding rigid driving consistency amount; and comparing the structure coupling keeping degree with a preset judgment threshold value, and judging whether the current clock-in behavior is a real clock-in according to a comparison result. The present application can improve the accuracy and security of face clock-in recognition.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and specifically to a method and system for identifying proxy attendance behavior based on image feature comparison. Background Technology

[0002] With the continuous development of facial recognition technology, facial recognition attendance has been widely replaced by traditional fingerprint and card-swiping attendance methods due to its advantages such as being contactless, efficient, and low-cost, and is applied in scenarios such as corporate attendance, park management, and remote office verification. However, with the rapid development of deepfake face-swapping technology and generative facial video technology, the accuracy and security of facial recognition attendance are facing serious challenges.

[0003] Existing facial recognition attendance systems typically use cameras to capture facial images of the person clocking in and compare these images with pre-stored employee facial templates to determine if the person clocking in is the target employee. Some systems also incorporate motion detection, requiring users to perform specified actions such as blinking, opening their mouth, and nodding to verify their identity, thereby reducing the risk of proxy clocking in using photos or video playback.

[0004] However, with advancements in deepfake and generative facial video technologies, forged videos can now generate relatively natural facial appearances and localized movement variations. Current technologies primarily focus on the similarity of facial appearances or simply determine whether a single action has been completed. When faced with complex deepfake scenarios, they lack deeper and more nuanced differentiation logic. Therefore, existing facial recognition methods still suffer from insufficient accuracy and security in practical applications. Summary of the Invention

[0005] This invention provides a method and system for identifying proxy attendance behavior based on image feature comparison, in order to solve existing problems.

[0006] The image feature comparison-based method and system for identifying proxy attendance behavior of the present invention adopts the following technical solution: One embodiment of the present invention provides a method for identifying proxy attendance behavior based on image feature comparison, the method comprising the following steps: In response to the check-in request from the target object, continuously capture the original face images of the target object in the current check-in process; Extract facial key points from consecutive frames of raw facial images and obtain key point information; Based on key point information, determine the closed reference frame before the mouth opening action begins and the mouth opening hold frame after the nodding action begins. Calculate the mouth opening change vector based on key point information in the closed reference frame and the open mouth holding frame; Using key point information, closed reference frames, open-mouth holding frames, and mouth opening change vectors, the support chain conduction is calculated. The peak frame of the nodding action is determined based on key point information, and the stiffness driving consistency of the nodding action is calculated using key point information, mouth opening holding frame and peak frame. The degree of structural coupling retention is calculated by utilizing the transmission capacity of the support chain and the consistency of the head stiffness drive. The degree of structural coupling retention is compared with a preset judgment threshold, and the current check-in behavior is judged as a genuine check-in based on the comparison result.

[0007] Furthermore, the specific steps for extracting facial key points from consecutive frames of original face images and obtaining key point information are as follows: Each frame of the original face image is subjected to image enhancement and size normalization processing to obtain the first intermediate image; Perform face detection and alignment on the first intermediate image to obtain consecutive frame face images; Each frame of face image is input into a pre-trained 106-point facial landmark localization model, which outputs the coordinates of 106 landmarks in each frame. By associating the coordinates of each keypoint, its corresponding number, and the frame time in which it is located, we can obtain the information of each keypoint.

[0008] Furthermore, the specific steps for determining the mouth-closing reference frame before the mouth-opening action and the mouth-opening holding frame after the head-nodding action begins, based on key point information, are as follows: The center points of the upper and lower lips are obtained from the lip key points in consecutive frames of face images; Based on the distance change curve between the center points of the upper and lower lips, a closed reference frame and an open-mouth holding frame are determined. The closed reference frame is the last frame before the mouth opening action begins, when the distance between the center points of the upper and lower lips is less than a preset first distance threshold. The open-mouth holding frame is the first frame after the nodding action begins, when the distance between the center points of the upper and lower lips is greater than a preset second distance threshold.

[0009] Furthermore, the specific steps for calculating the mouth opening change vector based on key point information in the closed-mouth reference frame and the open-mouth holding frame are as follows: Based on the information of each key point, the key points of the nose are obtained locally, and the key points of the nose are used as stable reference points; among them, the key points of the nose locally include at least the key points of the bridge of the nose, the tip of the nose, and the left and right nostrils. Using stable reference points, a two-dimensional similarity transformation is performed on key points in the closed-mouth reference frame and the open-mouth retention frame to align them, and the coordinates of the key points after the two frames are aligned are obtained. Key points of the lips are selected from the key point coordinates of the two frames after alignment, and the average coordinates of the key points of the lips are calculated to obtain the center point of the upper lip and the center point of the lower lip in each frame. The relative position vector from the center point of the upper lip to the center point of the lower lip is obtained in each frame. The relative position vector corresponding to the open mouth frame is subtracted from the relative position vector corresponding to the closed mouth reference frame to obtain the mouth opening change vector.

[0010] Furthermore, the specific steps for calculating the support chain conductance using key point information, closed reference frames, open-mouth holding frames, and mouth opening change vectors are as follows: The key points of the chin and jawline are used as primary support key points, and the key points of the lower edge of the left and right cheeks are used as secondary traction key points. For each primary support keypoint, subtract the coordinates of the keypoint in the aligned open mouth holding frame from the coordinates of the keypoint in the aligned closed mouth reference frame to obtain the two-dimensional displacement vector of the keypoint. Calculate the square of the magnitude of the mouth opening change vector, and divide the dot product of the two-dimensional displacement vector of each primary support key point and the mouth opening change vector by the square to obtain the first ratio of each primary support key point. The summation of the first ratios of all primary support key points is divided by the number of primary support key points to obtain the aggregate support response of the primary support key points. Obtain the aggregated support response volume of the secondary key points; The aggregated support response quantities of the primary support key points and the secondary traction key points are respectively positiveized and fault-tolerant to obtain the processed primary support response quantities and secondary traction response quantities. Multiply the processed primary support response and secondary traction response to obtain the support chain transmission quantity.

[0011] Furthermore, the specific steps for determining the peak frame of the nodding action based on key point information are as follows: The time interval between the open-mouth holding frame and the last frame of the entire acquisition process is defined as the duration of the nodding action; For each frame within the continuous time period, the head pitch angle of that frame is obtained using a head pose estimation algorithm, and the absolute difference between the head pitch angle of that frame and the head pitch angle of the open-mouth holding frame is calculated. The frame with the largest absolute difference is taken as the peak frame of the head nod.

[0012] Furthermore, the specific steps for calculating the head-nodding rigidity drive consistency using key point information, mouth-opening holding frames, and head-nodding peak frames are as follows: Based on the coordinates of the stable reference point in the mouth-opening frame and the head-nodding peak frame, a rigid motion mapping relationship is fitted. For each stable reference point, calculate the displacement between the coordinates of that point after rigid motion mapping and its coordinates in the mouth-open holding frame. Sum the displacements of all stable reference points and divide by the number of stable reference points to obtain the overall motion intensity. The union of primary support key points, secondary traction key points, and mouth center-related key points is taken as the set of lower face follow-up key points; among them, mouth center-related key points include at least the upper lip center point, the lower lip center point, and the left and right corners of the mouth key points; For each key point in the set of key points for the lower half of the face, subtract its coordinates in the open-mouth holding frame from the coordinates of the key point after rigid motion mapping to obtain the theoretical follow-up displacement vector of the key point; subtract its coordinates in the nodding peak frame from the coordinates of the key point in the open-mouth holding frame to obtain the actual follow-up displacement vector of the key point. Calculate the sum of the square of the magnitude of the theoretical follower displacement vector and the preset zero-prevention constant, and divide the dot product of the theoretical follower displacement vector and the actual follower displacement vector by this sum to calculate the follower response ratio of the key point. For each key point in the set of follow-up key points in the lower half of the face, the follow-up response ratio of the key point is multiplied by the theoretical follow-up displacement vector to obtain the first product vector, and the actual follow-up displacement vector is subtracted from the first product vector to obtain the residual vector. The first sum is obtained by summing the magnitudes of the residual vectors of all key points, and the second sum is obtained by summing the magnitudes of the actual displacement vectors of all key points. Dividing the first sum by the second sum yields the unexplained residual proportions of the lower half of the face. Divide the overall motion intensity by the unexplained residual proportion of the lower half of the face to calculate the head nodding rigid drive consistency.

[0013] Furthermore, the specific steps for calculating the structural coupling retention degree using the support chain conductance and the nodding rigidity driving consistency are as follows: Multiply the normalized support chain conduction, the normalized nodding rigid drive consistency, and 2 together to obtain the first product; The normalized support chain conductance, the normalized head stiffness drive consistency, and the preset zero-prevention constant are added together to obtain the third sum. Dividing the first product by the third sum yields the degree of structural coupling retention.

[0014] Furthermore, the specific steps for comparing the degree of structural coupling retention with a preset judgment threshold and determining whether the current check-in behavior is a genuine check-in based on the comparison result are as follows: If the degree of structural coupling retention is greater than or equal to the preset judgment threshold, then the current check-in behavior is determined to be a genuine check-in. If the degree of structural coupling retention is less than the preset judgment threshold, then it is determined that the current check-in behavior has the risk of proxy check-in.

[0015] This invention proposes a system for identifying proxy attendance based on image feature comparison, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the method for identifying proxy attendance based on image feature comparison.

[0016] The beneficial effects of the technical solution of this invention are as follows: This invention proposes a method and system for identifying proxy attendance behavior based on image feature comparison. This method performs temporal comparison of facial key points in consecutive frames of facial images of the target object. First, it analyzes the transmission relationship between the mouth opening change and the support chain between the jaw and cheeks during the mouth opening phase to determine whether the mouth opening action conforms to the biomechanical linkage law of a real person. Then, it further analyzes the rigid drive consistency relationship when performing a nodding action while maintaining a mouth open, verifying whether the lower half of the face undergoes overall rigid movement along with the stable nasal structure. Through the above two-stage collaborative analysis, this invention can effectively identify deepfake-based face-swapping attacks and various proxy attendance behaviors, significantly improving the accuracy and security of facial attendance recognition. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of the steps of the method for identifying proxy attendance based on image feature comparison in this invention; Figure 2 This is a block diagram of the proxy attendance recognition system based on image feature comparison according to the present invention. Detailed Implementation

[0019] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of the image feature comparison-based card-checking behavior recognition method and system proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0021] The following description, in conjunction with the accompanying drawings, details the specific scheme of the image feature comparison-based method and system for identifying proxy attendance provided by this invention.

[0022] Please see Figure 1 The diagram illustrates a flowchart of a method for identifying proxy attendance based on image feature comparison according to an embodiment of the present invention. The method includes the following steps: Step S001: In response to the check-in request of the target object, acquire continuous frame original face images of the target object during the current check-in process.

[0023] It should be noted that: the target object refers to the person who initiates the check-in request, i.e., the person whose identity needs to be verified. In this embodiment, the target object can be a company employee, visitor, person within the park, or remote worker, or other user whose identity needs to be verified.

[0024] Data Acquisition Method: When the attendance terminal (such as a fixed facial recognition check-in machine, smartphone, tablet, or computer equipped with a camera) receives a check-in request triggered by the target object via button, touchscreen, voice, or automatic sensing, it calls its front-facing or external camera to capture a real-time facial video stream of the target object at a preset frame rate (such as 15 or 30 frames per second), extracting continuous frames of raw facial images from it. During the acquisition process, the system will detect whether the face is within the camera's visible range and without significant obstruction. If the conditions are met, it will start recording continuous frame images; otherwise, it may issue a prompt requesting adjustment of posture or position.

[0025] Specifically, during the current attendance recognition process, in response to the attendance request of the target individual, the attendance terminal calls the camera to capture consecutive frames of raw facial images during the current attendance process. To meet confidentiality requirements and ensure the accuracy of subsequent differentiation, when capturing consecutive frames of raw facial images, it should be ensured that the target individual's face is within the camera's visible range and that the face is completely unobstructed. If the current attendance is a scenario such as remote attendance, re-attendance, first-time device attendance, or system spot check verification, the system outputs a composite motion perturbation command. This command is used to prompt the target individual to first perform an open-mouth action and hold it briefly, and then complete a nodding action while maintaining the open-mouth state.

[0026] Step S002: Extract facial key points from consecutive frames of original face images and obtain key point information.

[0027] Step S002 further includes: Step S0021: Perform image enhancement and size normalization processing on each frame of the original face image to obtain the first intermediate image.

[0028] It should be noted that: Image enhancement: Histogram equalization, adaptive gamma correction, or contrast-limited adaptive histogram equalization (CLAHE) are used to improve the quality of the original face image caused by uneven lighting, underexposure, or overexposure, and to enhance the texture contrast of key facial areas.

[0029] Size standardization: The enhanced original face image is scaled to a uniform size (e.g., 224×224 pixels or 256×256 pixels) to match the input image size requirements of the facial landmark localization model. Simultaneously, the original aspect ratio is maintained, and padding or cropping methods are used to avoid stretching and distortion.

[0030] Output the first intermediate image: After the above enhancement and normalization processing, each frame of the original face image is converted into an intermediate image with uniform size and optimized contrast, which is denoted as the first intermediate image.

[0031] Objective: To eliminate image differences caused by different acquisition devices, different ambient lighting, and different face distances, and to improve the stability and generalization ability of the facial landmark localization model.

[0032] Step S0022: Perform face detection and alignment on the first intermediate image to obtain consecutive frame face images.

[0033] It should be noted that: Face detection: Deep learning-based face detection algorithms (such as MTCNN, RetinaFace, or OpenCV's DNN module loaded with pre-trained SSD or YOLO-face models) are used to detect faces in the first intermediate image of each frame, outputting face bounding boxes. During the detection process, a confidence threshold (e.g., 0.9) is set to filter out non-face regions, ensuring that only the face region with the largest area and highest confidence is retained in each frame to exclude background interference or multiple faces.

[0034] Face alignment: Based on detected facial landmarks (usually five reference points: the center of the eyes, the tip of the nose, and the corners of the mouth) or through affine transformation, the face is corrected to a standard pose. Specifically, using the correspondence between the detected reference points and preset standard template points, an affine transformation matrix is ​​calculated, and then an affine transformation is performed on the entire face image to ensure that the eyes are in a horizontal position and the face scale is normalized. The alignment operation can effectively eliminate the influence of small-scale head tilts and other pose changes on subsequent landmark localization.

[0035] Obtaining consecutive frame face images: After detection and alignment, the aligned face region is cropped from the first intermediate image of each frame and rescaled to a fixed size (such as 112×112 pixels or 224×224 pixels) to obtain a consecutive frame face image sequence. The face position, pose, and scale of each frame in this sequence are basically consistent.

[0036] By eliminating faceless or multi-person interference frames through face detection and correcting head pose changes through alignment, the faces in the entire continuous frame face image sequence have uniform geometric alignment and size, thereby improving the accuracy and stability of the subsequent 106 key point localizations.

[0037] Step S0023: Input each frame of face image into the pre-trained 106-point face landmark localization model and output the coordinates of 106 landmarks in each frame.

[0038] It should be noted that when a real person performs a complex action, their facial key points exhibit continuous and coordinated structural changes. In contrast, deepfake attendance tracking typically only creates visual changes in localized areas, making it difficult to simultaneously maintain the lower facial support chain transmission relationship during the mouth-opening phase and the overall rigidity follow-up relationship during the head-nodding phase. Therefore, this embodiment specifically includes facial key point extraction and temporal coordinate construction, calculation of the consistency between the support chain transmission amount and the head-nodding rigidity drive, and calculation of the degree of structural coupling maintenance between the mouth-opening support chain and the head-nodding rigidity drive.

[0039] Specifically, the aligned face images (i.e., the pre-processed consecutive frame face images) are input frame by frame into a pre-trained 106-point face landmark localization model. The input images are usually single-channel grayscale images or three-channel color images, and their size is normalized to the fixed resolution required by the model (such as 112×112 or 224×224 pixels).

[0040] The 106-point facial landmark localization model can employ a regression model based on a convolutional neural network (CNN), such as MobileNet, ResNet, or the lighter ShuffleNet, as the backbone network. The output layer is a fully connected layer with an output dimension of 106×2 (x and y coordinates of each landmark). The model is pre-trained on a large dataset of face images annotated with 106 facial landmarks, using mean squared error (MSE) or Wing Loss as the loss function to accurately learn the mapping from face images to dense landmarks.

[0041] Each keypoint in the model corresponds to a fixed facial location, including but not limited to: the outer contour of the face (hairline, jawline), eyebrows (brow peak, brow tail), eyes (corner of the eye, upper and lower edges of the pupil), nose (bridge of the nose, tip of the nose, alar of the nose), mouth (upper and lower lip contours, corners of the mouth), chin, etc. The 106-point facial keypoint localization model used in this embodiment pays particular attention to the dense description of areas such as the jaw, cheeks, and lips to support subsequent support chain transmission analysis.

[0042] For each input image frame, the model outputs a vector set containing 106 two-dimensional coordinates, denoted as... ,in , Indicates the first The first frame The two-dimensional coordinates of each key point are stored together with the key point number and frame time to form a key point information sequence for subsequent time-series comparison and feature calculation.

[0043] By using 106 key points, precise localization of areas such as the mouth, jaw, cheeks, and nose is achieved, providing sufficient geometric constraints and motion details for analyzing the support chain transmission during the mouth opening phase and the rigid drive during the head nodding phase.

[0044] Step S0024: Associate the coordinates of each key point, the corresponding number of the key point, and the frame time of the key point to obtain the information of each key point.

[0045] Step S003: Based on key point information, determine the closed reference frame before the mouth opening action begins and the mouth opening hold frame after the head nodding action begins.

[0046] It's important to note that when a real person opens their mouth, the opening not only involves an increase in the distance between the upper and lower lips, but also simultaneously causes the chin and jawline to move downwards, and pulls on the lower edges of the cheeks due to the movement of the corners of the mouth and the jaw. In other words, a real mouth-opening action should present a continuous chain of structural changes in a two-dimensional image, meaning that the change in mouth opening occurs in conjunction with the movement of the chin, jawline, and the lower edges of the cheeks. This chain of changes can be defined as the "mouth-jaw-cheek support chain."

[0047] In contrast, in deepfake attendance-taking scenarios, the generated model often only alters the mouth texture, lip shape, or internal oral cavity area to make it appear as if the mouth is open. However, the chin, jawline, and lower edges of the cheeks do not exhibit any movement that matches the mouth opening, or the movement is insufficient. This is reflected in the 2D keypoint trajectory as follows: although the distance between the upper and lower lips changes, the displacement of the jawline, chin, and lower cheek keypoints is insufficient, or their displacement direction is inconsistent with the direction of the mouth opening.

[0048] Step S003 further includes: Step S0031: Obtain the center points of the upper and lower lips based on the key points of the lips in the continuous frame face images.

[0049] Specifically, among the extracted facial keypoints in each frame, a set of keypoints belonging to the upper lip contour region and a set of keypoints belonging to the lower lip contour region are selected based on the keypoint number. The average values ​​of the x and y coordinates of these two sets of keypoints are calculated to obtain the corresponding average coordinate points, which are the center points of the upper and lower lip in that frame. In this way, multiple keypoints of the lips are merged into a representative center point, which can reflect the overall position of the lips and effectively suppress the influence of errors in the detection of individual keypoints.

[0050] Step S0032: Based on the distance change curve between the center points of the upper and lower lips, determine the closed reference frame and the open-mouth holding frame; wherein, the closed reference frame is the last frame before the mouth opening action begins, when the distance between the center points of the upper and lower lips is less than a preset first distance threshold; the open-mouth holding frame is the first frame after the nodding action begins, when the distance between the center points of the upper and lower lips is greater than a preset second distance threshold.

[0051] Specifically, firstly, the Euclidean distance between the center points of the upper and lower lips is calculated frame by frame, resulting in a distance curve that varies over time, i.e., a distance change curve. This curve reflects the process of the mouth going from closed to open and then to a closed position.

[0052] To determine the reference frame for closing the mouth, a search begins after the system issues the mouth-opening command. The last frame before the mouth-opening action begins, where the distance change curve remains in a low, stable range (i.e., the distance between the center points of the upper and lower lips is less than a preset first distance threshold) for a certain number of frames, is selected as the reference frame for closing the mouth before the mouth-opening action begins. The first distance threshold is set based on the statistical value of the lip distance in a normal closed-mouth state; for example, it can be the upper limit of the statistical value of the lip distance in a closed-mouth state. In practical applications, a proportional threshold method can be used: taking the lip distance of the target object in a resting state (closed mouth) as a benchmark, the first distance threshold is usually set to 1.2 times the resting lip distance (i.e., allowing for slight fluctuations), or a normalized mouth aspect ratio (MAR) threshold can be used, in which case the MAR value corresponding to the first distance threshold is preferably 0.1 to 0.2. Taking a common camera resolution (e.g., 1920×1080) and the proportion of the face in the image (face width approximately 200-300 pixels) as an example, the first distance threshold can be specifically set to 5-10 pixels, but this value is only for illustration and should not be taken as a limitation.

[0053] To determine the mouth-opening hold frame, the system first detects whether the mouth-opening action is complete (i.e., the distance between the center points of the upper and lower lips is greater than a preset second distance threshold and remains stable). Then, after the nodding action begins (which can be determined by changes in head pitch angle or timing logic), the system selects the first frame where the distance change curve first exceeds the second distance threshold and remains stable as the mouth-opening hold frame after the nodding action begins. The second distance threshold is set based on the statistical value of the lip distance during the mouth-opening hold state, typically the lower limit of the normal mouth-opening amplitude, such as the median or slightly below the mean of the statistical values. Preferably, the second distance threshold can be 1.5 to 2.5 times the resting lip distance, or a MAR threshold can be used, with a preferred MAR value of 0.4 to 0.6. Taking common camera resolutions and face proportions as examples, the second distance threshold can be specifically set to 20 to 40 pixels, but this is only an example; the actual threshold needs to be determined through sample statistics based on specific devices and usage scenarios.

[0054] Step S004: Calculate the mouth opening change vector based on the key point information in the closed reference frame and the open mouth holding frame.

[0055] Step S004 further includes: Step S0041: Based on the information of each key point, obtain the key points of the nose in the local area, and use the key points of the nose in the local area as stable reference points; wherein, the key points of the nose in the local area include at least the key points of the bridge of the nose, the tip of the nose, and the left and right nostrils.

[0056] Specifically, based on the key point number, key points on the bridge of the nose, the tip of the nose, and the left and right nostrils were selected from 106 points as stable reference points. These points are located in the rigid areas of the face and undergo almost no local deformation during mouth opening and nodding movements, thus representing the overall posture and movement of the head.

[0057] Step S0042: Use stable reference points to perform a two-dimensional similarity transformation to align the key points in the closed reference frame and the open mouth holding frame, and obtain the coordinates of the key points after the two frames are aligned.

[0058] It should be noted that two-dimensional similarity transformation alignment refers to mapping the coordinates of one image (or a set of key points) to the coordinate system of another image (or another set of key points) through rotation, scaling, and translation operations, so that corresponding points in the two images coincide as much as possible. In this embodiment, the transformation parameters are specifically solved using the coordinates of stable reference points (i.e., key points at the bridge of the nose, tip of the nose, and left and right nostrils) in the closed-mouth reference frame and the open-mouth holding frame. Since these points undergo almost no local deformation during the mouth-opening action, their positional changes mainly come from the overall head movement. Therefore, the transformation fitted by them can represent the overall pose difference between the two frames.

[0059] The alignment process is as follows: Using the closed-mouth reference frame as a baseline, a two-dimensional similarity transformation consisting of rotation, scaling, and translation is calculated to minimize the overall difference between the coordinates of the stable reference points after the transformation and their coordinates in the open-mouth hold frame. The transformation parameters can be solved using optimization methods such as least squares. After obtaining the transformation, the coordinates of all key points (including the lips, jaw, cheeks, etc.) in the closed-mouth reference frame are applied one by one. That is, the position of each point is adjusted using rotation, scaling, and translation operations, thereby aligning the key points in the closed-mouth reference frame to the same head pose as the open-mouth hold frame. The resulting key point coordinates in the closed-mouth reference frame are the aligned coordinates, while the key point coordinates in the open-mouth hold frame remain unchanged. Alternatively, reverse alignment (using the open-mouth hold frame as a baseline) can be performed, as long as the stable nasal region overlaps in both frames.

[0060] Step S0043: Select the key points of the lips from the key point coordinates after the two frames are aligned, and calculate the average coordinates of the key points of the lips to obtain the center point of the upper lip and the center point of the lower lip in each frame.

[0061] Specifically, in the two aligned frames, the key points corresponding to the upper lip contour and the lower lip contour are selected respectively, and the average value of the coordinates of each set of key points is calculated to obtain the center point of the upper lip and the center point of the lower lip in each frame.

[0062] Step S0044: Obtain the relative position vector from the center point of the upper lip to the center point of the lower lip in each frame, and subtract the relative position vector corresponding to the closed reference frame from the relative position vector corresponding to the open mouth frame to obtain the mouth opening change vector.

[0063] Specifically, for each frame (including the closed reference frame and the open-mouth holding frame), based on the already determined center points of the upper and lower lip, the coordinates of the upper lip center point are subtracted from the coordinates of the lower lip center point to obtain a two-dimensional vector. This vector represents the relative position vector from the upper lip center point to the lower lip center point. The direction of this vector is roughly from the center of the lips downwards, and its length reflects the opening and closing distance between the upper and lower lips in the current frame.

[0064] For example, let the center point of the upper lip of the closed reference frame be... The center point of the lower lip is Then the relative position vector of this frame is Similarly, let the center point of the upper lip in the open-mouth hold frame be... The center point of the lower lip is The relative position vector of the open mouth frame is Then, subtract the relative position vector of the closed reference frame from the relative position vector of the open-mouthed frame, i.e. This yields a mouth opening change vector. This vector quantifies the displacement of the lower lip relative to the upper lip during the transition from a closed state to a mouth-open holding state: its direction represents the primary direction of mouth opening (usually downward, but may also include a slight horizontal offset), and its length represents the distance the lower lip moves relative to the upper lip (i.e., the increment in mouth opening amplitude). Since the closed reference frame and the mouth-open holding frame have been aligned using stable reference points, eliminating overall head movement, this change vector purely reflects the relative lip movement caused by the mouth opening action itself, providing a benchmark for subsequent analysis of whether the jaw and cheeks respond to this drive.

[0065] Step S005: Calculate the support chain conduction using key point information, closed reference frame, open mouth holding frame, and mouth opening change vector.

[0066] Step S005 further includes: Step S0051: Use the chin key point and jaw contour key point as primary support key points, and the lower edge key points of the left and right cheeks as secondary traction key points.

[0067] It should be noted that, based on the actual traction relationship of opening the mouth, the opening of the mouth will first cause changes in the support of the chin and jaw contour. Therefore, chin key points and jaw contour key points were selected from 106 key points to form a primary support key point set. Similarly, when a real person opens their mouth, the changes in the jawline further affect the lower edges of the left and right cheeks. Therefore, key points along the lower edges of the left and right cheeks were selected from 106 key points to form a secondary set of key points affecting the jawline. .

[0068] Step S0052: For each primary support key point, subtract the coordinates of the key point in the aligned mouth-opening reference frame from the coordinates of the key point in the aligned mouth-opening frame to obtain the two-dimensional displacement vector of the key point.

[0069] Specifically, for sets Each key point in Obtain the two-dimensional displacement vector of this key point from the closed reference frame to the open-mouth holding frame. .

[0070] Step S0053: Calculate the square of the magnitude of the mouth opening change vector, and divide the dot product of the two-dimensional displacement vector of each primary support key point and the mouth opening change vector by the square to obtain the first ratio of each primary support key point.

[0071] Specifically, Let be the first ratio, where Represents the vector of mouth opening change. Length of the module The square of; Indicates the primary support key point Two-dimensional displacement vector The dot product of the vector representing the change in the mouth opening.

[0072] Step S0054: Sum the first ratios of all primary support key points and divide by the number of primary support key points to obtain the aggregate support response of the primary support key points.

[0073] Specifically, calculate the pooled support response of the primary support key points: ; in, This represents the aggregated support response of the primary support key points.

[0074] Represents the set of primary support key points The total number of all key points, i.e. the number of primary support key points.

[0075] The numerator in the formula This represents the dot product of two two-dimensional vectors. Mathematically, the dot product reflects the consistency of the directions of two vectors and also measures the projected length of one vector in the direction of the other. Specifically, in this embodiment, the dot product quantifies how much of the displacement of the primary support key point occurs along the opening direction of the mouth. From a scenario perspective, this value is used to determine whether the movement of the current primary support key point effectively responds to the upstream action of "mouth opening".

[0076] denominator In mathematics, this represents the square length of the vector representing the change in mouth opening, i.e., the intensity of the change in mouth opening. From a contextual perspective, this denominator is used to eliminate the influence of different mouth opening amplitudes on the judgment of support response, allowing the analysis to focus on "whether each unit of mouth opening triggers corresponding jaw and cheek traction".

[0077] From a mathematical perspective, Represents a set The average normalized response of all key points to the opening-driven mechanism. From a scenario perspective, This is used to measure whether the chin and jawline contours form a corresponding primary support change after the mouth is opened.

[0078] Step S0055: Obtain the aggregated support response of the secondary key points.

[0079] Specifically, following the same method as for the primary support key points, the normalized response value of each key point in the set of secondary traction key points is calculated and averaged to obtain the aggregated support response of the secondary traction key points, which is used to measure the degree of traction of the lower edge of the cheek on the mouth opening action.

[0080] Calculate the pooled support response of the secondary support key points: ; in, This represents the aggregated support response of the secondary support key points.

[0081] Represents the set of secondary support key points The total number of all key points, i.e. the number of secondary support key points.

[0082] This indicates whether the lower edges of the left and right cheeks are affected by changes in the jaw and corners of the mouth when the mouth is open.

[0083] Step S0056: Perform positive transformation and fault tolerance processing on the aggregated support response quantities of the primary support key points and the secondary traction key points respectively to obtain the processed primary support response quantities and secondary traction response quantities.

[0084] It should be noted that: due to ( and ) is a set ( and The signed average response of all keypoints in the graph can be positive or negative. When, it indicates that the overall direction of movement in that area is consistent with the direction of the mouth opening drive; when This indicates that the entire area has not formed an effective support response, or that its direction of movement is not coordinated with the direction of the mouth opening drive. Therefore, for Perform positive feedback and fault tolerance processing.

[0085] The specific methods for positive feedback and fault tolerance are as follows: the aggregated support response of the primary support key points is... The convergence support response of the secondary key points is Add a preset fault tolerance coefficient to each. and Then take the maximum value between the result and zero, that is... , Among them, the fault tolerance coefficient and By collecting samples of real people performing mouth-opening actions, the number of samples in each sample was calculated using the same method. and And adopt the three-standard-deviation principle ( The value is determined by statistical methods such as the median. The purpose of this process is that only responses that are consistent with the direction of the mouth opening and exceed the normal fluctuation range are considered as effective support or effective traction; if the response is significantly reversed and exceeds the fault tolerance range, the positive result is set to zero, indicating that no effective support or traction has been formed in this area, and it no longer contributes to subsequent product calculations.

[0086] Step S0057: Multiply the processed primary support response and secondary traction response to obtain the support chain conduction quantity.

[0087] Specifically, a true mouth-opening action requires the simultaneous presence of primary support and secondary traction, forming a complete transmission chain of "mouth opening → chin / jaw support → lower cheek traction." Therefore, the transmission amount of the mouth-jaw-cheek support chain... The processed first-level support response quantity With secondary traction response quantity The product of these is obtained. Its significance in this scenario lies in determining whether changes in mouth opening are simultaneously and effectively transmitted to the chin / jawline contour and the lower edges of the left and right cheeks.

[0088] Step S006: Determine the peak frame of the nodding action based on the key point information, and calculate the consistency amount of the nodding rigid drive using the key point information, the mouth opening holding frame and the peak frame of the nodding action.

[0089] It should be noted that the support chain conductivity calculated above... It is mainly used to determine whether the mouth-jaw-cheek support chain, as expected of a real person, has been formed during the mouth-opening phase. However, relying solely on... This is still insufficient to fully identify deepfake attendance-taking behavior. The reason is that deepfake detection research shows that anomalies in fake videos exist not only in the local appearance of a single frame, but also in the coordinated motion relationships between different facial regions and the temporal relationships between consecutive frames. Therefore, even if the current image shows a seemingly reasonable mouth-jaw-cheek support state during the mouth-opening phase, it cannot be directly concluded that the subsequent nodding action is driven by genuine head movement. Therefore, further analysis is needed to determine whether a holistic motion-driven relationship, as expected of a real person, is formed between the stable facial structures and the lower half of the face during the nodding process.

[0090] When a real person performs the complex action of "nodding while keeping their mouth open," the nodding is not a simple local movement of the lower half of the face, nor is it a translation of the entire face as a 2D texture. Instead, it involves a holistic pitch change of the head around the neck and the center of the face. A realistic nodding action will exhibit structurally constrained overall motion characteristics in the 2D keypoint trajectory: key points such as the center of the nose, the center of the mouth, the chin, and the jawline change along the same pitch trend, while maintaining a relatively stable structural relationship between different facial layers.

[0091] In other words, a key characteristic of a realistic nod is that the nod involves the stable structures of the face and the lower face moving in unison, rather than the mouth, chin, or cheek moving in isolation. This is fundamentally different from opening the mouth—which involves a localized, non-rigid deformation of the lower face, while nodding represents a rigid movement of the entire head. When a real person nods while keeping their mouth open, even though the mouth, jaw, and cheeks are already deformed, these deformed lower facial areas should still move in the same direction as the nasal stabilizing structures after the nod.

[0092] Deepfake attendance tracking often exposes problems at this stage. The generative model might initially generate a seemingly reasonable open mouth position, but the following inconsistencies appear during the nodding phase: First, the nodding action is mainly caused by changes in local key points of the mouth, chin, or jaw, while the changes in the stabilizing structure of the nose are insufficient, indicating that only a local area may be regenerated in the image, rather than the entire head nodding. Second, the entire face or lower half of the face moves almost synchronously in the 2D image, lacking the hierarchical differences that should exist between key points in a real nod, making the image, despite having the visual effect of head movement, look more like a face texture that has been completely translated. Third, the movement between the upstream stabilizing structure and the downstream open mouth structure is asynchronous. These anomalies can all serve as important evidence for identifying forgery.

[0093] Among them, the peak frame of the nodding action determined based on key point information specifically includes: Step S0061: Determine the duration of the nodding action as the time interval between the open-mouth holding frame and the last frame of the entire acquisition process.

[0094] It should be noted that the time interval between the mouth-open holding frame (the head-nodding start point) and the last frame of the entire acquisition process is taken as the duration of the head-nodding action. This time interval completely covers the occurrence and end of the head-nodding action and is used to locate the peak head-nodding frame in subsequent acquisition.

[0095] Step S0062: For each frame within the continuous time period, the head pitch angle of the frame is obtained by using the head pose estimation algorithm, and the absolute difference between the head pitch angle of the frame and the open-mouth holding frame is calculated.

[0096] Specifically, during the defined duration of the nodding motion (from the mouth-opening frame) End of acquisition frame Within each frame, a head pose estimation algorithm is used to calculate the head pitch angle of the current frame. Commonly used head pose estimation algorithms include methods based on solving the PnP problem using keypoints: using the coordinates of facial keypoints (such as the tip of the nose, the left and right corners of the eyes, and the left and right corners of the mouth) in a 2D image and their corresponding points on a general 3D face model, the rotation angles (pitch, yaw, and roll) of the head in 3D space are calculated by iteratively minimizing the reprojection error. Alternatively, an end-to-end regression model based on deep learning can be used to directly output the head pose angles. The pitch angle is calculated for each frame. Then, it is compared with the reference frame (i.e., the open-mouth holding frame). pitch angle Subtract the values ​​and take the absolute value to obtain the change in pitch angle for that frame relative to the start of the head nod. This change reflects the cumulative magnitude of the head rotation from the start of the nod to the current frame's head turning down (or up).

[0097] Step S0063: Take the frame with the largest absolute difference as the head peak frame.

[0098] Specifically, during the nodding duration, the frame with the largest absolute difference in head pitch angle is taken as the nodding peak frame. This frame corresponds to the moment when the head movement amplitude is the largest during the nodding process and is used for subsequent rigid drive consistency analysis.

[0099] Specifically, by utilizing key point information, mouth-opening holding frames, and head-nodding peak frames, the head-nodding rigidity drive consistency is calculated, including: Step S061: Based on the coordinates of the stable reference point in the mouth-opening frame and the head-nodding peak frame, fit the rigid motion mapping relationship.

[0100] Specifically, the rigid motion mapping relationship is used to describe the overall posture changes of the head as a rigid body from the mouth-open holding frame (the start of nodding) to the nodding peak frame (the maximum nodding amplitude). Since the stable reference points (bridge of the nose, tip of the nose, left and right nostrils) are located in the rigid areas of the face, they undergo almost no local deformation during mouth opening and nodding; their coordinate changes are mainly caused by the overall head movement. Therefore, using the corresponding coordinates of these stable reference points in the two frames, a rigid transformation can be fitted. The transformation includes rotation, scaling, and translation components, and takes the form of a two-dimensional similarity transformation: ,in It is a rotation matrix (including uniform scaling). Let be the translation vector. The fitting process typically uses the least squares method to minimize the error between the transformed coordinates of the stable reference point and the actual coordinates, i.e., to solve for: ; in For a stable set of reference points, and These are the stable reference points s at the mouth-opening frame. and head-nodding peak frames The coordinates in the matrix. Find the optimal... and Then, the rigid motion mapping relationship is obtained. This mapping represents the overall rotation, distance change and translation of the head from the start of nodding to the peak of nodding, and can be used as a benchmark for predicting the theoretical displacement of key points in the lower half of the face.

[0101] Step S062: For each stable reference point, calculate the displacement between the coordinates of the point after rigid motion mapping and its coordinates in the mouth-open holding frame. Sum the displacements of all stable reference points and divide by the number of stable reference points to obtain the overall motion intensity.

[0102] Specifically, upon obtaining Then, the stable reference point is calculated from... arrive Overall exercise intensity : ; in, It represents the number of stable reference points. It is the coordinate of the stable reference point s after being mapped by rigid motion. It is the coordinate of the stable reference point s in the mouth-opening frame. It represents the Euclidean norm of a two-dimensional vector, used to calculate the displacement distance of key points in a two-dimensional image plane.

[0103] From a mathematical perspective, This represents the average displacement length of all key points in the nasal stabilization structure under the rigid transformation of the head nodding motion. From a scenario perspective, It is used to measure whether the current nodding action produces a significant overall movement on the nasal stabilizing structures. A higher value indicates that the nasal stabilizing structures did indeed undergo significant overall movement during nodding, making the nodding more likely to be driven by genuine head movement; conversely, a lower value indicates a lower value. The smaller the value, the weaker the significance of the overall head movement.

[0104] Step S063: The union of the primary support key points, the secondary traction key points, and the mouth center-related key points is taken as the lower half face follow-up key point set; among them, the mouth center-related key points include at least the upper lip center point, the lower lip center point, and the left and right corners of the mouth key points.

[0105] Specifically, the aforementioned set of stable reference points will be used. and the set of key support points at the primary level. and the set of key secondary traction points Simultaneously, from the 106 key points, the center point of the upper lip, the center point of the lower lip, and the corresponding key points at the left and right corners of the mouth were selected as the set of key points related to the center of the mouth. Because during a real nodding process, the overall head movement drives the lower half of the face to follow the stabilizing structures of the nose, thus requiring the lower half of the face to move in the same direction. , and Take the union of the sets to form the set of key points for the lower half of the face. .

[0106] Step S064: For each key point in the set of lower face follow-up key points, subtract its coordinates in the mouth-open holding frame from the coordinates of the key point after rigid motion mapping to obtain the theoretical follow-up displacement vector of the key point; subtract its coordinates in the nodding peak frame from the coordinates of the key point in the mouth-open holding frame to obtain the actual follow-up displacement vector of the key point.

[0107] It should be noted that D only represents the overall motion intensity generated by the nasal stabilization structure during the nodding phase. It indicates the presence of a nodding drive manifested by the nasal region in the current image sequence, but it cannot determine whether the key points of the lower face actually move along with this nodding drive. In deepfake attendance scenarios, there may be situations where the nasal region exhibits nodding changes, while the mouth, chin, jawline, or lower cheek are driven independently by local generation models, leading to inconsistencies between the lower face motion and the nasal nodding drive. Therefore, it is necessary to further calculate the proportion of the actual lower face motion that can be explained by the nasal nodding drive, as well as the remaining unexplained portions.

[0108] Specifically, for the set of key points for lower facial movement Each key point in First, utilize the rigidity change of the nose Predict its peak frame at the nodding position Position in the middle: At the same time, obtain this key point in True detection location in the frame Then, the theoretical follower displacement vector and the actual follower displacement vector are calculated separately: theoretical follower displacement vector This indicates the displacement (including direction and amplitude) of key points on the lower face that should follow the movement of the nasal stabilizing structures if nodding is driven by the actual overall movement of the head; the actual following displacement vector. This indicates the actual displacement of the key point during the nodding process. By comparing the two, the consistency between the lower face homing and the rigid drive of the nose can be quantified.

[0109] Step S065: Calculate the sum of the square of the magnitude of the theoretical follower displacement vector and the preset zero constant, and divide the dot product of the theoretical follower displacement vector and the actual follower displacement vector by the sum to calculate the follower response ratio of the key point.

[0110] Specifically, calculate the servo response ratio: ; in, This represents the proportion of the follow-up response at the key point q. Represents the theoretical follower displacement vector Length of the module The square of. This represents the preset zero-prevention constant, which is an extremely small positive number (e.g., ...). This is added to the denominator to prevent division by zero errors, ensure the numerical stability of the calculation, and not affect the normal calculation results.

[0111] molecular The dot product of two two-dimensional displacement vectors measures the consistency of the directions of the two vectors and the projected length of one vector onto the other. From a contextual perspective, it represents the first... The amount of motion that can be explained by the direction of the nose nodding drive in the actual movement of key points on the lower half of the face.

[0112] symbol This represents the Euclidean norm of a two-dimensional vector.

[0113] denominator It is the squared length of the predicted displacement vector, that is, the theoretical motion intensity that the nasal rigidity drive prediction brings to this key point. Its significance in the scenario is that the predicted motion intensity is used as a reference scale to measure the proportion of the actual lower face movement to the drive response.

[0114] From the formula as a whole, its mathematical meaning is to express the actual displacement. Projected onto predicted displacement In terms of direction, we obtain the normalized proportion of the actual displacement that can be explained by the rigid motion of the nose. The significance of this approach lies in quantifying the first... The degree of follow-up motion of a key point on the lower half of the face to the rigid drive of the nose nodding, that is, the degree of consistency between the actual displacement of the key point and the theoretical displacement predicted based on the rigid motion.

[0115] Step S066: For each key point in the set of follow-up key points of the lower half of the face, multiply the follow-up response ratio of the key point by the theoretical follow-up displacement vector to obtain the first product vector, and subtract the first product vector from the actual follow-up displacement vector to obtain the residual vector.

[0116] Specifically, Denote it as the first product vector. It is denoted as the residual vector, which is the uninterpretable residual displacement of the q-th key point.

[0117] Step S067: Summate the magnitudes of the residual vectors of all key points to obtain the first sum, and sum the magnitudes of the actual moving displacement vectors of all key points to obtain the second sum.

[0118] Specifically, Recorded as the first sum, This is denoted as the second sum.

[0119] Step S068: Divide the first sum by the second sum to obtain the unexplained residual proportion of the lower half of the face.

[0120] Specifically, the set of key points for motion tracking of the lower half of the face. The residual displacements of all key points in the middle are aggregated to obtain the unexplained residual proportions of the lower half. : ; molecular The mathematical meaning of "nodding" is the total residual motion of all key points in the lower face that cannot be explained by the rigidity of the nose during nodding. From a scenario perspective, during a real nodding, the stable structure of the nose represents the overall movement trend of the head, and most of the key points in the lower face should follow this trend; therefore, the actual displacement... Able to be better by The explanation is that the residual term is relatively small; however, in deepfake scenarios, although the locally generated regions appear to be moving, they do not follow the actual head nodding, causing the actual displacement to deviate from the overall nodding trend, resulting in a significantly larger residual term.

[0121] denominator The mathematical meaning of is the total actual movement of all key points on the lower half of the face during the nodding process. Its significance in the context is that the nodding amplitude varies among different people, and the amplitude may also differ at different times during the same check-in. Therefore, it is necessary to normalize the residual movement using the actual total movement.

[0122] The overall mathematical meaning of the formula is the proportion of the actual movement of the lower face that cannot be explained by the rigidity of the nose nodding, i.e., the degree of movement inconsistency. In practical terms: the smaller the value, the more the lower face appears to follow the actual head nodding movement; the larger the value, the more likely the lower face is to have anomalies such as local generation, texture drift, or asynchronous movement.

[0123] Step S069: Divide the overall motion intensity by the unexplained residual ratio of the lower half of the face to calculate the head nodding rigid drive consistency amount.

[0124] Specifically, in summary, for a real person to nod while keeping their mouth open, two conditions must be met simultaneously: first, the nasal stabilizing structures should produce a noticeable nodding movement, i.e., the overall movement intensity should be [missing information]. The value is relatively large; secondly, the movement of the lower half of the face should be explainable by the nodding, i.e., the inexplicable residual proportion. The value is relatively small. Therefore, by... and The ratio calculation yields the head-nodding rigid drive consistency amount. Its significance lies in determining whether the current nodding action is driven by the overall head movement represented by the stabilizing structures of the nose, and whether this driving force can effectively induce the lower half of the face to move together. In other words, This is used to measure whether a nodding motion is actually driven by the overall movement of the head, which in turn coordinates with the lower half of the face.

[0125] Step S007: Calculate the degree of structural coupling retention using the transmission amount of the support chain and the consistency amount of the head rigid drive.

[0126] It should be noted that: after obtaining the support chain conductivity... Consistent with rigid drive of nodding Then, the degree of structural coupling retention between the mouth-opening support chain and the head-nodding rigid drive was further calculated.

[0127] When a real person performs the compound action of "nodding while keeping their mouth open," the lower facial support chain formed during the mouth-opening phase should not break during the nodding phase, but should remain intact along with the overall rigid nodding movement of the head. In other words, the action of a real person not only requires... Larger, and also requires Larger. If Larger and A smaller value indicates that a relatively clear support chain may have formed during the mouth-opening phase, but this support chain failed to maintain consistency with the overall head movement during the nodding phase, possibly indicating local texture drift or asynchronous movement. Smaller and A relatively large value indicates a possible overall movement trend during the nodding phase, but the mouth-opening phase lacks the continuous support chain from the mouth to the jaw and cheeks that a real person would exhibit, potentially resulting in changes only in the lips or oral cavity area. If and If both are relatively small, it indicates that the mouth-opening support chain and the head-nodding rigid drive are insufficient, and the current check-in behavior has a high risk of proxy check-in.

[0128] Step S007 specifically includes: Step S0071: Multiply the normalized support chain conduction quantity, the normalized nodding rigid drive consistency quantity, and 2 together to obtain the first product.

[0129] Specifically, due to the transmission capacity of the support chain Consistent with rigid drive of nodding These originate from the mouth-opening and head-nodding phases, respectively, and their numerical scales typically differ. Therefore, we first apply their corresponding normalized reference thresholds. and ,Will and Mapped to a unified The interval is used to obtain the normalized support chain propagation confidence value. and the reliability of rigid driving of nodding .in, and They are from real samples. and The distribution statistics can be used, for example, to take the median of the corresponding sample set or a preset quantile value (lower quartile or upper quartile) to ensure that the normalized values ​​are comparable.

[0130] It is denoted as the first product.

[0131] Step S0072: Add the normalized support chain conduction quantity, the normalized head nodding rigid drive consistency quantity, and the preset zero-prevention constant to obtain the third sum.

[0132] Specifically, This is denoted as the third sum. A preset zero-prevention constant. It is a very small positive number (e.g.) This is used to prevent division by zero in the denominator, ensuring numerical stability without affecting normal calculation results.

[0133] Step S0073: Divide the first product by the third sum to obtain the degree of structural coupling retention.

[0134] Specifically, the degree of structural coupling retention between the mouth-opening support chain and the head-nodding rigid drive is calculated. : ; From a mathematical perspective, this formula is used to calculate the normalized support chain propagation confidence level. With head-nodding rigid drive credibility The degree of coordination between them. From a scenario perspective, it measures whether the lower facial support chain formed during the mouth-opening phase maintains structural coupling during the subsequent rigid nodding process. If A larger value indicates that the person has not only completed a relatively realistic mouth-opening support chain, but that the support chain can also remain consistent with the overall head movement during the nodding phase. In this case, the current check-in behavior is closer to the movement pattern of a real person.

[0135] Step S008: Compare the degree of structural coupling retention with the preset judgment threshold, and determine whether the current check-in behavior is a real check-in based on the comparison result.

[0136] Step S008 specifically includes: Step S0081: If the degree of structural coupling retention is greater than or equal to the preset judgment threshold, then the current check-in behavior is determined to be a real check-in.

[0137] Step S0082: If the degree of structural coupling retention is less than the preset judgment threshold, it is determined that the current check-in behavior has the risk of proxy check-in.

[0138] Specifically, after obtaining the degree of structural coupling preservation, it is compared with a preset judgment threshold. A comparison is made to determine whether the current check-in behavior carries the risk of proxy check-in. The threshold for judging the degree of structural coupling preservation is derived from the statistical results of the samples during the calibration phase (such as taking real samples). The lower quartile or median of the value, or the optimal cutoff point determined by the ROC curve. In attendance scenarios with high security requirements, priority should be given to controlling the probability of proxy attendance samples being misclassified as genuine attendance (i.e., reducing the false negative rate), and accordingly, the rate should be appropriately increased. Preferably, The value ranges from 0.6 to 0.8; 0.6 is suitable for ordinary scenarios, while 0.7 or 0.8 is suitable for high-security scenarios. The specific value can be calibrated experimentally based on the actual application environment and false alarm tolerance.

[0139] The judgment method is as follows: If If the current check-in behavior is judged to be authentic; If so, it is determined that the current check-in behavior carries the risk of being checked in by proxy.

[0140] When the current check-in behavior passes the authenticity judgment, the system confirms that the check-in is valid, records the check-in verification result, generates a normal check-in record, and includes it in the attendance statistics process.

[0141] When a current check-in fails the authenticity check, the system does not directly generate a normal check-in record. Instead, it outputs an abnormal check-in prompt and may prompt the person checking in to complete the composite action data collection again. If the data collection still fails the check-in after re-collection, the system marks this check-in as suspected proxy check-in and saves the relevant images and key information for further verification by the back-end management or manual review module.

[0142] In summary, in this embodiment of the invention, by performing temporal comparison of facial key points on consecutive frames of facial images of the target object, the transmission relationship between the mouth opening change and the support chain between the jaw and cheeks during the mouth opening phase is first analyzed to determine whether the mouth opening action conforms to the biomechanical linkage law of a real person; then, the rigid drive consistency relationship when performing a nodding action while maintaining a mouth opening is further analyzed to verify whether the lower half of the face undergoes overall rigid movement with the stabilizing structure of the nose. Through the above two-stage collaborative analysis, this embodiment can effectively identify deepfake-based face-swapping attacks and various proxy attendance behaviors, significantly improving the accuracy and security of facial attendance recognition.

[0143] This invention also proposes a system for recognizing proxy attendance based on image feature comparison; please refer to [link / reference]. Figure 2 The diagram illustrates a block diagram of a proxy attendance recognition system based on image feature comparison according to an embodiment of the present invention. The system includes:

[0144] The acquisition module 100 is used to acquire continuous frames of raw facial images of the target object in response to the target object's check-in request during the current check-in process;

[0145] Analysis module 200 is used to extract facial key points from consecutive frames of raw face images and obtain key point information;

[0146] Based on key point information, determine the closed reference frame before the mouth opening action begins and the mouth opening hold frame after the nodding action begins.

[0147] Calculate the mouth opening change vector based on key point information in the closed reference frame and the open mouth holding frame;

[0148] Using key point information, closed reference frames, open-mouth holding frames, and mouth opening change vectors, the support chain conduction is calculated.

[0149] The peak frame of the nodding action is determined based on key point information, and the stiffness driving consistency of the nodding action is calculated using key point information, mouth opening holding frame and peak frame.

[0150] The degree of structural coupling retention is calculated by utilizing the transmission capacity of the support chain and the consistency of the head stiffness drive.

[0151] The judgment module 300 is used to compare the degree of structural coupling retention with a preset judgment threshold, and to determine whether the current check-in behavior is a real check-in based on the comparison result.

[0152] It should be noted that the system provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image feature comparison-based attendance-taking behavior recognition system and the image feature comparison-based attendance-taking behavior recognition method embodiment provided in the above embodiments belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0153] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0154] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0155] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for identifying a proxy punching behavior based on image feature comparison, characterized in that, The method includes the following steps: In response to the check-in request from the target object, continuously capture the original face images of the target object in the current check-in process; Extract facial key points from consecutive frames of raw facial images and obtain key point information; Based on key point information, determine the closed reference frame before the mouth opening action begins and the mouth opening hold frame after the nodding action begins. Calculate the mouth opening change vector based on key point information in the closed reference frame and the open mouth holding frame; Using keypoint information, closed-mouth reference frames, open-mouth holding frames, and mouth opening change vectors, the support chain conductance is calculated, specifically including: The key points of the chin and jawline are used as primary support key points, and the key points of the lower edge of the left and right cheeks are used as secondary traction key points. For each primary support keypoint, subtract the coordinates of the keypoint in the aligned open mouth holding frame from the coordinates of the keypoint in the aligned closed mouth reference frame to obtain the two-dimensional displacement vector of the keypoint. Calculate the square of the magnitude of the mouth opening change vector, and divide the dot product of the two-dimensional displacement vector of each primary support key point and the mouth opening change vector by the square to obtain the first ratio of each primary support key point. The summation of the first ratios of all primary support key points is divided by the number of primary support key points to obtain the aggregate support response of the primary support key points. Obtain the aggregated support response volume of the secondary key points; The aggregated support response quantities of the primary support key points and the secondary traction key points are respectively positiveized and fault-tolerant to obtain the processed primary support response quantities and secondary traction response quantities. Multiply the processed primary support response and secondary traction response to obtain the support chain transmission quantity; The peak frame of the nodding action is determined based on key point information, and the stiffness driving consistency of the nodding action is calculated using key point information, mouth opening holding frame and peak frame. The degree of structural coupling retention is calculated by utilizing the transmission capacity of the support chain and the consistency of the head stiffness drive. The degree of structural coupling retention is compared with a preset judgment threshold, and the current check-in behavior is judged as a genuine check-in based on the comparison result.

2. The method for identifying proxy attendance based on image feature comparison according to claim 1, characterized in that, The specific steps for extracting facial key points from consecutive frames of original facial images and obtaining key point information are as follows: Each frame of the original face image is subjected to image enhancement and size normalization processing to obtain the first intermediate image; Perform face detection and alignment on the first intermediate image to obtain consecutive frame face images; Each frame of face image is input into a pre-trained 106-point facial landmark localization model, which outputs the coordinates of 106 landmarks in each frame. By associating the coordinates of each keypoint, its corresponding number, and the frame time in which it is located, we can obtain the information of each keypoint.

3. The method for identifying proxy attendance based on image feature comparison according to claim 2, characterized in that, The specific steps involved in determining the mouth-closing reference frame before the mouth-opening action and the mouth-opening hold frame after the head-nodding action begins, based on key point information, are as follows: The center points of the upper and lower lips are obtained from the lip key points in consecutive frames of face images; Based on the distance change curve between the center points of the upper and lower lips, a closed reference frame and an open-mouth holding frame are determined. The closed reference frame is the last frame before the mouth opening action begins, when the distance between the center points of the upper and lower lips is less than a preset first distance threshold. The open-mouth holding frame is the first frame after the nodding action begins, when the distance between the center points of the upper and lower lips is greater than a preset second distance threshold.

4. The method for identifying proxy attendance based on image feature comparison according to claim 3, characterized in that, The specific steps for calculating the mouth opening change vector based on key point information in the closed-mouth reference frame and the open-mouth holding frame are as follows: Based on the information of each key point, the key points of the nose are obtained locally, and the key points of the nose are used as stable reference points; among them, the key points of the nose locally include at least the key points of the bridge of the nose, the tip of the nose, and the left and right nostrils. Using stable reference points, a two-dimensional similarity transformation is performed on key points in the closed-mouth reference frame and the open-mouth retention frame to align them, and the coordinates of the key points after the two frames are aligned are obtained. Key points of the lips are selected from the key point coordinates of the two frames after alignment, and the average coordinates of the key points of the lips are calculated to obtain the center point of the upper lip and the center point of the lower lip in each frame. The relative position vector from the center point of the upper lip to the center point of the lower lip is obtained in each frame. The relative position vector corresponding to the open mouth frame is subtracted from the relative position vector corresponding to the closed mouth reference frame to obtain the mouth opening change vector.

5. The method for identifying proxy attendance based on image feature comparison according to claim 1, characterized in that, The specific steps involved in determining the peak frame of the nodding action based on key point information are as follows: The time interval between the open-mouth holding frame and the last frame of the entire acquisition process is defined as the duration of the nodding action; For each frame within the continuous time period, the head pitch angle of that frame is obtained using a head pose estimation algorithm, and the absolute difference between the head pitch angle of that frame and the head pitch angle of the open-mouth holding frame is calculated. The frame with the largest absolute difference is taken as the peak frame of the head nod.

6. The method for identifying proxy attendance based on image feature comparison according to claim 5, characterized in that, The specific steps for calculating the head nodding rigidity drive consistency amount using key point information, mouth opening and holding frames, and head nodding peak frames are as follows: Based on the coordinates of the stable reference point in the mouth-opening frame and the head-nodding peak frame, a rigid motion mapping relationship is fitted. For each stable reference point, calculate the displacement between the coordinates of that point after rigid motion mapping and its coordinates in the mouth-open holding frame. Sum the displacements of all stable reference points and divide by the number of stable reference points to obtain the overall motion intensity. The union of primary support key points, secondary traction key points, and mouth center-related key points is taken as the set of lower face follow-up key points; among them, mouth center-related key points include at least the upper lip center point, the lower lip center point, and the left and right corners of the mouth key points; For each key point in the set of key points for the lower half of the face, subtract its coordinates in the open-mouth holding frame from the coordinates of the key point after rigid motion mapping to obtain the theoretical follow-up displacement vector of the key point; subtract its coordinates in the nodding peak frame from the coordinates of the key point in the open-mouth holding frame to obtain the actual follow-up displacement vector of the key point. Calculate the sum of the square of the magnitude of the theoretical follower displacement vector and the preset zero-prevention constant, and divide the dot product of the theoretical follower displacement vector and the actual follower displacement vector by this sum to calculate the follower response ratio of the key point. For each key point in the set of follow-up key points in the lower half of the face, the follow-up response ratio of the key point is multiplied by the theoretical follow-up displacement vector to obtain the first product vector, and the actual follow-up displacement vector is subtracted from the first product vector to obtain the residual vector. The first sum is obtained by summing the magnitudes of the residual vectors of all key points, and the second sum is obtained by summing the magnitudes of the actual displacement vectors of all key points. Dividing the first sum by the second sum yields the unexplained residual proportions of the lower half of the face. Divide the overall motion intensity by the unexplained residual proportion of the lower half of the face to calculate the head nodding rigid drive consistency.

7. The method for identifying proxy attendance based on image feature comparison according to claim 6, characterized in that, The specific steps for calculating the structural coupling retention degree using the support chain conductance and the head stiffness driving consistency are as follows: Multiply the normalized support chain conduction, the normalized nodding rigid drive consistency, and 2 together to obtain the first product; The normalized support chain conductance, the normalized head stiffness drive consistency, and the preset zero-prevention constant are added together to obtain the third sum. Dividing the first product by the third sum yields the degree of structural coupling retention.

8. The method for identifying proxy attendance based on image feature comparison according to claim 7, characterized in that, The specific steps involved in comparing the degree of structural coupling retention with a preset judgment threshold and determining whether the current check-in behavior is a genuine check-in are as follows: If the degree of structural coupling retention is greater than or equal to the preset judgment threshold, then the current check-in behavior is determined to be a genuine check-in. If the degree of structural coupling retention is less than the preset judgment threshold, then it is determined that the current check-in behavior has the risk of proxy check-in.

9. A system for recognizing proxy attendance based on image feature comparison, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for identifying proxy attendance behavior based on image feature comparison as described in any one of claims 1-8.

Images