Intelligent attendance management system based on multi-modal fusion

Abstract

The invention belongs to the technical field of attendance management, and particularly relates to an intelligent attendance management system based on multi-modal fusion, which realizes leap-type upgrade from identity management to state insight by introducing deep learning face detection and expression recognition and fusing a large model workflow and a physical examination report knowledge base, changes passive management into active attention, and improves the efficiency of attendance management. Real-time risk intervention is achieved, and active risk identification and real-time risk intervention are achieved by introducing expression identification and large model workflow.

Description

Technical Field

[0001] This invention relates to the field of employee attendance management technology, specifically to a smart attendance management system based on multimodal fusion. Background Technology

[0002] Employee attendance is a crucial aspect of corporate management. However, with the development of artificial intelligence, traditional attendance methods are increasingly unable to meet the needs of intelligent office operations and employee well-being. Currently, most attendance systems (such as traditional card swiping, fingerprint, or basic facial recognition) have relatively simple functions. Their core objective is to verify identity and record on-duty time, making it difficult to capture and analyze employees' emotional and health status. This "sign-in" management approach fails to fully utilize employees' emotional states during attendance to conduct reasonable mental health assessments. This prevents management from timely understanding employees' true work emotions and potential mental health issues, leading to a series of risks for the company due to a lack of humanistic care (such as the loss of key personnel, reduced work efficiency, or operational errors). Currently, there are two main implementation methods: traditional biometric / media-based attendance systems and basic facial recognition attendance systems. Traditional biometric / media-based attendance systems primarily rely on fingerprint recognition or IC card (employee badge) swiping. The implementation involves employees physically contacting the attendance terminal (fingerprinting) or swiping their card. The terminal compares the employee's identity information with a local or online database, and records the current timestamp after successful verification. This method is simple to implement and relatively low-cost, but it suffers from inefficiency (prone to queuing), hygiene risks (fingerprint contact), and the easy loss or forgetting of the contact information. Furthermore, it cannot obtain any information other than the time of identification. Basic facial recognition attendance systems are currently a more advanced solution, such as the system disclosed in patent CN114093050A. The implementation involves a backend cloud platform distributing the employee's facial database to the frontend facial recognition attendance machine. Employees undergo facial recognition at the attendance machine, which performs liveness detection and 1:N face comparison. After identity confirmation, the attendance machine saves the attendance record locally and simultaneously uploads it to the cloud management platform. Administrators can manage the face database and view attendance records on the cloud platform. This method achieves contactless attendance and is highly efficient, but its essential function is still "identity verification + time recording." Mainstream facial recognition attendance systems suffer from three significant drawbacks: First, limited information dimensions: the system's core only addresses identity verification ("who" and "when"), completely ignoring the rich information conveyed by an employee's face, such as fatigue and stress, and failing to perceive their true emotions or mental health. Second, lagging data application: data is only used for post-event management such as payroll calculation, resulting in a passive and delayed application model. The system lacks proactive early warning capabilities, failing to provide timely alerts when employees exhibit persistent abnormal states (such as continuous fatigue). Third, isolated management functions: attendance data forms an "information silo," confined to the human resources domain, unable to be linked and analyzed with key business data such as project progress and team performance, hindering data-driven in-depth decision-making. This invention addresses these issues by proposing a multimodal intelligent attendance management system that combines high-precision facial recognition technology with large-scale models for longitudinal health trend analysis. Summary of the Invention

[0003] The purpose of this section is to outline some aspects of the embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0004] Therefore, the purpose of this invention is to provide a smart attendance management system based on multimodal fusion. By introducing deep learning face detection and expression recognition, and integrating a large model workflow and a physical examination report knowledge base, it achieves a leapfrog upgrade from identity management to status insight, transforming passive management into proactive attention, and realizing real-time risk intervention. By introducing expression recognition and a large model workflow, it achieves proactive risk identification and real-time intervention.

[0005] To address the aforementioned technical problems, according to one aspect of the present invention, the present invention provides the following technical solution: A smart attendance management system based on multimodal fusion, comprising: The input module is used to collect facial image data of employees in real time; The algorithm detection and recognition module, deployed on an edge computing device, is used to process the facial image data and perform face identity recognition and facial expression recognition in parallel. The status assessment and early warning module is used to perform continuous or frequency analysis on the identified facial expression sequences based on a preset time window and expression fluctuation assessment threshold, and generate an early warning signal when the early warning conditions are met. The multimodal fusion analysis module, deployed on a server or in the cloud, responds to the warning signal and is used to dynamically call the employee's historical health text data stored in the knowledge base based on the employee's identity information, and use a large model workflow to fuse and analyze the visual emotion modality of real-time facial expression recognition with the historical health text modality to generate health assessment suggestions. The output and push module is used to push the health assessment recommendations to the visual attendance management platform and / or employees' mobile devices.

[0006] As a preferred embodiment of the intelligent attendance management system based on multimodal fusion described in this invention, the algorithm detection and recognition module includes: The identity recognition unit uses the YOLOV8-face model for face detection and key point localization, and performs high-speed comparison between the extracted face feature vector and the face feature database to complete identity verification. The expression recognition unit calculates the geometric relationship between key points based on the facial key points located by the YOLOV8-face model to form a feature vector, and / or inputs the cropped face region image into a convolutional neural network to extract deep features, and outputs the probability distribution of belonging to a predefined expression category through a classifier.

[0007] As a preferred embodiment of the intelligent attendance management system based on multimodal fusion described in this invention, the preset facial expression fluctuation assessment threshold in the state assessment and early warning module is specifically: In the facial expression recognition results of the most recent N consecutive attendance check-ins, the number of times the specified negative expression category appeared exceeded M, where N and M are integers greater than 1, and M is less than N.

[0008] As a preferred embodiment of the intelligent attendance management system based on multimodal fusion described in this invention, the knowledge base in the multimodal fusion analysis module is a structured physical examination report knowledge base, which stores employees' historical physical examination index data and medical terminology information; The large model workflow automatically queries and associates corresponding data in the knowledge base based on the employee identity index.

[0009] As a preferred embodiment of the intelligent attendance management system based on multimodal fusion described in this invention, the input module is a face capture attendance machine, which integrates a high-definition network camera, a voice prompt unit, and an access control signal output interface. The visual attendance management platform in the output and push module is used to display employee attendance status, abnormal expression records, and health assessment suggestions to managers to assist in management decisions.

[0010] A smart attendance management method based on multimodal fusion, applied to any of the systems described above, the method comprising: S301. Employees use a facial recognition attendance machine to take photos of their faces to clock in. S302, The edge computing device receives captured facial photos; S303. Edge computing devices call algorithm models to locate and detect faces; S304. The algorithm model matches the detected face with identity information and recognizes facial expressions; S305. Determine whether the face in the photo is an employee face in the face database and return the result; S306. Face matching successful. Employee completes attendance check-in and uploads the check-in record to the visual attendance management platform. S307. Determine whether the faces in the photo show abnormal expressions such as fatigue or frustration, assess the expression fluctuation threshold, and return the results. S308. When an employee displays abnormal facial expressions, the employee's identity and facial expression information are transmitted to the cloud for health analysis and prediction. S309. The cloud-based health checkup report knowledge base is accessed for in-depth analysis and online searches to provide health assessment suggestions. S310. Send employee health assessment recommendations to the attendance management platform and push them to employees' mobile devices; S311. The attendance management platform exports employee attendance status records and health advice assessments to assist managers in decision-making. S312, End.

[0011] As a preferred embodiment of the intelligent attendance management method based on multimodal fusion described in this invention, in step S304, the algorithm model is a state assessment based on a preset facial expression fluctuation assessment threshold, and the algorithm model specifically includes: Maintain a sequence of employee facial expression recognition results within a time window; Statistically analyze the frequency or persistence of a specified negative facial expression category in the sequence; When the frequency or duration exceeds a preset threshold, an early warning is triggered.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows: By introducing deep learning face detection and expression recognition, and integrating large model workflows with a health check report knowledge base, a leapfrog upgrade from identity management to status insight has been achieved; Transforming passive management into proactive attention and enabling real-time risk intervention, the system introduces facial expression recognition and large-scale model workflows to achieve proactive risk identification and real-time intervention. When the system identifies persistent negative emotions, it can immediately trigger analysis and push processes, presenting potential mental health issues or job burnout risks to managers and employees at the nascent stage. By promptly sending personalized health assessments and suggestions, employees can feel the company's personalized care based on objective data. This greatly enhances employees' sense of belonging and satisfaction, safeguards their physical and mental health, and promotes the company's development in a more humane direction. Attached Figure Description

[0013] To more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and detailed embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a schematic block diagram of the system modules of the present invention; Figure 2 This is a flowchart of the detection and recognition module of the algorithm in this invention; Figure 3 This is a flowchart of the health prediction large-scale model of the present invention; Figure 4 This is a schematic diagram illustrating the steps of the multimodal fusion-based intelligent attendance management method of the present invention. Detailed Implementation

[0014] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0015] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0016] Secondly, the present invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of the present invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not according to the usual scale. Furthermore, the schematic diagrams are merely examples and should not limit the scope of protection of the present invention. In addition, actual fabrication should include three-dimensional spatial dimensions of length, width, and depth.

[0017] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0018] This invention provides a smart attendance management system based on multimodal fusion. Please refer to [link / reference]. Figure 1-4 ,include: The input module is used to collect facial image data of employees in real time; The algorithm detection and recognition module, deployed on an edge computing device, is used to process the facial image data and perform face identity recognition and facial expression recognition in parallel. The status assessment and early warning module is used to perform continuous or frequency analysis on the identified facial expression sequences based on a preset time window and expression fluctuation assessment threshold, and generate an early warning signal when the early warning conditions are met. The multimodal fusion analysis module, deployed on a server or in the cloud, responds to the warning signal and is used to dynamically call the employee's historical health text data stored in the knowledge base based on the employee's identity information, and use a large model workflow to fuse and analyze the visual emotion modality of real-time facial expression recognition with the historical health text modality to generate health assessment suggestions. The output and push module is used to push the health assessment recommendations to the visual attendance management platform and / or employees' mobile devices; The algorithm detection and recognition module includes: The identity recognition unit uses the YOLOV8-face model for face detection and key point localization, and performs high-speed comparison between the extracted face feature vector and the face feature database to complete identity verification. The facial expression recognition unit calculates the geometric relationship between key points to form a feature vector based on the facial key points located by the YOLOV8-face model, and / or inputs the cropped face region image into a convolutional neural network to extract deep features, and outputs the probability distribution of belonging to a predefined expression category through a classifier; The preset facial expression fluctuation assessment threshold in the state assessment and early warning module is specifically as follows: In the facial expression recognition results of the most recent N consecutive attendance check-ins, the number of times the specified negative expression category appeared exceeded M, where N and M are integers greater than 1, and M is less than N; The knowledge base in the multimodal fusion analysis module is a structured physical examination report knowledge base, which stores employees' historical physical examination index data and medical terminology information; The large model workflow is based on the employee identity index and automatically queries and associates the corresponding data in the knowledge base. The input module is a face capture attendance machine, which integrates a high-definition network camera, a voice prompt unit, and an access control signal output interface; The visual attendance management platform in the output and push module is used to display employee attendance status, abnormal expression records, and health assessment suggestions to managers to assist in management decisions.

[0019] A smart attendance management method based on multimodal fusion includes: S301. Employees use a facial recognition attendance machine to take photos of their faces to clock in. S302, The edge computing device receives captured facial photos; S303. Edge computing devices call algorithm models to locate and detect faces; S304. The algorithm model matches the detected face with identity information and recognizes facial expressions; S305. Determine whether the face in the photo is an employee face in the face database and return the result; S306. Face matching successful. Employee completes attendance check-in and uploads the check-in record to the visual attendance management platform. S307. Determine whether the faces in the photo show abnormal expressions such as fatigue or frustration, assess the expression fluctuation threshold, and return the results. S308. When an employee displays abnormal facial expressions, the employee's identity and facial expression information are transmitted to the cloud for health analysis and prediction. S309. The cloud-based health checkup report knowledge base is accessed for in-depth analysis and online searches to provide health assessment suggestions. S310. Send employee health assessment recommendations to the attendance management platform and push them to employees' mobile devices; S311. The attendance management platform exports employee attendance status records and health advice assessments to assist managers in decision-making. S312, End; In step S3, a state assessment is performed based on a preset facial expression fluctuation assessment threshold, specifically including: Maintain a sequence of employee facial expression recognition results within a time window; Statistically analyze the frequency or persistence of a specified negative facial expression category in the sequence; When the frequency or duration exceeds a preset threshold, it is determined that an early warning is triggered; in: Identity recognition (based on YOLOV8-face); The algorithm first uses the YOLOv8-face model to perform face detection and keypoint localization on the input image. YOLOv8-face is a dedicated face detection model based on the YOLOv8 framework, capable of simultaneously outputting high-precision face bounding boxes and facial keypoints. Its core idea is to treat the detection task as an ensemble loss optimization problem, with the total loss function... Typically composed of classification loss ( ), regression loss ( ) and keypoint / distribution loss ( or )constitute: ; in: The classification loss typically uses binary cross-entropy loss (BCE Loss) to determine whether the predicted bounding box contains a face.

[0020] The bounding box regression loss uses a method such as Complete IoU Loss to minimize the geometric differences (including overlap, center distance, and aspect ratio) between the predicted and ground truth boxes.

[0021] ; (where is) Intersection, Union, and Comparison Distance from the center point The diagonal of the smallest bounding rectangle. Used to measure aspect ratio consistency.

[0022] After obtaining facial features, the system calls a face matching algorithm to perform high-speed comparison (such as using cosine similarity) between the extracted feature vector and the features of N identities in the face database, and finds the identity with the highest similarity that exceeds a preset threshold (such as 0.9) to complete identity recognition. Facial Expression Recognition; Based on YOLOV8-face localization of faces and key points (such as the outline points of eyebrows, eyes, nose, and mouth), the system starts the expression recognition subtask. The algorithm forms feature vectors by calculating the geometric relationships between key points (such as distance and angle), or inputs the cropped face image into a lightweight convolutional neural network (CNN) to extract deep features. This feature vector (in The input features are fed into a classifier, and the final output of the classifier (such as a fully connected layer) is then processed. The function is converted into a probability distribution to determine the probability of belonging to each predefined expression category (e.g., normal, happy, tired, frustrated). ; in It is a classification layer for the first The output logits for each emoji category This is the total number of expression categories; the system selects the category with the highest probability as the final recognition result. State Assessment and Early Warning Module: To avoid alarm fatigue caused by occasional emotional fluctuations (such as yawning), an facial expression fluctuation assessment and early warning mechanism is introduced. This module does not issue early warnings for single abnormal facial expressions (such as fatigue or frustration), but rather maintains a sequence of employee facial expressions over a time window (e.g., the most recent 5 working days) on edge computing devices or servers. The system assesses the persistence or frequency of negative emotions through threshold judgment (step 307). For example, when an employee's facial expression is detected... In the last 5 check-ins, there were The warning condition is triggered only when the employee is identified as tired or frustrated, and the employee's identity and facial expression information (step 308) are passed to the next module; Multimodal fusion analysis module: This module consists of a server / cloud large model (16) and a physical examination report knowledge base, and is the core of realizing in-depth health assessment.

[0023] The health check report knowledge base is a structured, secure, and controllable data source that stores employees' historical health data (such as medical terminology and indicator values), which has been preprocessed into machine-readable knowledge assets.

[0024] Upon receiving employee identity and real-time facial expression information (Modal 1: Visual Emotion) from the alert module, the server initiates a large-scale model workflow built on platforms such as Dify (see...). Figure 3 The workflow uses employee identity as an index to automatically retrieve the employee's recent physical examination report from the knowledge base. The large model performs deep fusion and causal analysis on these two distinct modalities of data (step 309).

[0025] Although the present invention has been described above with reference to embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of the invention. In particular, as long as there is no structural conflict, the features in the disclosed embodiments can be combined with each other in any manner. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A smart attendance management system based on multimodal fusion, characterized in that, include: The input module is used to collect facial image data of employees in real time; The algorithm detection and recognition module, deployed on an edge computing device, is used to process the facial image data and perform face identity recognition and facial expression recognition in parallel. The status assessment and early warning module is used to perform continuous or frequency analysis on the identified facial expression sequences based on a preset time window and expression fluctuation assessment threshold, and generate an early warning signal when the early warning conditions are met. The multimodal fusion analysis module, deployed on a server or in the cloud, responds to the warning signal and is used to dynamically call the employee's historical health text data stored in the knowledge base based on the employee's identity information, and use a large model workflow to fuse and analyze the visual emotion modality of real-time facial expression recognition with the historical health text modality to generate health assessment suggestions. The output and push module is used to push the health assessment recommendations to the visual attendance management platform and / or employees' mobile devices.

2. The intelligent attendance management system based on multimodal fusion according to claim 1, characterized in that, The algorithm detection and recognition module includes: The identity recognition unit uses the YOLOV8-face model for face detection and key point localization, and performs high-speed comparison between the extracted face feature vector and the face feature database to complete identity verification. The expression recognition unit calculates the geometric relationship between key points based on the facial key points located by the YOLOV8-face model to form a feature vector, and / or inputs the cropped face region image into a convolutional neural network to extract deep features, and outputs the probability distribution of belonging to a predefined expression category through a classifier.

3. The intelligent attendance management system based on multimodal fusion according to claim 1, characterized in that, The preset facial expression fluctuation assessment threshold in the state assessment and early warning module is specifically as follows: In the facial expression recognition results of the most recent N consecutive attendance check-ins, the number of times the specified negative expression category appeared exceeded M, where N and M are integers greater than 1, and M is less than N.

4. The intelligent attendance management system based on multimodal fusion according to claim 1, characterized in that, The knowledge base in the multimodal fusion analysis module is a structured physical examination report knowledge base, which stores employees' historical physical examination index data and medical terminology information; The large model workflow automatically queries and associates corresponding data in the knowledge base based on the employee identity index.

5. The intelligent attendance management system based on multimodal fusion according to claim 1, characterized in that, The input module is a face capture attendance machine, which integrates a high-definition network camera, a voice prompt unit, and an access control signal output interface; The visual attendance management platform in the output and push module is used to display employee attendance status, abnormal expression records, and health assessment suggestions to managers to assist in management decisions.

6. A smart attendance management method based on multimodal fusion, characterized in that, The method, applied to the system of any one of claims 1 to 5, comprises: S301. Employees use a facial recognition attendance machine to take photos of their faces to clock in. S302, The edge computing device receives captured facial photos; S303. Edge computing devices call algorithm models to locate and detect faces; S304. The algorithm model matches the detected face with identity information and recognizes facial expressions; S305. Determine whether the face in the photo is an employee face in the face database and return the result; S306. Face matching successful. Employee completes attendance check-in and uploads the check-in record to the visual attendance management platform. S307. Determine whether the faces in the photo show abnormal expressions such as fatigue or frustration, assess the expression fluctuation threshold, and return the results. S308. When an employee displays abnormal facial expressions, the employee's identity and facial expression information are transmitted to the cloud for health analysis and prediction. S309. The cloud-based health checkup report knowledge base is accessed for in-depth analysis and online searches to provide health assessment suggestions. S310. Send employee health assessment recommendations to the attendance management platform and push them to employees' mobile devices; S311. The attendance management platform exports employee attendance status records and health advice assessments to assist managers in decision-making. S312, End.

7. The intelligent attendance management method based on multimodal fusion according to claim 6, characterized in that, In step S304, the algorithm model performs state assessment based on a preset facial expression fluctuation assessment threshold. The algorithm model specifically includes: Maintain a sequence of employee facial expression recognition results within a time window; Statistically analyze the frequency or persistence of a specified negative facial expression category in the sequence; When the frequency or duration exceeds a preset threshold, an early warning is triggered.

Images