Multifunctional intelligent entrance guard admission system based on facial information fusion

Abstract

The invention discloses a multifunctional intelligent entrance guard admission system based on facial information fusion, and belongs to the technical field of human resources. The system comprises an information storage module, an information acquisition module, an information processing module and an execution module, the information storage module is used for storing information of the information acquisition module and the information processing module; the information acquisition module is used for acquiring information of workers; the information processing module is used for analyzing and processing the information of the information storage module and the information acquisition module; the information processing module comprises identity recognition, anti-fraud detection, fatigue analysis and entry permission judgment of a person to be entered; the execution module is used for executing the information of the information processing module; according to the invention, through anti-fraud detection, illegal persons are prevented from illegally using identities of others to enter a protection area, through the analyzed fatigue degree, high-intensity fatigue operation of persons waiting to enter is avoided, and the accident occurrence risk is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of human resources, and in particular to a multifunctional intelligent access control system based on facial information fusion. Background Art

[0002] With the development of society and the process of urbanization, the demand for security management of buildings and specific production areas is increasing. At present, the traditional access control system has two main defects: first, there is a lack of preventive measures against face fraud and identity disguise; second, the traditional access control system has a single function, and the access detection method only analyzes the visual features in the face video, lacks the analysis of the physiological features hidden in the video to realize the recognition and detection of the physiological state of the access personnel; Since workers in protected areas often undertake high-risk and high-pressure tasks, illegal personnel may enter protected areas by impersonating others, or workers may be slow to react or make mistakes in judgment when working in a fatigued state, which increases the probability of production accidents and brings serious safety hazards and harms. Therefore, people urgently need a multifunctional intelligent access control system based on facial information fusion to solve the above problems. Summary of the invention

[0003] The purpose of the present invention is to provide a multifunctional intelligent access control system based on facial information fusion to solve the problems raised in the above-mentioned background technology.

[0004] In order to solve the above technical problems, the present invention provides the following technical solutions: A multifunctional intelligent access control system based on facial information fusion, the system includes an information storage module, an information collection module, an information processing module and an execution module; The information storage module is used to store information of the information collection module and the information processing module; the information collection module is used to collect information of staff members; The information processing module is used to analyze and process the information of the information storage module and the information collection module; the information processing module includes identity recognition of the person to be entered, anti-fraud detection, fatigue analysis and entry permission judgment; The execution module is used to execute the information of the information processing module.

[0005] According to the above technical solution, the information storage module includes a historical information unit and a real-time information unit; The historical information unit is used to store historical data information; the real-time information unit is used to store real-time data information.

[0006] According to the above technical solution, the information collection module includes an identity information unit and a face collection unit; The personnel information unit is used to collect the identity information of the staff; the face collection unit is used to collect the face information of the staff.

[0007] According to the above technical solution, the execution module includes a display unit and an entry execution unit; The display unit is used to display personnel information and entry information; the entry execution unit is used to execute control instructions when the staff enters the protected area.

[0008] According to the above technical solution, the information processing module includes: S1. Number all staff members in the protected area, collect and input facial data of each staff member; set standard sleep duration, control the sleep duration of staff members, and analyze the fatigue of staff members according to the length of sleep duration of staff members; S2, collecting facial RGB videos of each staff member at different fatigue levels, using the FaceMesh model, identifying the facial key points of the staff member in the facial RGB video and dividing the ROI area, wherein the ROI area includes the eye ROI area, the mouth ROI area, the cheek ROI area and the contour ROI area; S3. Analyze the eye aspect ratio of the staff member according to the eye ROI area; analyze the mouth aspect ratio of the staff member according to the mouth ROI area; extract the rPPG signal from the cheek ROI area, and analyze the average level and concentration of the heart rate of the staff member according to the extracted rPPG signal; S4. Establish a neural network model, take the eye aspect ratio, mouth aspect ratio, and average level and concentration of the heart rate of the same staff member at different fatigue levels as input variables, take the fatigue level of the staff member as the output variable, and train the neural network model; S5. Collect the current facial data of the person to be entered, compare it with the recorded facial data, and perform identity recognition; after the identity recognition is passed, collect the facial RGB video of the person to be entered, divide the ROI area, and perform anti-fraud detection based on the cheek ROI area and the contour ROI area; after the anti-fraud detection is passed, analyze the fatigue of the person to be entered based on the eye ROI area, mouth ROI area and cheek ROI area, combined with the neural network model, to determine whether the person is allowed to enter the protected area.

[0009] According to the above technical solution, the specific process of S1 is as follows: S1-1. Number all staff members in the protected area, collect and input facial data of each staff member; When recording facial data using a camera, try to ensure that the person being recorded maintains a natural posture and expression, avoid tilting the head excessively and blocking important facial features, and ensure that data is collected under good lighting conditions to avoid strong or dark light environments that affect the recording effect; S1-2, set the standard sleep duration, recorded as T, control the sleep duration of the staff, and record the actual sleep duration of the staff as T 实 ; When T 实 >T, it means the staff is not tired; When T 实 When ≤T, the fatigue degree of the staff is calculated according to the following formula; ; in, Represents the fatigue level of the staff, m∈[0,1].

[0010] According to the above technical solution, the specific process of S2 is as follows: S2-1. Collect the facial RGB videos of each staff member at different fatigue levels, and record the facial RGB video of staff member n at fatigue level m as P n，m , n∈[1, N1], N1 is a positive integer; S2-2. Using the FaceMesh model, identify P n，m The facial key points of worker n when the fatigue level is m are divided into ROI areas, and the eye ROI area of ​​worker n when the fatigue level is m is recorded as R1 n，m ; The mouth ROI area of ​​worker n when fatigue level is m is recorded as R2 n，m ; The cheek ROI area of ​​worker n when fatigue level is m is recorded as R3 n，m ; The contour ROI area of ​​worker n when fatigue level is m is recorded as R4 n，m ; FaceMesh is a facial calibration technology developed by Google that can track faces and generate 478 facial key points in real-time video using deep learning and computer vision technology.

[0011] According to the above technical solution, the specific process of S3 is as follows: S3-1, according to R1 n，m , the coordinate of the leftmost end point of the eyelid of worker n when the fatigue level is m is recorded as ; The coordinate of the rightmost end point of the eyelid of worker n when the fatigue level is m is recorded as ; The coordinate of the upper end point of the upper eyelid of worker n when the fatigue level is m is recorded as ; The coordinate of the lowest end point of the lower eyelid of worker n when the fatigue level is m is recorded as ; Calculate the worker's eye aspect ratio using the following formula: ; in, represents the eye aspect ratio of worker n when fatigue level is m; It reflects the frequency and duration of blinking and is a strong indicator for detecting fatigue; S3-2, according to R2 n，m , the coordinate of the leftmost end point of the lip of worker n when the fatigue level is m is recorded as ; The coordinate of the rightmost end point of the lip of worker n when the fatigue level is m is recorded as ; The coordinate of the uppermost end point of the upper lip of worker n when the fatigue level is m is recorded as ; The coordinate of the lowest end point of the lower lip of worker n when the fatigue level is m is recorded as ; Calculate the worker's mouth aspect ratio using the following formula: ; in, represents the aspect ratio of the mouth of worker n when the fatigue level is m; S3-3. Extract R3 based on singular value decomposition and local invariance LGI algorithm n，m According to the extracted rPPG signal, the RR interval signal and the number of RR interval signals of the heart rate variability characteristics of worker n when the fatigue level is m are extracted in the time domain, and the number of RR interval signals of the heart rate variability characteristics of worker n when the fatigue level is m is recorded as N2; the i-th RR interval signal of the heart rate variability characteristics of worker n when the fatigue level is m is recorded as RR n，m，i , i∈[1,N2]; Calculate the average heart rate of workers according to the following formula: ; in, It represents the average heart rate of worker n when fatigue level is m; The concentration of the worker's heart rate is calculated according to the following formula: ; in, It represents the concentration of heart rate of worker n when fatigue level is m; Remote Photoplethysmography (rPPG) is a non-contact optical detection technology that can capture tiny changes in the color of a person's face through an ordinary camera to infer the heart's pulse; The LGI (Local Group Invariance) algorithm is used for extraction. On the one hand, compared with the traditional method of separating the green channel, this algorithm has a certain degree of robustness against signal interference; on the other hand, compared with algorithms such as POS (Plant Orthogonal Skin) and deep learning, it has higher computational efficiency and lower latency, and is more applicable in the application scenarios of the access system.

[0012] According to the above technical solution, the specific process of S4 is as follows: Establish a neural network model to analyze the worker n at different fatigue levels , , and As the input variable; the fatigue level of worker n is used as the output variable to train the neural network model.

[0013] According to the above technical solution, the specific process of S5 is as follows: S5-1. The current time is recorded as time t; the number of the person waiting to enter at time t is recorded as n, and the facial data of the person waiting to enter at time t is collected. The feature vector of the facial data of the person waiting to enter at time t and the feature vector of the facial data of the person waiting to enter at time t and the recorded facial data of the person waiting to enter at time n are obtained through the FaceNet model. The jth feature vector of the facial data of the person waiting to enter at time t is recorded as A j ; The jth feature vector of the facial data of the person to be entered, n, is recorded as B j ; j∈[1, N3], N3 is a positive integer; The similarity is calculated according to the following formula: ; in, Indicates the similarity between the incoming person n and the recorded facial data at time t; like , then the identity recognition fails. It is a fixed parameter preset by the system; like The identity recognition is passed; S5-2. After the identity recognition is passed, the facial RGB video of the person n waiting to enter at time t is collected, the ROI area is divided, the rPPG signal of the cheek ROI area is extracted, and the facial texture information of the contour ROI area is extracted. The extracted facial texture information and rPPG signal are used as input data for anti-fraud detection; Gram-Net is used to extract abstract features from facial texture, and the abstract features are concatenated with rPPG signals to form a feature vector. The feature vector is fused through the Transformer’s attention mechanism; The feature vector after feature fusion is passed through the fully connected layer, ReLU activation function and Dropout layer in sequence, and then through the fully connected layer of binary classification to output the result of anti-fraud detection, which includes anti-fraud detection passing and anti-fraud detection failing; S5-3. After the anti-fraud detection is passed, the eye aspect ratio, mouth aspect ratio, and average level and concentration of the heart rate of the person n waiting to enter at time t are obtained based on the eye ROI area, mouth ROI area, and cheek ROI area. Combined with the neural network model, the fatigue level of the person n waiting to enter at time t is obtained, recorded as m t，n ; like , the person is not allowed to enter the protected area. It is a fixed parameter preset by the system; like , the person is allowed to enter the protected area.

[0014] Compared with the prior art, the beneficial effects achieved by the present invention are: The present invention uses computer vision and artificial intelligence technology, utilizes rPPG technology to collect surface pulse wave signals as physiological features, and integrates visual features in facial videos as input data for the system. On the one hand, anti-fraud detection is performed on the face of the person to be entered to ensure that the identity of the person to be entered is authentic and valid, and prevent illegal persons from impersonating others to enter the protection area. On the other hand, for the sake of safe production and employee health, the fatigue level of the person to be entered is analyzed to avoid high-intensity fatigue work of the person to be entered and reduce the risk of accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 It is a structural schematic diagram of the multifunctional intelligent access control system based on facial information fusion of the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] See also Figure 1 , a multifunctional intelligent access control system based on facial information fusion, the system includes an information storage module, an information collection module, an information processing module and an execution module; The information storage module is used to store information of the information collection module and the information processing module; the information collection module is used to collect information of staff members; The information processing module is used to analyze and process the information of the information storage module and the information collection module; the information processing module includes identity recognition of the person to be entered, anti-fraud detection, fatigue analysis and entry permission judgment; The execution module is used to execute the information of the information processing module.

[0018] The information storage module includes a historical information unit and a real-time information unit; The historical information unit is used to store historical data information; the real-time information unit is used to store real-time data information.

[0019] The information collection module includes an identity information unit and a face collection unit; The personnel information unit is used to collect the identity information of the staff; the face collection unit is used to collect the face information of the staff.

[0020] The execution module includes a display unit and an entry execution unit; The display unit is used to display personnel information and entry information; the entry execution unit is used to execute control instructions when the staff enters the protected area.

[0021] According to the above technical solution, the information processing module includes: S1. Number all staff members in the protected area, collect and input facial data of each staff member; set standard sleep duration, control the sleep duration of staff members, and analyze the fatigue of staff members according to the length of sleep duration of staff members; The specific process of S1 is as follows: S1-1. Number all staff members in the protected area, collect and input facial data of each staff member; When recording facial data using a camera, try to ensure that the person being recorded maintains a natural posture and expression, avoid tilting the head excessively and blocking important facial features, and ensure that data is collected under good lighting conditions to avoid strong or dark light environments that affect the recording effect; S1-2, set the standard sleep duration, recorded as T, control the sleep duration of the staff, and record the actual sleep duration of the staff as T实 ; When T 实 >T, it means the staff is not tired; When T 实 When ≤T, the fatigue degree of the staff is calculated according to the following formula; ; in, represents the fatigue degree of the staff, m∈[0,1]; Analyze the fatigue level of the staff according to their sleep duration, control the sleep duration to change the fatigue level, and prepare for collecting facial RGB videos of each staff member at different fatigue levels.

[0022] S2, collecting facial RGB videos of each staff member at different fatigue levels, using the FaceMesh model, identifying the facial key points of the staff member in the facial RGB video and dividing the ROI area, wherein the ROI area includes the eye ROI area, the mouth ROI area, the cheek ROI area and the contour ROI area; The specific process of S2 is as follows: S2-1. Collect the facial RGB videos of each staff member at different fatigue levels, and record the facial RGB video of staff member n at fatigue level m as P n，m , n∈[1, N1], N1 is a positive integer; S2-2. Using the FaceMesh model, identify P n，m The facial key points of worker n when the fatigue level is m are divided into ROI areas, and the eye ROI area of ​​worker n when the fatigue level is m is recorded as R1 n，m ; The mouth ROI area of ​​worker n when fatigue level is m is recorded as R2 n，m ; The cheek ROI area of ​​worker n when fatigue level is m is recorded as R3 n，m ; The contour ROI area of ​​worker n when fatigue level is m is recorded as R4 n，m ; FaceMesh is a facial calibration technology developed by Google that can track faces and generate 478 facial key points in real-time video using deep learning and computer vision technology.

[0023] S3. Analyze the eye aspect ratio of the staff member according to the eye ROI area; analyze the mouth aspect ratio of the staff member according to the mouth ROI area; extract the rPPG signal from the cheek ROI area, and analyze the average level and concentration of the heart rate of the staff member according to the extracted rPPG signal; The specific process of S3 is as follows: S3-1, according to R1n，m , the coordinate of the leftmost end point of the eyelid of worker n when the fatigue level is m is recorded as ; The coordinate of the rightmost end point of the eyelid of worker n when the fatigue level is m is recorded as ; The coordinate of the upper end point of the upper eyelid of worker n when the fatigue level is m is recorded as ; The coordinate of the lowest end point of the lower eyelid of worker n when the fatigue level is m is recorded as ; Calculate the worker's eye aspect ratio using the following formula: ; in, represents the eye aspect ratio of worker n when fatigue level is m; It reflects the frequency and duration of blinking and is a strong indicator for detecting fatigue; S3-2, according to R2 n，m , the coordinate of the leftmost end point of the lip of worker n when the fatigue level is m is recorded as ; The coordinate of the rightmost end point of the lip of worker n when the fatigue level is m is recorded as ; The coordinate of the uppermost end point of the upper lip of worker n when the fatigue level is m is recorded as ; The coordinate of the lowest end point of the lower lip of worker n when the fatigue level is m is recorded as ; Calculate the worker's mouth aspect ratio using the following formula: ; in, represents the aspect ratio of the mouth of worker n when the fatigue level is m; S3-3. Extract R3 based on singular value decomposition and local invariance LGI algorithm n，m According to the extracted rPPG signal, the RR interval signal and the number of RR interval signals of the heart rate variability characteristics of worker n when the fatigue level is m are extracted in the time domain, and the number of RR interval signals of the heart rate variability characteristics of worker n when the fatigue level is m is recorded as N2; the i-th RR interval signal of the heart rate variability characteristics of worker n when the fatigue level is m is recorded as RR n，m，i , i∈[1,N2]; Calculate the average heart rate of workers according to the following formula: ; in, It represents the average heart rate of worker n when fatigue level is m; The concentration of the worker's heart rate is calculated according to the following formula: ; in, It represents the concentration of heart rate of worker n when fatigue level is m; Remote Photoplethysmography (rPPG) is a non-contact optical detection technology that can capture tiny changes in the color of a person's face through an ordinary camera to infer the heart's pulse; The LGI (Local Group Invariance) algorithm is used for extraction. On the one hand, compared with the traditional method of separating the green channel, this algorithm has a certain degree of robustness against signal interference; on the other hand, compared with algorithms such as POS (Plant Orthogonal Skin) and deep learning, it has higher computational efficiency and lower latency, and is more applicable in the application scenarios of the access system.

[0024] S4. Establish a neural network model, take the eye aspect ratio, mouth aspect ratio, and average level and concentration of the heart rate of the same staff member at different fatigue levels as input variables, take the fatigue level of the staff member as the output variable, and train the neural network model; The specific process of S4 is as follows: Establish a neural network model to analyze the worker n at different fatigue levels , , and As the input variable; the fatigue level of worker n is used as the output variable to train the neural network model; Through the trained neural network model, the fatigue analysis of incoming personnel can be made faster and more accurate.

[0025] S5. Collect the current facial data of the person to be entered, compare it with the recorded facial data, and perform identity recognition; after the identity recognition is passed, collect the facial RGB video of the person to be entered, divide the ROI area, and perform anti-fraud detection based on the cheek ROI area and the contour ROI area; after the anti-fraud detection is passed, analyze the fatigue of the person to be entered based on the eye ROI area, mouth ROI area and cheek ROI area, combined with the neural network model, and determine whether to allow the person to enter the protected area; The specific process of S5 is as follows: S5-1. The current time is recorded as time t. The number of the person waiting to enter at time t is recorded as n. The facial data of the person waiting to enter at time t is collected. Through the FaceNet model, the FaceNet model outputs a 128-dimensional feature vector. The feature vector of the facial data of the person waiting to enter at time t and the feature vector of the facial data of the person waiting to enter at time t are obtained. The jth feature vector of the facial data of the person waiting to enter at time t is recorded as A. j ; The jth feature vector of the facial data of the person to be entered, n, is recorded as B j ; j∈[1, N3], N3 is a positive integer; The similarity is calculated according to the following formula: ; in, Indicates the similarity between the incoming person n and the recorded facial data at time t; As the angle between two vectors approaches 0, The closer it is to 1, the closer the two vectors are to each other and the higher the similarity. like , then the identity recognition fails. It is a fixed parameter preset by the system; like The identity recognition is passed; S5-2. After the identity recognition is passed, the facial RGB video of the person n waiting to enter at time t is collected, the ROI area is divided, the rPPG signal of the cheek ROI area is extracted, and the facial texture information of the contour ROI area is extracted. The extracted facial texture information and rPPG signal are used as input data for anti-fraud detection; Gram-Net is used to extract abstract features from facial texture, and the abstract features are concatenated with rPPG signals to form a feature vector. The feature vector is fused through the Transformer’s attention mechanism; The feature vector after feature fusion is passed through the fully connected layer, ReLU activation function and Dropout layer in sequence, and then through the fully connected layer of binary classification to output the result of anti-fraud detection, which includes anti-fraud detection passing and anti-fraud detection failing; S5-3. After the anti-fraud detection is passed, the eye aspect ratio, mouth aspect ratio, and average level and concentration of the heart rate of the person n waiting to enter at time t are obtained based on the eye ROI area, mouth ROI area, and cheek ROI area. Combined with the neural network model, the fatigue level of the person n waiting to enter at time t is obtained, recorded as m t，n ; like , the person is not allowed to enter the protected area. It is a fixed parameter preset by the system; like , the person is allowed to enter the protected area. Example

[0026] The staff member numbered n needs to enter the protected area at time t and first undergo identity recognition; Calculate: ; because Identity verification passed; After the identity recognition is passed, the facial RGB video of the person n waiting to enter at time t is collected, the ROI area is divided, the rPPG signal of the cheek ROI area is extracted, and the facial texture information of the contour ROI area is extracted. The extracted facial texture information and rPPG signal are used as input data for anti-fraud detection; Gram-Net is used to extract abstract features from facial texture, and the abstract features are concatenated with rPPG signals to form a feature vector. The feature vector is fused through the Transformer’s attention mechanism; The feature vector after feature fusion passes through the fully connected layer, ReLU activation function and Dropout layer in sequence, and then passes through the fully connected layer of 2 classification, and the anti-fraud detection result is output as anti-fraud detection passed; After the anti-fraud detection is passed, the eye aspect ratio, mouth aspect ratio, and average level and concentration of the heart rate of person n waiting to enter at time t are obtained based on the eye ROI area, mouth ROI area, and cheek ROI area. Combined with the neural network model, the fatigue level of person n waiting to enter at time t is obtained. ; because , the person is not allowed to enter the protected area.

[0027] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0028] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multifunctional intelligent access control system based on facial information fusion, characterized by: The system includes an information storage module, an information collection module, an information processing module and an execution module; The information storage module is used to store information of the information acquisition module and the information processing module; The information collection module is used to collect information of staff members; The information processing module is used to analyze and process the information of the information storage module and the information collection module; the information processing module includes identity recognition of the person to be entered, anti-fraud detection, fatigue analysis and entry permission judgment; The execution module is used to execute the information of the information processing module.

2. The multifunctional intelligent access control system based on facial information fusion according to claim 1 is characterized in that: The information storage module includes a historical information unit and a real-time information unit; The historical information unit is used to store historical data information; the real-time information unit is used to store real-time data information.

3. The multifunctional intelligent access control system based on facial information fusion according to claim 2 is characterized in that: The information collection module includes an identity information unit and a face collection unit; The personnel information unit is used to collect the identity information of the staff; the face collection unit is used to collect the face information of the staff.

4. The multifunctional intelligent access control system based on facial information fusion according to claim 3 is characterized in that: The execution module includes a display unit and an entry execution unit; The display unit is used to display personnel information and entry information; the entry execution unit is used to execute control instructions when the staff enters the protected area.

5. The multifunctional intelligent access control system based on facial information fusion according to claim 4 is characterized in that: The information processing module includes: S1. Number all staff members in the protected area, collect and input facial data of each staff member; set standard sleep duration, control the sleep duration of staff members, and analyze the fatigue of staff members according to the length of sleep duration of staff members; S2, collecting facial RGB videos of each staff member at different fatigue levels, using the FaceMesh model, identifying the facial key points of the staff member in the facial RGB video and dividing the ROI area, wherein the ROI area includes the eye ROI area, the mouth ROI area, the cheek ROI area and the contour ROI area; S3. Analyze the eye aspect ratio of the staff member according to the eye ROI area; analyze the mouth aspect ratio of the staff member according to the mouth ROI area; extract the rPPG signal from the cheek ROI area, and analyze the average level and concentration of the heart rate of the staff member according to the extracted rPPG signal; S4. Establish a neural network model, take the eye aspect ratio, mouth aspect ratio, and average level and concentration of the heart rate of the same staff member at different fatigue levels as input variables, take the fatigue level of the staff member as the output variable, and train the neural network model; S5. Collect the current facial data of the person to be entered, compare it with the recorded facial data, and perform identity recognition; after the identity recognition is passed, collect the facial RGB video of the person to be entered, divide the ROI area, and perform anti-fraud detection based on the cheek ROI area and the contour ROI area; after the anti-fraud detection is passed, analyze the fatigue of the person to be entered based on the eye ROI area, mouth ROI area and cheek ROI area, combined with the neural network model, to determine whether the person is allowed to enter the protected area.

6. The multifunctional intelligent access control system based on facial information fusion according to claim 5 is characterized in that: The specific process of S1 is as follows: S1-1. Number all staff members in the protected area, collect and input facial data of each staff member; S1-2, set the standard sleep duration, recorded as T, control the sleep duration of the staff, and record the actual sleep duration of the staff as T 实 ; When T 实 >T, it means the staff is not tired; When T 实 When ≤T, the fatigue degree of the staff is calculated according to the following formula; ； in, Represents the fatigue level of the staff, m∈[0,1].

7. The multifunctional intelligent access control system based on facial information fusion according to claim 6 is characterized in that: The specific process of S2 is as follows: S2-1. Collect the facial RGB videos of each staff member at different fatigue levels, and record the facial RGB video of staff member n at fatigue level m as P n，m , n∈[1, N1], N1 is a positive integer; S2-2. Using the FaceMesh model, identify P n，m The facial key points of worker n when the fatigue level is m are divided into ROI areas, and the eye ROI area of ​​worker n when the fatigue level is m is recorded as R1 n，m ; The mouth ROI area of ​​worker n when fatigue level is m is recorded as R2 n，m ; The cheek ROI area of ​​worker n when fatigue level is m is recorded as R3 n，m ; The contour ROI area of ​​worker n when fatigue level is m is recorded as R4 n，m .

8. The multifunctional intelligent access control system based on facial information fusion according to claim 7 is characterized in that: The specific process of S3 is as follows: S3-1, according to R1 n，m , the coordinate of the leftmost end point of the eyelid of worker n when the fatigue level is m is recorded as ; The coordinate of the rightmost end point of the eyelid of worker n when the fatigue level is m is recorded as ; The coordinate of the upper end point of the upper eyelid of worker n when the fatigue level is m is recorded as ; The coordinates of the lowest end point of the lower eyelid of worker n when the fatigue level is m are recorded as ; Calculate the worker's eye aspect ratio using the following formula: ； in, represents the eye aspect ratio of worker n when fatigue level is m; S3-2, according to R2 n，m , the coordinate of the leftmost end point of the lip of worker n when the fatigue level is m is recorded as ; The coordinate of the rightmost end point of the lip of worker n when the fatigue level is m is recorded as ; The coordinate of the uppermost end point of the upper lip of worker n when the fatigue level is m is recorded as ; The coordinate of the lowest end point of the lower lip of worker n when the fatigue level is m is recorded as ; Calculate the worker's mouth aspect ratio using the following formula: ； in, represents the aspect ratio of the mouth of worker n when the fatigue level is m; S3-3. Extract R3 based on singular value decomposition and local invariance LGI algorithm n，m According to the extracted rPPG signal, the RR interval signal and the number of RR interval signals of the heart rate variability characteristics of worker n when the fatigue level is m are extracted in the time domain, and the number of RR interval signals of the heart rate variability characteristics of worker n when the fatigue level is m is recorded as N2; the i-th RR interval signal of the heart rate variability characteristics of worker n when the fatigue level is m is recorded as RR n，m，i , i∈[1,N2]; Calculate the average heart rate of workers according to the following formula: ； in, It represents the average heart rate of worker n when fatigue level is m; The concentration of the worker's heart rate is calculated according to the following formula: ； in, It indicates the concentration of the heart rate of worker n when his fatigue level is m.

9. The multifunctional intelligent access control system based on facial information fusion according to claim 8 is characterized in that: The specific process of S4 is as follows: Establish a neural network model to analyze the worker n at different fatigue levels , , and as input variables; The fatigue level of worker n is used as the output variable to train the neural network model.

10. The multifunctional intelligent access control system based on facial information fusion according to claim 9 is characterized in that: The specific process of S5 is as follows: S5-1. The current time is recorded as time t; the number of the person waiting to enter at time t is recorded as n, and the facial data of the person waiting to enter at time t is collected. The feature vector of the facial data of the person waiting to enter at time t and the feature vector of the facial data of the person waiting to enter at time t and the recorded facial data of the person waiting to enter at time n are obtained through the FaceNet model. The jth feature vector of the facial data of the person waiting to enter at time t is recorded as A j ; The jth feature vector of the facial data of the person to be entered, n, is recorded as B j ; j∈[1, N3], N3 is a positive integer; The similarity is calculated according to the following formula: ； in, Indicates the similarity between the incoming person n and the recorded facial data at time t; like , then the identity recognition fails. It is a fixed parameter preset by the system; like The identity recognition is passed; S5-2. After the identity recognition is passed, the facial RGB video of the person n waiting to enter at time t is collected, the ROI area is divided, the rPPG signal of the cheek ROI area is extracted, and the facial texture information of the contour ROI area is extracted. The extracted facial texture information and rPPG signal are used as input data for anti-fraud detection; Gram-Net is used to extract abstract features from facial texture, and the abstract features are concatenated with rPPG signals to form a feature vector. The feature vector is fused through the Transformer’s attention mechanism; The feature vector after feature fusion is passed through the fully connected layer, ReLU activation function and Dropout layer in sequence, and then through the fully connected layer of binary classification to output the result of anti-fraud detection, which includes anti-fraud detection passing and anti-fraud detection failing; S5-3. After the anti-fraud detection is passed, the eye aspect ratio, mouth aspect ratio, and average level and concentration of the heart rate of the person n waiting to enter at time t are obtained based on the eye ROI area, mouth ROI area, and cheek ROI area. Combined with the neural network model, the fatigue level of the person n waiting to enter at time t is obtained, recorded as m t，n ; like , then the person is not allowed to enter the protected area, It is a fixed parameter preset by the system; like , the person is allowed to enter the protected area.

Images