Access control system based on face recognition and recognition method

Abstract

The invention provides an access control system based on face recognition and a recognition method, and the system comprises a multi-modal image collection subsystem which integrates a visible light camera, a depth camera and an infrared thermal imaging camera and is used for synchronously collecting a visible light image, a depth image and an infrared thermal imaging image of a face; the image preprocessing unit is used for carrying out corresponding preprocessing on the image; the feature extraction network module is used for extracting image features; the multi-modal feature fusion and alignment module is used for calculating a mutual information matrix by adopting a feature fusion strategy based on mutual information maximization and adaptive weighting, determining an adaptive weight and performing weighted summation, and performing feature alignment by applying a method based on canonical correlation analysis; and the face recognition classification and verification module verifies the image by adopting a multi-level fusion strategy. According to the method, the false recognition rate and the rejection rate are effectively reduced, the strong anti-cheating capability is achieved, and cheating attacks of various fake face means can be accurately recognized and resisted.

Description

Technical Field

[0001] The present invention relates to the technical field of face recognition, and in particular to an access control system and a recognition method based on face recognition. Background Art

[0002] Facial recognition access control systems play a key role in the security system of modern society and are widely used in various places, such as corporate offices, residential areas, schools, financial institutions, etc. Its convenience and efficiency have significantly improved the efficiency of personnel access management. However, facial recognition technology also faces various challenges in real-world use, which affect its reliability and security.

[0003] First, face recognition methods based on visible light images are extremely susceptible to interference from fluctuations in lighting conditions in practical applications. For example, in a scene with direct strong light, the shadow area of ​​the face may obscure key facial features, causing deviations in feature extraction; in a low-light environment, image noise increases, detail information is lost, and recognition accuracy is significantly reduced. Facial occlusion is also a major problem. Whether it is wearing common items such as glasses, masks, hats, or hair blocking part of the face, some features may not be obtained, thus affecting the recognition results. In addition, posture changes such as side faces, lowering heads, and raising heads will change the visual appearance of the face, making it difficult for models trained based on a single perspective to accurately recognize.

[0004] Although face recognition technology that relies solely on depth information can supplement the three-dimensional structural information of the face to a certain extent, its ability to distinguish individuals with similar appearances is limited. Since depth information mainly reflects the three-dimensional contours of the face and lacks rich visual features such as texture and color, it is difficult to capture the subtle differences in appearance between individuals, which is prone to misjudgment.

[0005] Secondly, with the development of science and technology, the means of forging faces are becoming increasingly complex and diverse, and deception tools such as high-definition photos, videos, and 3D masks have appeared frequently, posing a huge threat to the security of traditional face recognition systems. These forgeries can simulate some of the appearance features of a human face, making it difficult for traditional face recognition systems to distinguish between the real and the fake, thus leading to security loopholes and providing opportunities for criminals to take advantage of.

[0006] Therefore, there is an urgent need for an innovative and highly reliable face recognition access control system and method to effectively resist video attacks and ensure the security of the access control system. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide an access control system and recognition method based on face recognition, which can maintain high-precision face recognition accuracy under various complex environmental conditions, effectively reduce the false recognition rate and rejection rate. It has strong anti-spoofing ability, can accurately identify and resist various fake face attacks such as high-definition photos, videos, 3D masks, etc., to ensure the security and reliability of the face recognition system.

[0008] To solve the above technical problems, an embodiment of the present invention provides the following technical solution: an access control system based on face recognition, comprising: The multimodal image acquisition subsystem integrates a visible light camera, a depth camera, and an infrared thermal imaging camera to synchronously acquire visible light images, depth images, and infrared thermal imaging images of a person's face. An image preprocessing unit, which performs dynamic range adjustment, geometric normalization based on facial key points, and denoising based on guided filtering on the visible light image; performs three-dimensional Gaussian filtering smoothing and normalization on the depth image; and performs temperature drift correction and wavelet transform-based enhancement on the infrared thermal imaging image; The feature extraction network module includes convolutional neural networks for extracting visible light image features, deep image features and infrared thermal imaging image features respectively. The visible light image feature extraction network adopts a multi-branch feature extraction architecture, including a parallel convolution layer, a residual attention module, a multi-scale feature fusion layer, a global average pooling layer and a fully connected layer; the deep image feature extraction network is optimized based on a lightweight MobileNetV3 architecture, and is provided with a multi-scale deep feature extraction branch and an SE module; the infrared thermal imaging image feature extraction network adopts a Vision Transformer-based architecture, including a ViT encoder, a global average pooling layer and a fully connected layer; The multimodal feature fusion and alignment module adopts a feature fusion strategy based on mutual information maximization and adaptive weighting, calculates the mutual information matrix to determine the adaptive weights and weighted sums them, and uses a method based on canonical correlation analysis to perform feature alignment; The face recognition classification and verification module adopts a multi-level fusion strategy, including multiple deep learning classification networks with different depths and structures and multiple support vector machines with different kernel functions. The face recognition results are determined through a hierarchical fusion voting mechanism, and a confidence assessment module is provided to calculate the confidence score based on the results and historical accuracy. If the confidence score is above the threshold, the verification is passed, and if it is below the threshold, a secondary verification mechanism is triggered.

[0009] Furthermore, the visible light camera lens is equipped with optical image stabilization technology and autofocus function, intelligent exposure compensation and color correction function, and the frame rate of the camera can be adjusted between 1-120 frames per second; The depth camera has the function of integrating time-of-flight ToF technology and structured light technology; The infrared thermal imaging camera uses a highly sensitive uncooled focal plane detector to sense infrared rays in the 8-14 micron band.

[0010] The present invention also proposes a recognition method based on the above-mentioned access control system based on face recognition, comprising the following steps: Visible light cameras, depth cameras and infrared thermal imaging cameras are used to synchronously collect visible light images, depth images and infrared thermal imaging images of human faces; The visible light image is subjected to dynamic range adjustment, geometric normalization based on facial key points, and denoising based on guided filtering; the depth image is subjected to three-dimensional Gaussian filtering smoothing and normalization processing; the infrared thermal imaging image is subjected to temperature drift correction and enhancement processing based on wavelet transform; Image features are extracted through a feature extraction network, which includes a visible light image feature extraction network, a deep image feature extraction network, and an infrared thermal imaging image feature extraction network; the visible light image feature extraction network adopts a multi-branch feature extraction architecture, including a parallel convolution layer, a residual attention module, a multi-scale feature fusion layer, a global average pooling layer, and a fully connected layer; the deep image feature extraction network is optimized based on a lightweight MobileNetV3 architecture, and is provided with a multi-scale deep feature extraction branch and an SE module; the infrared thermal imaging image feature extraction network adopts a VisionTransformer-based architecture, including a ViT encoder, a global average pooling layer, and a fully connected layer; A feature fusion strategy based on mutual information maximization and adaptive weighting is adopted. The mutual information matrix is ​​calculated to determine the adaptive weights and weighted summation is performed, and a method based on canonical correlation analysis is used for feature alignment. A multi-level fusion strategy is adopted, which includes multiple deep learning classification networks with different depths and structures and multiple support vector machines with different kernel functions. The face recognition results are determined through a hierarchical fusion voting mechanism, and a confidence assessment is performed. The confidence score is calculated based on the results and historical accuracy. If it is above the threshold, the verification is passed, and if it is below the threshold, the secondary verification mechanism is triggered.

[0011] Furthermore, the visible light image dynamic range adjustment adopts a local contrast enhancement algorithm, specifically:

[0012] in is the output image pixel value, is the input image pixel value, and are the mean and standard deviation of the local area of ​​the input image, is the mean value of the target area, and k is the contrast enhancement coefficient.

[0013] Furthermore, the denoising algorithm formula of the visible light image based on guided filtering is: ,in is the value of the pixel after filtering, is the value of the input image pixel, and It is the coefficient calculated by minimizing the guided filter cost function.

[0014] Furthermore, the three-dimensional Gaussian filtering formula of the depth image is: ,in is the pixel depth value after filtering at point P, is the original pixel depth value of point q, S is the neighborhood of pixel p, , and are Gaussian kernel functions in the x, y and z directions, respectively, used for weighting in the x, y and z directions. is the normalization coefficient. Indicates the distance weight between point p and point q in the x direction; Indicates the distance weight between point p and point q in the y direction; Represents the distance weight between point p and point q in the z direction.

[0015] Furthermore, in the multimodal feature fusion and alignment module, a feature fusion strategy based on mutual information maximization and adaptive weighting is adopted, the mutual information matrix is ​​calculated to determine the adaptive weights and weighted sum, and a method based on canonical correlation analysis is used to perform feature alignment, specifically: First, the mutual information matrix M between the three modal eigenvectors is calculated. The calculation formula is:

[0016] Where X and Y represent the eigenvectors of two different modes, is their joint probability distribution, and are the respective marginal probability distributions; According to the mutual information matrix, calculate the importance score of each modal feature ,in is an element in the mutual information matrix, representing the mutual information between the i-th modal feature and the j-th modal feature; Calculate adaptive weights based on importance scores , ensure that the sum of the weights is 1; Finally, the three modal eigenvectors F1, F2, and F3 are weighted and summed according to the adaptive weights to obtain the fused eigenvector:

[0017] Before feature fusion, feature alignment operation is performed.

[0018] Furthermore, the deep learning classification network in the face recognition classification and verification module includes a network based on the ResNet architecture, the DenseNet architecture and the EfficientNet architecture, the SVM includes a linear kernel, a polynomial kernel and a radial basis function kernel, and the weight calculation formula of the hierarchical fusion voting mechanism is:

[0019] in, is the weight coefficient of historical accuracy, is the historical accuracy of the classifier, is the confidence score of the current classification.

[0020] Furthermore, before the feature fusion, a feature alignment operation is performed, specifically: the feature vectors of the three modes are projected into a common feature subspace using a method based on canonical correlation analysis (CCA), and the alignment of the features at the semantic level is achieved by finding the projection direction that maximizes the correlation between the feature vectors of the two modes.

[0021] Furthermore, the data sources of the visible light image, depth image and infrared thermal imaging image of the face cover people of different ages, genders and races; the collection scenarios include various extreme and common lighting environments, different facial postures, different facial occlusion conditions and facial images in different temperature environments; after the collection is completed, the collected data is labeled, and the labeled information includes but is not limited to facial identity information, image collection environment information, facial posture information, whether there is facial occlusion and the type of occlusion.

[0022] The beneficial effects of the above technical solution of the present invention are as follows: 1. The present invention integrates visible light, depth and infrared thermal imaging cameras. It is difficult to simulate the features of these three modalities at the same time by counterfeiting human faces. The three modal images are processed in a targeted manner. The dynamic range adjustment, geometric normalization and filtering denoising of visible light images, the three-dimensional Gaussian filtering and smoothing of depth images, and the temperature drift correction and enhancement processing of infrared thermal imaging images improve the accuracy of feature extraction.

[0023] 2. In the feature extraction network module, special feature extraction networks are designed for different modal images. The multi-branch feature extraction architecture of visible light images combined with the residual attention module and the multi-scale feature fusion layer can focus on key feature areas and fuse multi-scale features; the deep image feature extraction network is based on the optimized MobileNetV3 architecture, which can effectively extract deep features; the infrared thermal imaging image feature extraction network adopts the Vision Transformer architecture, which can capture temperature feature associations, and the extracted features are more discriminative, which helps to improve recognition accuracy.

[0024] 3. The multimodal feature fusion and alignment module adopts a strategy based on mutual information maximization and adaptive weighting, as well as a feature alignment method based on typical correlation analysis. It can fully integrate the effective information of the three modalities, ensure the compatibility and effectiveness of different modal features in the fusion process, and further improve the recognition accuracy.

[0025] 4. The face recognition classification and verification module adopts a multi-level fusion strategy, combining multiple deep learning classification networks of different depths and structures with support vector machines with different kernel functions. The results are determined through a hierarchical fusion voting mechanism, and the voting weights are dynamically adjusted according to the historical accuracy and current confidence scores, thereby combining the advantages of multiple classifiers, improving the accuracy and reliability of face recognition, and significantly reducing the false recognition rate and rejection rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a principle block diagram of the access control system based on face recognition of the present invention; Figure 2 The figure is a working diagram of the access control system based on face recognition of the present invention. DETAILED DESCRIPTION

[0027] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0028] like Figure 1 As shown, the present invention discloses a door access control system based on face recognition, comprising: The multimodal image acquisition subsystem 101 integrates a visible light camera, a depth camera and an infrared thermal imaging camera, and is used to synchronously acquire a visible light image, a depth image and an infrared thermal imaging image of a human face; The image preprocessing unit 102 performs dynamic range adjustment, geometric normalization based on facial key points, and denoising based on guided filtering on the visible light image; performs three-dimensional Gaussian filtering smoothing and normalization on the depth image; and performs temperature drift correction and wavelet transform-based enhancement on the infrared thermal imaging image. The feature extraction network module 103 comprises convolutional neural networks for extracting visible light image features, depth image features and infrared thermal imaging image features respectively. The visible light image feature extraction network adopts a multi-branch feature extraction architecture, including a parallel convolution layer, a residual attention module, a multi-scale feature fusion layer, a global average pooling layer and a fully connected layer; the depth image feature extraction network is optimized based on a lightweight MobileNetV3 architecture, and is provided with a multi-scale depth feature extraction branch and an SE module; the infrared thermal imaging image feature extraction network adopts a Vision Transformer-based architecture, including a ViT encoder, a global average pooling layer and a fully connected layer; The multimodal feature fusion and alignment module 104 adopts a feature fusion strategy based on mutual information maximization and adaptive weighting, calculates the mutual information matrix to determine the adaptive weights and performs weighted summation, and performs feature alignment using a method based on canonical correlation analysis; The face recognition classification and verification module 105 adopts a multi-level fusion strategy, including multiple deep learning classification networks with different depths and structures and multiple support vector machines with different kernel functions. The face recognition result is determined by a hierarchical fusion voting mechanism, and a confidence evaluation module is provided to calculate the confidence score based on the result and historical accuracy. If the confidence score is above the threshold, the verification is passed, and if it is below the threshold, a secondary verification mechanism is triggered.

[0029] The workflow of each of the above modules is as follows: After detecting that a face enters the preset recognition area or receiving a face recognition command, the multimodal image acquisition subsystem synchronously starts the visible light camera, depth camera and infrared thermal imaging camera to collect multimodal image data of the face and transmit the data to the image preprocessing unit.

[0030] The image preprocessing unit performs corresponding preprocessing operations on the images of the three modalities, including dynamic range adjustment, geometric normalization and filtering denoising of visible light images, three-dimensional Gaussian filtering and normalization of depth images, temperature drift correction and enhancement processing of infrared thermal imaging images, etc. The processed image data is transmitted to the feature extraction network module.

[0031] The three specially designed feature extraction networks in the feature extraction network module perform deep feature extraction on the corresponding images respectively, and output visible light image feature vectors, depth image feature vectors and infrared thermal imaging image feature vectors to the multimodal feature fusion and alignment module.

[0032] The multimodal feature fusion and alignment module first calculates the mutual information matrix between the three modal feature vectors, determines the adaptive weights, then performs feature alignment operations, and finally performs weighted summation of the three feature vectors according to the adaptive weights to obtain the fused feature vector, which is then transmitted to the face recognition classification and verification module.

[0033] In the face recognition classification and verification module, multiple deep learning classification networks and multiple SVMs classify the fused feature vectors respectively, and then fuse the classification results through the hierarchical fusion voting mechanism to obtain the final face recognition result. If the recognition fails, the secondary verification mechanism is triggered.

[0034] The following is a detailed description of the above workflow: The multimodal image acquisition subsystem 101 integrates a visible light camera, a depth camera and an infrared thermal imaging camera, wherein: In this embodiment, the visible light camera uses a high-resolution, high-sensitivity image sensor with a wide dynamic range, which can capture clear and delicate facial images under different lighting conditions (from 0.01 lux in extremely low light to 100,000 lux in strong light). The lens is equipped with advanced optical image stabilization technology and autofocus function to ensure image stability and clarity. The frame rate of the camera can be flexibly adjusted between 1-120 frames per second to meet the real-time and image quality requirements of different application scenarios. At the same time, it has intelligent exposure compensation and color correction functions, which can automatically adjust the exposure time and gain according to the ambient light to optimize the brightness and color reproduction of the image.

[0035] The depth camera is based on a deep fusion solution of time-of-flight (ToF) technology and structured light technology. ToF technology can quickly obtain depth information of a distant face, while structured light technology can accurately capture the detailed depth changes of a close face. Its depth measurement accuracy is as high as 0.05 mm, and its resolution can reach 1920x1080 pixels, which can clearly present the slight depth differences of facial features and the overall three-dimensional contour of the face. The field of view can be flexibly adjusted according to the actual application scenario, from a narrow 30 degrees to a wide 120 degrees, ensuring that the depth information of the face can be fully and accurately obtained at various distances and angles.

[0036] The infrared thermal imaging camera uses a highly sensitive uncooled focal plane detector that can sense infrared rays in the 8-14 micron band and is extremely sensitive to human thermal radiation. The camera has automatic temperature calibration and temperature compensation functions, and can accurately measure the temperature distribution of the face at different ambient temperatures (from -20°C to 50°C). Its temperature resolution can be as low as 0.02°C, which can clearly distinguish extremely subtle temperature differences in different parts of the face. The image frame rate is 5-60 frames per second, which can be optimized according to system requirements. The generated thermal imaging image has high contrast and high resolution, effectively highlighting the temperature characteristic distribution of the face.

[0037] The image preprocessing unit 102 includes visible light image preprocessing, depth image preprocessing, and infrared thermal imaging image preprocessing, wherein: (1) Visible light image preprocessing: First, the dynamic range is adjusted, and the local contrast enhancement algorithm is used to adaptively stretch the contrast according to the local grayscale distribution of the image. The formula is:

[0038] in is the output image pixel value, is the input image pixel value, and are the mean and standard deviation of the local area of ​​the input image, is the mean value of the target area, and k is the contrast enhancement coefficient.

[0039] Then, we perform geometric normalization based on facial key points. We use deep learning algorithms to detect facial key points and locate key parts of the face (such as the corners of the eyes, the corners of the mouth, the tip of the nose, etc.). Then, we calculate the affine transformation matrix based on these key points, rotate, scale, and translate the image, and correct the face image to a standard posture and size. For example, we adjust the line connecting the centers of the two eyes to be horizontal, and scale the face image to 300x300 pixels to ensure the consistency and accuracy of subsequent feature extraction.

[0040] Finally, image filtering and denoising are performed, using a denoising algorithm based on guided filtering. This algorithm can better retain the edge and texture information of the image while removing noise. The filtering formula is: ,in is the value of the pixel after filtering, is the value of the input image pixel, and It is the coefficient calculated by minimizing the guided filter cost function.

[0041] (2) Deep image preprocessing: A three-dimensional Gaussian filtering algorithm is implemented to smooth the depth image. This algorithm can take into account the spatial distribution characteristics of the depth information while removing noise, and effectively retain the edge information of the depth image. The filtering formula is: ,in is the pixel depth value after filtering at point P, is the original pixel depth value of point q, S is the neighborhood of pixel p, , and are Gaussian kernel functions in the x, y and z directions, respectively, used for weighting in the x, y and z directions. is the normalization coefficient. Indicates the distance weight between point p and point q in the x direction; Indicates the distance weight between point p and point q in the y direction; Represents the distance weight between point p and point q in the z direction.

[0042] Normalize the depth value and map it to a specific interval, such as 0-1, to facilitate subsequent feature extraction and calculation. The normalization formula is:

[0043] in is the normalized depth value, D is the original depth value, and are the minimum and maximum depth values ​​in the image, respectively.

[0044] (3) Infrared thermal imaging image preprocessing: The temperature drift correction algorithm is executed to take into account the ambient temperature changes and the drift characteristics of the thermal imaging sensor, and the temperature values ​​in the thermal imaging image are corrected in real time to ensure the accuracy and stability of the temperature measurement.

[0045] Thermal imaging image enhancement processing is performed by using image enhancement technology based on wavelet transform. The image is decomposed into sub-bands of different frequencies, the low-frequency sub-bands are enhanced to improve the overall contrast, the high-frequency sub-bands are sharpened to highlight the edges and detail information, and then the enhanced image is obtained through inverse wavelet transform.

[0046] The feature extraction network module 103 includes a visible light image feature extraction network, a depth image feature extraction network, and an infrared thermal imaging image feature extraction network, wherein: (1) Visible light image feature extraction network, specifically: First, an innovative multi-branch feature extraction network architecture is constructed. The network starts with multiple parallel convolutional layers, using convolution kernels of different sizes (such as 3x3, 5x5, 7x7) and different step sizes (such as 1, 2) to simultaneously extract image features of different scales and resolutions. For example, small convolution kernels and small step sizes focus on extracting detailed texture features of the image, while large convolution kernels and large step sizes focus more on the macroscopic structural features of the image.

[0047] The residual attention module was then introduced, which combines the advantages of the residual network and the attention mechanism. The residual connection can effectively solve the gradient vanishing problem caused by the increase of network depth, while the attention mechanism can automatically assign weights according to the importance of features, focusing on the key feature areas for face recognition, such as the eyes, nose, mouth and other parts, and suppressing the interference of irrelevant information such as background and noise.

[0048] Then, a multi-scale feature fusion layer is set to fuse the features extracted by different branches. The fusion method adopts a weighted summation method based on feature maps. Different weights are assigned according to the importance of feature maps. The weight calculation adopts a channel attention-based method. The formula is: in is the weight of the cth channel, is the channel attention score calculated by global average pooling and fully connected layers, and C is the number of channels.

[0049] A global average pooling layer and a fully connected layer are used at the end of the network to map the fused features to a low-dimensional feature vector, such as a 2048-dimensional feature vector, which can comprehensively and effectively characterize the visible light image features of the face.

[0050] (2) Deep image feature extraction network: The optimized design is based on a lightweight MobileNetV3 architecture. By introducing the SE (Squeeze-and-Excitation) module, the feature channels can be recalibrated, the expression ability of important feature channels can be enhanced, and the feature extraction efficiency can be improved.

[0051] Multi-scale deep feature extraction branches are set in the network structure to extract the overall contour features of the face depth image, the depth ratio features of the facial features, and the depth change features caused by facial expressions. For example, one branch uses a large convolution kernel to extract global contour features, and the other branch uses a small convolution kernel to focus on the deep feature extraction of the facial features. The features extracted by each branch are fused at the end of the network.

[0052] Finally, the fused features are converted into a 2048-dimensional deep feature vector through a fully connected layer, which matches the dimension of the feature vector of the visible light image of the face, facilitating subsequent multimodal feature fusion operations.

[0053] (3) Infrared thermal imaging image feature extraction network: A feature extraction network based on the Vision Transformer (ViT) architecture is used. The self-attention mechanism of the ViT architecture can automatically capture the correlation between features at different positions in the image, and has unique advantages in the correlation analysis of temperature features at different parts of the face in infrared thermal imaging images.

[0054] The network first divides the infrared thermal imaging image into multiple small blocks, each of which is used as an input sequence element and encoded by the ViT encoder. The encoder consists of multiple multi-head self-attention layers and feedforward neural network layers. The multi-head self-attention layer can focus on image features from multiple angles and enhance feature expression capabilities.

[0055] After being processed by the ViT encoder, the global average pooling layer and the fully connected layer are used to convert the features into a 2048-dimensional infrared thermal imaging feature vector, which can fully reflect the temperature distribution characteristics of the face and the intrinsic connection between temperature features.

[0056] The multimodal feature fusion and alignment module 104 is specifically: First, a feature fusion strategy based on mutual information maximization and adaptive weighting is designed. First, the mutual information matrix M between the three modal feature vectors is calculated. The calculation formula is:

[0057] Where X and Y represent the eigenvectors of two different modes, is their joint probability distribution, and are their respective marginal probability distributions.

[0058] According to the mutual information matrix, calculate the importance score of each modal feature ,in is an element in the mutual information matrix, representing the mutual information between the i-th modal feature and the j-th modal feature.

[0059] Then, the adaptive weights are calculated based on the importance scores , ensuring that the sum of the weights is 1.

[0060] Finally, the three modal eigenvectors F1, F2, and F3 are weighted and summed according to the adaptive weights to obtain the fused eigenvector:

[0061] Before feature fusion, feature alignment is performed. The feature vectors of the three modalities are projected into a common feature subspace using a method based on canonical correlation analysis (CCA). The CCA method achieves feature alignment at the semantic level by finding the projection direction that maximizes the correlation between the feature vectors of the two modalities, ensuring the compatibility and effectiveness of different modal features during the fusion process.

[0062] The face recognition classification and verification module 105 adopts a multi-level fusion face recognition classification strategy. First, multiple deep learning classification networks with different depths and structures are constructed, including a classification network based on the ResNet architecture, a classification network based on the DenseNet architecture, and a classification network based on the EfficientNet architecture. Each classification network is independently trained on a large amount of multimodal feature data of faces to learn different feature representations and classification modes.

[0063] At the same time, multiple support vector machines (SVMs) with different kernel functions are trained, such as linear kernel SVM, polynomial kernel SVM (polynomial degree is 2-5), and radial basis function (RBF) kernel SVM (RBF kernel parameters are optimized through grid search), to classify the fused feature vectors. SVMs with different kernel functions can classify feature vectors from different angles and capture different linear and nonlinear relationships in the feature space.

[0064] In the face recognition and classification process, the fused feature vectors are input into multiple deep learning classification networks and multiple SVMs for classification. Each classifier outputs its predicted face identity category and the corresponding confidence score.

[0065] A hierarchical fusion voting mechanism is used to fuse the results of multiple classifiers. First, at the first level, the classifier results based on the same type of architecture (such as convolutional neural network architecture) are voted and fused to obtain preliminary classification results and confidence scores. Then, at the second level, the results of the first level are voted and fused again with the classification results of SVM to obtain the final face recognition results. The voting weight is dynamically adjusted according to the historical accuracy of the classifier during training and the current confidence score. The formula is:

[0066] in is the weight coefficient of historical accuracy (can be adjusted between 0.6 - 0.8 according to experimental results), is the historical accuracy of the classifier, is the confidence score of the current classification.

[0067] Finally, based on the final face recognition result, if the recognition is successful and the confidence score is higher than the threshold (such as 0.8), the corresponding authorization operation is performed, such as opening the access control system, passing the financial transaction authentication, etc.; if the recognition fails or the confidence score is lower than the threshold, the secondary verification mechanism is triggered, such as requiring the user to enter a password, answer security questions, or perform additional face image collection and recognition.

[0068] Example 2 The specific implementation process of the present invention is as follows: 1. Calibration and debugging of multimodal image acquisition subsystem: For visible light cameras, professional optical testing equipment, such as high-precision resolution test cards, color reproduction standard plates, and light intensity meters, are used to fully calibrate them. Image acquisition tests are performed under different light intensity environments (such as 0.01 lux, 10 lux, 1000 lux, and 100,000 lux). By adjusting the parameters of the intelligent exposure compensation and color correction functions, it is ensured that the image brightness, contrast, and color reproduction can meet the feature extraction requirements under various lighting conditions. At the same time, the calibration tools for optical image stabilization and autofocus functions are used to verify their stability and accuracy, ensuring that they can focus quickly and accurately and obtain clear images at different distances and motion states.

[0069] For the depth camera, a high-precision flat calibration plate and a stereo calibration object with a known depth are used to calibrate and verify the depth measurement accuracy. The calibration object is placed at different distances (such as 0.3 meters, 0.5 meters, 1 meter, 2 meters, etc.), and by comparing with the standard depth value, the relevant parameters of the ToF technology and structured light technology are adjusted to achieve the design requirement of 0.05 mm depth measurement accuracy. The field of view angle measurement tool is used to accurately adjust the field of view angle parameters to ensure that the face area can be fully and accurately covered within the preset face recognition distance range (such as 0.3-2 meters).

[0070] The infrared thermal imaging camera is calibrated with the help of a high-precision blackbody radiation source and temperature sensor. The temperature measurement accuracy is calibrated in different temperature environments (such as -20℃, 0℃, 20℃, 50℃, etc.), the blackbody radiation source is set to different temperatures, the temperature value measured by the infrared thermal imaging camera is compared with the actual temperature of the blackbody, and the temperature resolution is adjusted to the design standard of 0.02℃ by adjusting the parameters in the temperature drift correction algorithm. At the same time, the performance of the uncooled focal plane detector is checked to ensure its stability and reliability under different ambient temperatures.

[0071] (II) Data collection and labeling: Collect large-scale, rich and diverse multimodal facial image data. The data sources cover people of different ages (infants, children, teenagers, adults, and the elderly), genders, and races (white, black, yellow, brown, etc.). The collection scenarios include various extreme and common lighting environments (such as extremely weak light indoors, strong direct light indoors, cloudy outdoors, sunny outdoors, street lights at night, etc.), different facial postures (front face, side face 30 degrees, side face 60 degrees, head up 30 degrees, head down 30 degrees, etc.), different facial occlusion conditions (wearing different styles of glasses, masks, hats, scarves, etc.) and facial images in different temperature environments (cold winter outdoors, normal temperature indoors, hot summer outdoors).

[0072] The collected data is annotated in detail, including facial identity information, image acquisition environment information (light intensity, temperature, humidity, etc.), facial posture information, whether there is facial occlusion, and the type of occlusion.

[0073] (III) Feature extraction network module training: The visible light image feature extraction network was trained using the labeled data. The stochastic gradient descent (SGD) optimization algorithm was used, combined with a momentum term (e.g., momentum = 0.9) and a learning rate decay strategy (e.g., the initial value of the learning rate was 0.01, and it decayed by 0.1 every 5 training cycles). During the training process, the cross entropy loss function was used as the loss metric to monitor the loss value and accuracy of the network on the validation set. By adjusting the network's hyperparameters, such as the number of convolution kernels in each layer, the parameters of the residual attention module, and the weight calculation parameters of the multi-scale feature fusion layer, the accuracy of the network on the validation set was gradually improved until the preset convergence standard was reached (e.g., the accuracy was stable above 96% or the loss value was less than 0.03).

[0074] The training process of the deep image feature extraction network is similar. The SGD optimization algorithm is used to set appropriate hyperparameters according to the characteristics of the MobileNetV3 architecture, such as the parameters of the SE module, the convolution kernel size and number of layers of the multi-scale deep feature extraction branch, etc. The network is trained using deep image data, and the loss value and accuracy are monitored through the validation set. The hyperparameters are adjusted until the network converges to ensure that the network can accurately extract facial features in deep images and map them to 2048-dimensional feature vectors.

[0075] The infrared thermal imaging image feature extraction network is trained based on the ViT architecture. The Adam optimization algorithm is used to set an appropriate learning rate (such as 0.0005) and weight decay coefficient (such as 0.0001). During the training process, the parameters of the ViT architecture are adjusted according to the performance of the network on the validation set, such as the number of heads in the multi-head self-attention layer, the number of neurons in the feedforward neural network layer, the size of the image patch, etc., so that the network can effectively extract the face temperature features in the infrared thermal imaging image and generate a 2048-dimensional feature vector.

[0076] (III) Training of multimodal feature fusion and alignment module and face recognition classification and verification module: Based on the trained feature extraction network, feature vectors of a large amount of multimodal image data are obtained for training the mutual information matrix calculation, adaptive weighting algorithm and feature alignment method in the multimodal feature fusion and alignment module. By optimizing these parameters, the fused feature vector can maximize the integration of the effective information of the three modalities and improve the discriminability of the features.

[0077] Train multiple deep learning classification networks and multiple SVMs in the face recognition classification and verification module. For the deep learning classification network, an optimization strategy similar to that of the feature extraction network training is adopted, and the hyperparameters are adjusted according to the characteristics of different architectures to achieve a high accuracy rate in the face identity classification task. For SVM, the kernel function parameters and penalty coefficients are optimized by the grid search method. For example, for the polynomial kernel SVM, the best parameter combination with the polynomial degree between 2-5 is searched, and for the RBF kernel SVM, the appropriate kernel width parameter is searched. At the same time, the historical accuracy weight coefficient in the hierarchical fusion voting mechanism is determined according to the historical accuracy of each classifier on the training set and the verification set, so that the final face recognition system can combine the advantages of multiple classifiers, accurately identify face identities and provide reliable confidence assessment.

[0078] 4. Face recognition operation stage The multimodal image acquisition subsystem monitors the recognition area in real time and starts synchronous acquisition immediately once a face is detected. For example, in an intelligent security monitoring system, when a person enters a specific area of ​​the monitoring screen (such as within 0.3-2 meters from the camera), the system triggers three cameras to collect images simultaneously to obtain sufficient image data. The visible light camera collects clear facial images, and the depth camera and infrared thermal imaging camera collect depth images and thermal imaging images respectively.

[0079] The image preprocessing unit quickly processes the collected image data, and the processed image data is quickly transmitted to the feature extraction network module.

[0080] The feature extraction network module efficiently extracts the feature vectors of the three modal images, and then transmits the feature vectors to the multimodal feature fusion and alignment module. The multimodal feature fusion and alignment module completes the feature fusion and alignment operations, generates the fused feature vectors, and transmits them to the face recognition classification and verification module. The module first calculates the mutual information matrix between the three modal feature vectors, determines the adaptive weights, then performs feature alignment operations, and finally weighted sums the three feature vectors according to the adaptive weights.

[0081] Multiple classifiers in the face recognition classification and verification module process the fused feature vector in parallel, and the hierarchical fusion voting mechanism determines the final face recognition result and calculates the confidence score. If the confidence score is higher than the threshold (such as 0.8) and the recognition result matches the preset authorized person identity, the system will execute the access control system to open; if the confidence score is lower than the threshold or the recognition result does not match, the secondary verification mechanism will be triggered, such as prompting the user to enter a password or answer a security question on the display screen of the access control system, or collecting more face images for re-recognition, which meets the real-time requirements of face recognition in most practical application scenarios.

[0082] In summary, the present invention integrates visible light, depth and infrared thermal imaging cameras, and it is difficult to simulate the features of these three modalities at the same time by counterfeiting human faces. The three modal images are processed in a targeted manner. The dynamic range adjustment, geometric normalization and filtering denoising of the visible light image, the three-dimensional Gaussian filtering and smoothing of the depth image, and the temperature drift correction and enhancement processing of the infrared thermal imaging image improve the accuracy of feature extraction.

[0083] In the feature extraction network module, special feature extraction networks are designed for different modal images. The multi-branch feature extraction architecture of visible light images combined with the residual attention module and the multi-scale feature fusion layer can focus on key feature areas and fuse multi-scale features; the deep image feature extraction network is based on the optimized MobileNetV3 architecture, which can effectively extract deep features; the infrared thermal imaging image feature extraction network adopts the Vision Transformer architecture, which can capture temperature feature associations, and the extracted features are more discriminative, which helps to improve recognition accuracy.

[0084] The multimodal feature fusion and alignment module adopts a strategy based on mutual information maximization and adaptive weighting, as well as a feature alignment method based on canonical correlation analysis. It can fully integrate the effective information of the three modalities, ensure the compatibility and effectiveness of different modal features in the fusion process, and further improve the recognition accuracy.

[0085] The face recognition classification and verification module adopts a multi-level fusion strategy, combining multiple deep learning classification networks of different depths and structures with support vector machines with different kernel functions. The results are determined through a hierarchical fusion voting mechanism, and the voting weights are dynamically adjusted according to the historical accuracy and current confidence scores, thereby combining the advantages of multiple classifiers, improving the accuracy and reliability of face recognition, and significantly reducing the false recognition rate and rejection rate.

[0086] Through the implementation of the above-mentioned face recognition system and method based on deep learning and multimodal feature fusion, it is possible to achieve high-precision, high-reliability face recognition that can effectively resist deception attacks in complex and changing environments, which has broad application prospects and important social value.

[0087] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An access control system based on face recognition, characterized in that: include: The multimodal image acquisition subsystem integrates a visible light camera, a depth camera, and an infrared thermal imaging camera to synchronously acquire visible light images, depth images, and infrared thermal imaging images of a person's face. An image preprocessing unit, which performs dynamic range adjustment, geometric normalization based on facial key points, and denoising based on guided filtering on the visible light image; performs three-dimensional Gaussian filtering smoothing and normalization on the depth image; and performs temperature drift correction and wavelet transform-based enhancement on the infrared thermal imaging image; The feature extraction network module includes convolutional neural networks for extracting visible light image features, deep image features and infrared thermal imaging image features respectively. The visible light image feature extraction network adopts a multi-branch feature extraction architecture, including a parallel convolution layer, a residual attention module, a multi-scale feature fusion layer, a global average pooling layer and a fully connected layer; the deep image feature extraction network is optimized based on a lightweight MobileNetV3 architecture, and is provided with a multi-scale deep feature extraction branch and an SE module; the infrared thermal imaging image feature extraction network adopts a Vision Transformer-based architecture, including a ViT encoder, a global average pooling layer and a fully connected layer; The multimodal feature fusion and alignment module adopts a feature fusion strategy based on mutual information maximization and adaptive weighting, calculates the mutual information matrix to determine the adaptive weights and weighted sums them, and uses a method based on canonical correlation analysis to perform feature alignment; The face recognition classification and verification module adopts a multi-level fusion strategy, including multiple deep learning classification networks with different depths and structures and multiple support vector machines with different kernel functions. The face recognition results are determined through a hierarchical fusion voting mechanism, and a confidence assessment module is provided to calculate the confidence score based on the results and historical accuracy. If the confidence score is above the threshold, the verification is passed, and if it is below the threshold, a secondary verification mechanism is triggered.

2. The access control system based on face recognition according to claim 1 is characterized in that: The visible light camera lens is equipped with optical image stabilization technology and autofocus function, intelligent exposure compensation and color correction function, and the frame rate of the camera can be adjusted between 1-120 frames per second; The depth camera has the function of integrating time-of-flight ToF technology and structured light technology; The infrared thermal imaging camera uses a highly sensitive uncooled focal plane detector to sense infrared rays in the 8-14 micron band.

3. A recognition method for a door access control system based on face recognition according to claim 1 or 2, characterized in that: The following steps are involved: Visible light cameras, depth cameras and infrared thermal imaging cameras are used to synchronously collect visible light images, depth images and infrared thermal imaging images of human faces; The visible light image is subjected to dynamic range adjustment, geometric normalization based on facial key points, and denoising based on guided filtering; the depth image is subjected to three-dimensional Gaussian filtering smoothing and normalization processing; the infrared thermal imaging image is subjected to temperature drift correction and enhancement processing based on wavelet transform; Image features are extracted through a feature extraction network, which includes a visible light image feature extraction network, a deep image feature extraction network, and an infrared thermal imaging image feature extraction network; the visible light image feature extraction network adopts a multi-branch feature extraction architecture, including a parallel convolution layer, a residual attention module, a multi-scale feature fusion layer, a global average pooling layer, and a fully connected layer; the deep image feature extraction network is optimized based on a lightweight MobileNetV3 architecture, and is provided with a multi-scale deep feature extraction branch and an SE module; the infrared thermal imaging image feature extraction network adopts a VisionTransformer-based architecture, including a ViT encoder, a global average pooling layer, and a fully connected layer; A feature fusion strategy based on mutual information maximization and adaptive weighting is adopted. The mutual information matrix is ​​calculated to determine the adaptive weights and weighted summation is performed, and a method based on canonical correlation analysis is used for feature alignment. A multi-level fusion strategy is adopted, which includes multiple deep learning classification networks with different depths and structures and multiple support vector machines with different kernel functions. The face recognition results are determined through a hierarchical fusion voting mechanism, and a confidence assessment is performed. The confidence score is calculated based on the results and historical accuracy. If it is above the threshold, the verification is passed, and if it is below the threshold, the secondary verification mechanism is triggered.

4. The identification method of the access control system based on face recognition according to claim 3 is characterized in that: The visible light image dynamic range adjustment adopts a local contrast enhancement algorithm, specifically: in is the output image pixel value, is the input image pixel value, and are the mean and standard deviation of the local area of ​​the input image, is the mean value of the target area, and k is the contrast enhancement coefficient.

5. The identification method of the access control system based on face recognition according to claim 3 is characterized in that: The denoising algorithm formula of the visible light image based on guided filtering is: ,in is the value of the pixel after filtering, is the value of the input image pixel, and It is the coefficient calculated by minimizing the guided filter cost function.

6. The identification method of the access control system based on face recognition according to claim 3, characterized in that: The three-dimensional Gaussian filtering formula of the depth image is: ,in is the pixel depth value after filtering at point P, is the original pixel depth value of point q, S is the neighborhood of pixel p, , and are Gaussian kernel functions in the x, y and z directions, respectively, used for weighting in the x, y and z directions. is the normalization coefficient. Indicates the distance weight between point p and point q in the x direction; Indicates the distance weight between point p and point q in the y direction; Represents the distance weight between point p and point q in the z direction.

7. The identification method of the access control system based on face recognition according to claim 3 is characterized in that: In the multimodal feature fusion and alignment module, a feature fusion strategy based on mutual information maximization and adaptive weighting is adopted, the mutual information matrix is ​​calculated to determine the adaptive weights and weighted sum, and a method based on canonical correlation analysis is used to perform feature alignment, specifically: First, the mutual information matrix M between the three modal eigenvectors is calculated. The calculation formula is: Where X and Y represent the eigenvectors of two different modes, is their joint probability distribution, and are the respective marginal probability distributions; According to the mutual information matrix, calculate the importance score of each modal feature ,in is an element in the mutual information matrix, representing the mutual information between the i-th modal feature and the j-th modal feature; Calculate adaptive weights based on importance scores , ensure that the sum of the weights is 1; Finally, the three modal eigenvectors F1, F2, and F3 are weighted and summed according to the adaptive weights to obtain the fused eigenvector: Before feature fusion, feature alignment operation is performed.

8. The identification method of the access control system based on face recognition according to claim 3, characterized in that: The deep learning classification network in the face recognition classification and verification module includes networks based on ResNet architecture, DenseNet architecture and EfficientNet architecture, SVM includes linear kernel, polynomial kernel and radial basis function kernel, and the weight calculation formula of the hierarchical fusion voting mechanism is: in, is the weight coefficient of historical accuracy, is the historical accuracy of the classifier, is the confidence score of the current classification.

9. The identification method of the access control system based on face recognition according to claim 7, characterized in that: Before feature fusion, a feature alignment operation is performed, specifically: the feature vectors of the three modes are projected into a common feature subspace using a method based on canonical correlation analysis (CCA), and the alignment of features at the semantic level is achieved by finding the projection direction that maximizes the correlation between the feature vectors of the two modes.

10. The identification method of the access control system based on face recognition according to claim 3, characterized in that: The data sources of the visible light image, depth image and infrared thermal imaging image of the face cover people of different ages, genders and races; the collection scenarios include various extreme and common lighting environments, different facial postures, different facial occlusion situations and facial images in different temperature environments; after the collection is completed, the collected data is labeled, and the labeled information includes but is not limited to facial identity information, image collection environment information, facial posture information, whether there is facial occlusion and the type of occlusion.

Images